--- title: Elastic Search --- # Elastic Search ###### tags: `infrastructure` `Golang` [toc] ## 重要概念 如果遇到欄位類型報錯，需要於創建index時指定欄位類型 https://www.elastic.co/guide/cn/elasticsearch/guide/current/dynamic-mapping.html ``` PUT /match_live_stats { "mappings" : { "properties" : { "stats_live" : { "properties" : { "away" : { "type" : "float" }, "home" : { "type" : "float" }, "type" : { "type" : "integer" } } }, //以上是我需要的欄位，以下是get mappings 自動產生的 "type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 // 指定長度 } } } } } } ``` 查看index mapping ``` get /match_live_stats/_mapping ``` ## 超多索引沒有性能問題 ![](https://i.imgur.com/5A0IDG9.png) ## 安裝紀錄 1. docker run -p 9200:9200 -d --name elasticsearch elasticsearch:8.2.2 2. docker run -p 9800:9800 -d --link elasticsearch:demo containerize/elastichd 遇到錯誤: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 3. 成功安裝以後API皆無回應。 改安裝舊版本測試則成功 : 4. docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name=myes docker.elastic.co/elasticsearch/elasticsearch:7.17.1 5. http://localhost:9200 可看到ES回應 6. docker run --link myes:elasticsearch -p 5601:5601 docker.elastic.co/kibana/kibana:7.17.1 7. http://localhost:5601 可進入kibana頁面 ps: docker run --link 指令， “：”前面是發送消息來源，後面是給來源的alias。 功能是讓兩個容器之間互通。 但link 指令後續將被捨棄，改使用network 安裝參考文：https://myapollo.com.tw/zh-tw/docker-elasticsearch： ## Golang連接測試 參考代碼: https://github.com/ronnielin8862/go-practice/blob/master/cmd/db/ES/main.go ## 帳密設定 1. 進入container /usr/share/elasticsearch/ 2. elasticsearch-useradd RONNIE 接著會要你設定密碼 3. elasticsearch-password 會設定一票密碼 包含遠端登入 ## kibana 操作 官方文黨: https://www.elastic.co/guide/cn/elasticsearch/guide/current/create-doc.html ### 查詢 #### 查詢所有index: ``` GET _cat/indices ``` #### index search by wildcard: ``` GET match_live_foo* GET _cat/indices/*3809489*?v ``` #### 查詢index,type 下的所有document: (默認返回十筆,size設定默認最大返回一萬筆超過報錯, 可修改) ``` 52.221.194.38:9200/es_chat_history/_doc/_search ``` #### 查詢index,type 下的指定id: ``` 52.221.194.38:9200/es_chat_history/_doc/1 ``` #### 查詢任意field = yess: ``` GET es_chat_history/_doc/_search?q=yess ``` #### 大量資料查詢: https://blog.csdn.net/woshicheng1990/article/details/49452051 ``` GET /old_index/_search?scroll=10m { "query": { "match_all": {}}, "size": 1000 } ``` #### 模糊匹配: ``` GET football_text_live_3747688/_doc/_search?q=*第1个进球* ``` #### 排序: ``` GET football_text_live_3753612/_search { "query": { "match_all": {} } , "sort": [ { "create_time": { "order": "desc"} } ] } ``` ### filter ``` GET match_live_basketball_score/_search { "query": { "match": { "_id": 3677562 } } } ``` ### 更改index ``` POST _reindex { "source": { "index": "history-prices-python" }, "dest": { "index": "history-prices-daily" } } ``` https://www.cnblogs.com/progor/p/11557869.html#请求体条件搜索 ### 刪除 DELETE /index_one,index_two delete: https://www.elastic.co/guide/cn/elasticsearch/guide/current/_deleting_an_index.html delete by query: (此刪除方式，不受限於find by query的最大返回數限制) https://www.elastic.co/guide/en/elasticsearch/reference/7.17/docs-delete-by-query.html ``` POST match_live_text_3699878/_delete_by_query?routing=1 { "query": { "range" : { "create_time" : { "gte": 1658157367, "lte": 1658159341 } } } } ``` ex: delete by id ``` POST /match_live_stats/_delete_by_query?routing=1 { "query": { "match": { "_id": "3679177" } } } ``` ### 更多查詢參考文章 https://www.jianshu.com/p/c6708ea88710?u_atoken=faacd07f-3ff0-4a39-aa06-faef158020aa&u_asession=01fYt5mnLpKdYrQQRx2gK18_jqCBGxxra9MSj-k6sVkt5vUuJSw3dK1kigcFt4OOr7X0KNBwm7Lovlpxjd_P_q4JsKWYrT3W_NKPr8w6oU7K959Tn58jRF0t4oIvoNNtX_MKWrbBzYAhXhkL4v5_cjQmBkFo3NEHBv0PZUm6pbxQU&u_asig=05ZKWgk1Uml8jErQXZk-a32P3sFHgSSOCAQG3DIUxM-67MJCx0umisRB8iZKDny9i-_lOD7501n6nMgtNGO79_p61Z62eIJLur5wStFNujY2NqHN2_kPzB4-ZYN-R_X-epUre-4rTosNvl45JsVwlZ1raoYCsqcTrZVYKlX3s60rP9JS7q8ZD7Xtz2Ly-b0kmuyAKRFSVJkkdwVUnyHAIJzXQeFmdBv3_OotNWJghqKRpeKcqCodD-eobjPGOL6sjtdf9JIAkyKervFWgmMgV8j-3h9VXwMyh6PgyDIVSG1W8-UtyZ8wcWzDZng4vWRy-nc4qWnPeNnkyWEERpY2mCvdtlRKnam-pLphqATahY6eJmS5vDudGEYBMwDs_6XSB0mWspDxyAEEo4kbsryBKb9Q&u_aref=jtChMTeqKTfYzOhPg4tEOJ6K64U%3D https://www.cnblogs.com/wupeixuan/p/12483846.html ## Golang 操作 range query: ``` bq := elastic.NewBoolQuery().Filter(elastic.NewRangeQuery("create_time").Gt(1657959002)) do, err := client.Search("match_live_text_3666726").Query(bq).Size(1000).Do(context.Background()) ``` 範例採用github.com/oliver/elastic/v7: https://www.topgoer.com/%E6%95%B0%E6%8D%AE%E5%BA%93%E6%93%8D%E4%BD%9C/go%E6%93%8D%E4%BD%9Celasticsearch/%E6%93%8D%E4%BD%9Celasticsearch.html https://www.tizi365.com/archives/858.html 坑點: 1. 排序文字時會報錯 :elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception] 需要在排序的字段後面加上 ".keyword", 例如 ``` sort := elastic.NewFieldSort("create_time.keyword").Order(true) res, err := server.GetEs().Search(fmt.Sprint(messages.TextLive, "_", strconv.Itoa(usr.RoomId))).SortBy(sort).Size(300).Do(context.Background()) ``` 官方文檔: https://www.elastic.co/guide/en/elasticsearch/reference/7.14/getting-started.html ## 其他ES配置參考 (要看一下) #### 最大索引: ``` PUT /_cluster/settings { "transient": { "cluster": { "max_shards_per_node":10000 } } } ``` #### 模糊匹配刪除 : ``` PUT /_cluster/settings { "persistent": { "action.destructive_requires_name": false } } ``` true的設定會禁止使用通配符刪除 #### 開始自動創建索引: ``` PUT /_cluster/settings { "persistent" : { "action": { "auto_create_index": "true" } } } ``` #### 確認版本: ``` curl -XGET 'http://localhost:9200' ``` #### 調整最大查詢返回數量: 全局調整: ``` PUT _all/_settings { "index.max_result_window": 1000000 } ``` 另外也可以針對特定索引調整 ## 冷熱分離策略 https://blog.51cto.com/u_15050720/2562253 ## 坑 1. compressor detection can only be called on some xcontent bytes or compressed xcontent bytes elastic 插入的值即便是jason formate, 也不能是array, 必須是map裡頭包裝array