# Elastic Search
## Arquivos para baixar
* http://media.sundog-soft.com/es7/series.json
* http://media.sundog-soft.com/es7/shakes-mapping.json
* http://media.sundog-soft.com/es7/shakespeare_7.0.json
* http://media.sundog-soft.com/es7/movies.json
* http://media.sundog-soft.com/es7/series.json
* http://media.sundog-soft.com/es/flattened.txt
* http://media.sundog-soft.com/es/sayt.txt
* http://files.grouplens.org/datasets/movielens/ml-latest-small.zip
* http://media.sundog-soft.com/es7/MoviesToJson.py
* http://media.sundog-soft.com/es7/IndexRatings.py
* http://media.sundog-soft.com/es/access_log
* http://files.grouplens.org/datasets/movielens/ml-100k.zip
* https://raw.githubusercontent.com/coralogix-resources/elk-course-samples/master/csv-schema-short-numerical.csv
* https://raw.githubusercontent.com/coralogix-resources/elk-course-samples/master/csv-read.conf
## 1-Instalando no Linux
```
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch |
sudo apt-key add -
```
```
sudo apt-get install apt-transport-https
```
```
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" |
sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
```
```
sudo apt-get update && sudo apt-get install elasticsearch
```
### Editar o arquivo `elasticsearch.yml`
```
sudo gedit /etc/elasticsearch/elasticsearch.yml
```
- descomentar node name -> node.name: node-1
- alterar network -> network.host: 0.0.0.0
- alterar discovery.seed_hosts: ["127.0.0.1"]
- cluster.initial_master_nodes: ["node-1"]
### Iniciar o Elastic Search pelo Linux
```
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo /bin/systemctl start elasticsearch.service
sudo /bin/systemctl status elasticsearch.service
```
```
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" |
sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
```
:arrow_right: Baixar os arquivos:
`http://media.sundog-soft.com/es7/shakes-mapping.json`
`http://media.sundog-soft.com/es7/shakespeare_7.0.json`
:x: DEU ERRO:
:warning: Digitar o comando em uma única linha
```
curl -H "Content-Type: application/json" -XPUT 127.0.0.1:9200/shakespeare
--data-binary @shakes-mapping.json
```
:x: DEU ERRO:
```
curl -XPUT 127.0.0.1:9200/shakespeare --data-binary @shakes-mapping.json
```
:arrow_right: Indexando os dados:
:warning: Digitar o comando em uma única linha
```
curl -H "Content-Type: application/json" -XPOST
'127.0.0.1:9200/shakespeare/_bulk' --data-binary @shakespeare_7.0.json
```
.
:arrow_right: Pesquisando o termo "ser ou nao ser" no arquivo e json `shakespeare.json`
```
curl -H "Content-Type: application/json" -XGET
'127.0.0.1:9200/shakespeare/_search?pretty' -d '
{
"query": {
"match_phrase": {
"text_entry": "to be or not to be"
}
}
}'
```
.
## 2-Mapenado e Indexando
:arrow_right: Baixar o arquivo `movies.json`
`http://media.sundog-soft.com/es7/movies.json`
:arrow_right: Mapear o arquivo
:x: DEU ERRO:
```
curl -H "Content-Type: application/json" -XPUT '127.0.0.1:9200/movies' -d '
{
"mappings": {
"properties": {
"year": {"type": "date"}
}
}
}'
```
:arrow_right: Pesquisando um filme
```
curl -H "Content-Type: application/json" -XPUT
'127.0.0.1:9200/movies/_doc/109487' -d '
{
"genres": ["IMAX", "Sci-Fi"],
"title": "Interestellar",
"year": 2014
}'
```
.
:arrow_right: Arrumando o codigo json Indexando
```
curl -H "Content-Type: application/json" -XPOST '127.0.0.1:9200/movies/_bulk'
--data-binary @movies.json
```
.
:x: DEU ERRO:
```
curl -H "Content-Type: application/json" -XPUT '127.0.0.1:9200/movies' -d '
{
"mappings": {
"properties": {
"year": {
"type": "date"
}
}
}
}'
```
:x: DEU ERRO:
```
curl -H "Content-Type: application.json" -XGET 127.0.0.1:9200/movies/_mapping
```
:arrow_right: Gravar um documento no json:
```
curl -H "Content-Type: application/json" -XPOST
127.0.0.1:9200/movies/_doc/109487 -d '
{
"genre": ["IMAX", "Sci-Fi"],
"title": "Interstellar",
"year": 2014
}'
```
.
:arrow_right: Pesquisando todos os filmes no arquivo
```
curl -H "Content-Type: application/json" -XGET
127.0.0.1:9200/movies/_search?pretty
```
.
:arrow_right: Indexando
```
curl -H "Content-Type: application/json" -XPUT
127.0.0.1:9200/movies/_bulk?pretty --data-binary @movies.json
```
.
:arrow_right: Pesquisando todos
```
curl -H "Content-Type: application/json" -XGET 127.0.0.1:9200/movies/_search?pretty
```
.
:arrow_right: Alterando um filme, apenas o campo titulo para "Interstellares"
```
curl -H "Content-Type: application/json" -XPOST
127.0.0.1:9200/movies/_doc/109487/_update -d '
{
"doc": {
"title": "Interstellares"
}
}
'
```
.
:arrow_right: Pesquisando "Interstellares" pelo titulo
```
curl -H "Content-Type: application/json" -XGET
'127.0.0.1:9200/movies/_search?pretty' -d '
{
"query": {
"match_phrase": {
"title": "Interstellares"
}
}
}'
```
.
:arrow_right: **Alterando de novo** o mesmo nome para mostrar a versão
:x: DEU ERRO:
```json=
curl -H "Content-Type: application/json" -XPOST
'127.0.0.1:9200/movies/_doc/109487/_update' -d '
{
"genre": ["IMAX", "Sci-Fi"],
"title": "Interstellares Angulares",
"year": 2014
}
'
```
:arrow_right: **Alterando** um filme, pelo campo
```json=
curl -H "Content-Type: application/json" -XPOST
'127.0.0.1:9200/movies/_doc/109487/_update' -d '
{
"doc": {
"title": "Interstellares wow"
}
}
'
```
.
:arrow_right: **Apagando** um documento
```json=
curl -H "Content-Type: application/json" -XDELETE
127.0.0.1:9200/movies/_doc/109487
```
.
:arrow_right: Pesquisando **pelo titulo** "Dark" sem o pretty
```json=
curl -H "Content-Type: application/json" -XGET
127.0.0.1:9200/movies/_search?q=Dark
```
.
:arrow_right: **Apagando** um documento de outra forma
```json=
curl -H "Content-Type: application/json" -XDELETE
127.0.0.1:9200/movies/_doc/58559?pretty
```
.
:arrow_right: Pesquisando novamente. Não foi encontrado.
:arrow_right: **Gravando** "Interstellar" novamente para testar concorrencia
```json=
curl -H "Content-Type: application/json" -XPOST
127.0.0.1:9200/movies/_doc/109487 -d '
{
"genre": ["IMAX", "Sci-Fi"],
"title": "Interstellar",
"year": 2014
}'
```
.
:arrow_right: Listando "Interstellar" **pelo id**.
```json=
curl -H "Content-Type: application/json" -XGET
127.0.0.1:9200/movies/_doc/109487?pretty
```
:arrow_right: Observe o `_seq_no` e `_primary_term`.
.
:arrow_right: Alterando com base nas concorrencias
```json=
curl -H "Content-Type: application/json" -XPUT
"127.0.0.1:9200/movies/_doc/109487?if_seq_no=17&if_primary_term=7" -d '
{
"genre": ["IMAX", "Sci-Fi"],
"title": "Interstellar foo",
"year": 2014
}'
```
.
:arrow_right: Listando novamente. Repare que o numero da **sequencia** e a **versão** mudaram.
.
:arrow_right: Se tentar gravar novamente com uma sequencia anterior (antiga) dará :x: erro pois já estamos em outro sequencial.
.
:arrow_right: Usando um parametro de pesquisa antes de gravar
```json=
curl -H "Content-Type: application/json" -XPOST
"127.0.0.1:9200/movies/_doc/109487/_update?retry_on_conflict=5" -d '
{
"doc": {
"title": "Interstellar foo2"
}
}'
```
.
:arrow_right: Pesquisando todos de forma compacta
```json=
curl -H "Content-Type: application/json" -XGET
127.0.0.1:9200/movies/_search
```
.
:arrow_right: Usando tokennizers e analysers. Buscando pelo campo `title`
```json=
curl -H "Content-Type: application/json" -XGET
"127.0.0.1:9200/movies/_search?pretty" -d '
{
"query": {
"match": {
"title": "Star Trek"
}
}
}'
```
:exclamation::exclamation: Trouxe dois resultados: Star Trek e Star Wars... Trouxe correspondencias parciais. Ele trouxe com base nos "scores", o maior "score" primeiro.
.
:arrow_right: Pesquisou pelo genero especifico de filme "sci", independe de letras maiusculas ou minusculas.
```json=
curl -H "Content-Type: application/json" -XGET
"127.0.0.1:9200/movies/_search?pretty" -d '
{
"query": {
"match_phrase": {
"genre": "sci"
}
}
}'
```
.
:arrow_right: **Deletando** os indices de movies
```json=
curl -H "Content-Type: application/json" -XDELETE 127.0.0.1:9200/movies
```
.
:arrow_right: Criando um **novo mapeamento** de propriedades
```json=
curl -H "Content-Type: application/json" -XPUT "127.0.0.1:9200/movies" -d '
{
"mappings": {
"properties": {
"id": {"type": "integer"},
"year": {"type": "date"},
"genre": {"type": "keyword"},
"title": {"type": "text", "analyzer": "english"}
}
}
}'
```
.
:arrow_right: Criando um **novo indice**
```json=
curl -H "Content-Type: application/json" -XPUT
127.0.0.1:9200/_bulk?pretty --data-binary @movies.json
```
.
:arrow_right: Buscando pelo **genero**
```json=
curl -H "Content-Type: application/json" -XGET
127.0.0.1:9200/movies/_search?pretty -d '
{
"query": {
"match": {
"genre": "sci"
}
}
}
'
```
.
:arrow_right: Baixar o arquivo de series.
**`http://media.sundog-soft.com/es7/series.json`**
:arrow_right: Criando o mapeamento do arquivo
```json=
curl -H "Content-Type: application/json" -XPUT 127.0.0.1:9200/series -d '
{
"mappings": {
"properties": {
"film_to_franchise": {
"type": "join",
"relations": {
"franchise": "film"
}
}
}
}
}
'
```
.
:arrow_right: **Indexando**
```json=
curl -H "Content-Type: application/json" -XPUT
127.0.0.1:9200/_bulk?pretty --data-binary @series.json
```
.
:arrow_right: Pesquisando **pela franquia** (franchise) q
```json=
curl -H "Content-Type: application/json" -XGET
127.0.0.1:9200/series/_search?pretty -d '
{
"query": {
"has_parent": {
"parent_type": "franchise",
"query": {
"match": {
"title": "Star Wars"
}
}
}
}
}'
```
.
:arrow_right: Pesquisando pela **child** (titulo)
```json=
curl -H "Content-Type: application/json" -XGET
127.0.0.1:9200/series/_search?pretty -d '
{
"query": {
"has_child": {
"type": "film",
"query": {
"match": {
"title": "The Force Awakens"
}
}
}
}
}'
```
.
:arrow_right: Baixar outro arquivo para ter base
`http://media.sundog-soft.com/es/flattened.txt`
:arrow_right: Criando um arquivo proprio de log
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/demo-default/_doc/1" -d'{
"message":
"[5592:1:0309/123054.737712:ERROR:child_process_sandbox_support_impl_linux.cc(79)]
FontService unique font name matching request did not receive a response.",
"fileset": {
"name": "syslog"
},
"process": {
"name": "org.gnome.Shell.desktop",
"pid": 3383
},
"@timestamp": "2020-03-09T18:00:54.000+05:30",
"host": {
"hostname": "bionic",
"name": "bionic"
}
}'
```
.
:arrow_right: **Mapenado** o demo_default
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/demo-default/_mapping?pretty=true"
```
.
:arrow_right: Criando o **cluster** com base no meu notebook
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/_cluster/state?pretty=true" >> es-cluster-state.json
```
.
:arrow_right: Criando o arquivo
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/demo-flattened"
```
.
:arrow_right: Criando o **mapeamento**
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/demo-flattened/_mapping" -d'{
"properties": {
"host": {
"type": "flattened"
}
}
}'
```
.
:arrow_right: **Indexando** o arquivo
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/demo-flattened/_doc/1" -d'{
"message":
"[5592:1:0309/123054.737712:ERROR:child_process_sandbox_support_impl_linux.cc(79)]
FontService unique font name matching request did not receive a response.",
"fileset": {
"name": "syslog"
},
"process": {
"name": "org.gnome.Shell.desktop",
"pid": 3383
},
"@timestamp": "2020-03-09T18:00:54.000+05:30",
"host": {
"hostname": "bionic",
"name": "bionic"
}
}'
```
.
:arrow_right: **Mapeando** novamente
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/demo-flattened/_mapping?pretty=true"
```
.
:arrow_right: **Alterando e inserindo** campos no arquivo
```json=
curl -H "Content-Type: application/json" -XPOST
"http://127.0.0.1:9200/demo-flattened/_update/1" -d'{
"doc" : {
"host" : {
"osVersion": "Bionic Beaver",
"osArchitecture":"x86_64"
}
}
}'
```
.
:arrow_right: Verificando se o arquivo foi atualizado
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/demo-flattened/_doc/1?pretty=true"
```
.
:arrow_right: Mapeando o arquivo
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/demo-flattened/_mappings?pretty=true"
```
.
:arrow_right: Buscando o campo **host**
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/demo-flattened/_search?pretty=true" -d'{
"query": {
"term": {
"host.osVersion": "Bionic Beaver"
}
}
}'
```
.
:arrow_right: Buscando o campo **SO**
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/demo-flattened/_search?pretty=true" -d'{
"query": {
"match": {
"host": "Bionic Beaver"
}
}
}'
```
.
:arrow_right: Buscando o campo SO com parte do nome
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/demo-flattened/_search?pretty=true" -d'{
"query": {
"match": {
"host": "Beaver"
}
}
}'
```
.
:arrow_right: Exemplo de mapeamento:
```json=
{
"mappings": {
"properties": {
"timestamp": {"type": "date"},
"service": {"type": "keyword"},
"host_ip": {"type": "ip"},
"port": {"type": "integer"},
"message": {"type": "text"}
}
}
}
```
## 3-Pesquisando no ElasticSearch
### Utilizando Query Lite para estudo (não para produção)
Ex:
```
/movies/_search?q=title:star
/movies/_search?q=+year:>2010+title:trek
```
:arrow_right: Testando a query lite no browser
**`http://localhost:9200/movies/_search?q=title:star`**
.
**`http://localhost:9200/movies/_search?q=+year:%3E2010+title:trek`**
.
:arrow_right: Pesquisando da forma normal junto com Query Lite pelo titulo
```json
curl -XGET "http://127.0.0.1:9200/movies/_search?q=title:star&pretty"
```
.
:arrow_right: Pesquisando da forma normal junto com Query Lite pelo ano e titulo
```json=
curl -XGET
"http://127.0.0.1:9200/movies/_search?q=+year>2010+title:trek&pretty"
```
.
:arrow_right: ou Visualizando no browser
**`http://127.0.0.1:9200/movies/_search?q=+year%3E2010+title:trek&pretty`**
.
### Usando o Request Body Search
:arrow_right: O equivalente da Query Lite
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?pretty" -d'{
"query": {
"match": {
"title": "star"
}
}
}'
```
:arrow_right: Pesquisando utilizando um booleano, titulo e range de anos.
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?pretty" -d'{
"query": {
"bool": {
"must": {"term": {"title": "trek"}},
"filter": {"range": {"year": {"gte": 2010}}}
}
}
}'
```
.
:arrow_right: Pesquisando frases inteira
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?pretty" -d'{
"query": {
"match_phrase": {
"title": "star wars"
}
}
}'
```
.
### Paginação
:arrow_right: Limitando o tamanho para 2 resultados.
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?pretty" -d'
{
"from":2,
"size":2,
"query": {"match": {"genre": "Sci-Fi"}}
}
'
```
.
### Ordenação de dados
:arrow_right: Ordenando pelo ano
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?sort=year&pretty"
```
.
:arrow_right: **Excluir o indice** para poder buscar pelo titulo. (Senão dá erro!)
```json=
curl -H "Content-Type: application/json" -XDELETE
"http://127.0.0.1:9200/movies/"
```
.
:arrow_right: Criar um novo **mapeamento** para poder buscar pelo titulo.
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/movies/" -d '
{
"mappings": {
"properties": {
"title": {
"type": "text",
"fields": {
"raw": { "type": "keyword" }
}
}
}
}
}
'
```
.
:arrow_right: Criar um novo **index** para poder buscar pelo titulo.
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/_bulk" --data-binary @movies.json
```
.
:arrow_right: **Ordenando** pelo titulo
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?sort=title.raw&pretty"
```
.
### Pesquisa parcial
:arrow_right: Para utilizar a buscar parcial, precisamos **deletar** o indice e criar um novo mapeamento.
```json=
curl -H "Content-Type: application/json" -XDELETE
"http://127.0.0.1:9200/movies/"
```
.
:arrow_right: Criando **novo mapeamento**
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/movies/" -d '
{
"mappings": {
"properties": {
"year": { "type": "text" }
}
}
}'
```
.
:arrow_right: Criando **novo indice**
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/_bulk" --data-binary @movies.json
```
.
:arrow_right: Buscando por parte do ano
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?pretty" -d '
{
"query": {
"prefix": {
"year": "201"
}
}
}'
```
.
### N-grams
Ex: "star"
unigram: [s,t,a,r]
bigram: [st,ta,ar]
trigram: [sta, tar]
4-gram: [star]
:arrow_right: Criando um auto-completes. Exemplos:
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/movies?pretty" -d '
{
"settings": {
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 20
}
},
"analyzer": {
"autocomplete": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"autoconplete_filter"
]
}
}
}
}
}'
```
.
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/movies?pretty" -d '
{
"properties": {
"title": {
"type": "text",
"analyzer": "autocomplete"
}
}
}'
```
.
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/movies?pretty" -d '
{
"query": {
"match": {
"title": {
"query": "sta",
"analyzer": "standard"
}
}
}
}'
```
:arrow_right: Para trabalharmos com o **gram**, teremos de **deletar** nosso indice.
```json =
curl -H "Content-Type: application/json" -XDELETE "http://127.0.0.1:9200/movies/"
```
.
:arrow_right: Criando **novo mapeamento**
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/movies?pretty" -d '
{
"settings": {
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 20
}
},
"analyzer": {
"autocomplete": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"autocomplete_filter"
]
}
}
}
}
}'
```
.
:arrow_right: Aplicando o filtro. Unigran, bigram e trigran
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_analyze?pretty" -d '
{
"analyzer": "autocomplete",
"text": "Sta"
}'
```
.
:arrow_right: Mapear o analizador
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/movies/_mapping?pretty" -d '
{
"properties": {
"title": {
"type": "text",
"analyzer": "autocomplete"
}
}
}'
```
.
:arrow_right: Criar o indice
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/_bulk" --data-binary @movies.json
```
.
:arrow_right: Pesquisar por parte da palavra "sta". Mas veio resultados a mais.
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?pretty" -d '
{
"query": {
"match": {
"title": "sta"
}
}
}'
```
.
:arrow_right: Nova consulta utilizando o analyzer. Buscando a palavra "sta"
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?pretty" -d '
{
"query": {
"match": {
"title": {
"query": "sta", "analyzer": "standard"
}
}
}
}'
```
.
### Autocomplete e busca pelo tipo
:arrow_right: Baixar o arquivo de comandos
`http://media.sundog-soft.com/es/sayt.txt`
:arrow_right: Criando novo filtro
```json=
curl -H "Content-Type: application/json" --silent --request POST
'http://localhost:9200/movies/_analyze?pretty' --data-raw '{
"tokenizer" : "standard",
"filter": [{"type":"edge_ngram", "min_gram": 1, "max_gram": 4}],
"text" : "Star"
}'
```
.
:arrow_right: Criando novo filtro pelo titulo e genero
```json=
curl -H "Content-Type: application/json" --request PUT
'http://localhost:9200/autocomplete' \
-d '{
"mappings": {
"properties": {
"title": {
"type": "search_as_you_type"
},
"genre": {
"type": "search_as_you_type"
}
}
}
}'
```
.
:arrow_right: Pesquisando novamente. Resultado foi de **5 filmes.**
```json=
curl -H "Content-Type: application/json" --silent --request POST
'http://localhost:9200/_reindex?pretty' --data-raw '{
"source": {
"index": "movies"
},
"dest": {
"index": "autocomplete"
}
}' | grep "total\|created\|failures"
```
.
:arrow_right: Mapeando o **autocomplete**
```json=
curl -H "Content-Type: application/json"
"http://127.0.0.1:9200/autocomplete/_mapping?pretty"
```
.
## 4-Importando dados usando scripts
### Importando com Python
:arrow_right: Baixar o arquivo zipado e descompactar.
`http://files.grouplens.org/datasets/movielens/ml-latest-small.zip`
:arrow_right: Baixar o codigo em pyton
`http://media.sundog-soft.com/es7/MoviesToJson.py`
:arrow_right: Criar um arquivo em pyhton com o nome: `MoviesToJson.py`. Atenção para o caminho do arquivo.
```python=
import csv
import re
csvfile = open('movies.csv', 'r')
reader = csv.DictReader( csvfile )
for movie in reader:
print ("{ \"create\" : { \"_index\": \"movies\", \"_id\" : \"" ,
movie['movieId'], "\" } }", sep='')
title = re.sub(" \(.*\)$", "", re.sub('"','', movie['title']))
year = movie['title'][-5:-1]
if (not year.isdigit()):
year = "2016"
genres = movie['genres'].split('|')
print ("{ \"id\": \"", movie['movieId'], "\", \"title\": \"",
title, "\", \"year\":",
year, ", \"genre\":[", end='', sep='')
for genre in genres[:-1]:
print("\"", genre, "\",", end='', sep='')
print("\"", genres[-1], "\"", end = '', sep='')
print ("] }")
```
:arrow_right: Abrir o `CMD` como administrador e digitar o comando de importação em Python:
```json=
python3 MoviesToJson.py > moremovies.json
```
.
:star: O documento foi importado com SUCESSO!
.
:arrow_right: **Criar o indice do novo arquivo.**
```json=
curl -H "Content-Type: application/json" -XPUT
"http://127.0.0.1:9200/_bulk" --data-binary @moremovies.json
```
.
:arrow_right: **Pesquisando um titulo:** `Mary Poppins`
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/movies/_search?q=mary%20poppins&pretty"
```
.
:arrow_right: **Baixar o codigo em python**
`http://media.sundog-soft.com/es7/IndexRatings.py`
:arrow_right: Criar um arquivo em python com o nome: `IndexRatings`. Atenção para o caminho do arquivo.
```python=
import csv
from collections import deque
import elasticsearch
from elasticsearch import helpers
def readMovies():
csvfile = open('ml-latest-small/movies.csv', 'r')
reader = csv.DictReader( csvfile )
titleLookup = {}
for movie in reader:
titleLookup[movie['movieId']] = movie['title']
return titleLookup
def readRatings():
csvfile = open('ml-latest-small/ratings.csv', 'r')
titleLookup = readMovies()
reader = csv.DictReader( csvfile )
for line in reader:
rating = {}
rating['user_id'] = int(line['userId'])
rating['movie_id'] = int(line['movieId'])
rating['title'] = titleLookup[line['movieId']]
rating['rating'] = float(line['rating'])
rating['timestamp'] = int(line['timestamp'])
yield rating
es = elasticsearch.Elasticsearch()
es.indices.delete(index="ratings",ignore=404)
deque(helpers.parallel_bulk(es,readRatings(),index="ratings"), maxlen=0)
es.indices.refresh()
```
:arrow_right: Criar o indice do novo arquivo. :x: DEU ERRO:
```josn=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/ratings/_search?pretty
```
.
### Introdução e Instalação Logstash
#### Baixar o Logstash
`https://www.elastic.co/pt/downloads/logstash`
.
.
.
.
#### Baixar o log de exemplo:
`http://media.sundog-soft.com/es/access_log`
.
#### Criar o arquivo de configuração do logstash
Salvar dentro do diretorio `config` da pasta do Logstash
Ex: D:\Lu\Downloads\logstash-7.14.0-windows-x86_64\logstash-7.14.0\config
**logstash.conf**
```json=
input {
file { path => "C:\tempelastic\access_log"
start_position => "beginning" }
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
```
#### Comando para iniciar o Logstash no windows:
:warning: Lembrando: O comando precisa estar numa linha só.
Abrir o CMD dentro da pasta do Logstash
Ex: **D:\Lu\Downloads\logstash-7.14.0-windows-x86_64\logstash-7.14.0>**
```json=
.\bin\logstash.bat -f D:\Lu\Downloads\logstash-7.14.0-windows-x86_64\
logstash-7.14.0\config\logstash.conf
```
ou
```json=
.\bin\logstash.bat -f .\config\logstash.conf
```
.
Aguardar a leitura do log.
:warning: Pode demorar...
.
Leitura finalizada. Desligar o terminal: `CTRL+C`
.
Para criar novos indices.
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/_cat/indices?v"
```
.
Com base nos indices criados, fazer uma pesquisa no log `logstash-2021.08.29-000001`
```json=
curl -H "Content-Type: application/json" -XGET
"127.0.0.1:9200/logstash-2021.08.29-000001/_search?pretty"
```
.
Consultando outro log. Esse log não foi encontrado :x:
```json=
curl -H "Content-Type: application/json" -XGET
"http://127.0.0.1:9200/logstash-2019.05.07-000001/_search?pretty"
```
.
### Logstach e MySql
Baixar o arquivo e descompactar. Digitar o endereço na web.
```
http://files.grouplens.org/datasets/movielens/ml-100k.zip
```
Abrir o **MySql** e digitar os comandos para criar o banco e a tabela
```
#CRIAÇÃO DO BANCO
create database movielens;
#CRIAÇÃO DA TABELA
create table movielens.movies (
movieid int primary key not null,
title text,
releaseDate Date
);
```
.
Comando para importação do arquivo
```
#IMPORTAÇÃO DO ARQUIVO
load data local infile
'C:/tempelastic/ml-100k/u.item' into table movielens.movies fields
terminated by '|'
(movieId, title, @var3)
set releaseDate = STR_TO_DATE(@var3, '%d-%M-%Y');
```
.
Mostrar tabela criada.
```
show tables;
```
.
Arquivos importados
```
select * from movies
```
.
```
select * from movies where title like 'Star%';
```
.
Dentro da pasta **logstash/config** criar o arquivo `mysql.conf`
```=
input{
jdbc{
jdbc_connection_string => "jdbc:mysql://localhost:3306/movielens"
jdbc_user => "root"
jdbc_password => "arq2020"
jdbc_driver_library => "C:/tempelastic/mysql-connector-java-5.1.23.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "select * from movies"
}
}
output{
stdout{
codec => json_lines
}
elasticsearch {
hosts => ["localhost: 9200"]
index => "movielens-sql"
}
}
```
.
.
:warning: Lembrando: O comando precisa estar numa linha só.
Abrir o CMD dentro da pasta do Logstash
Ex: **D:\Lu\Downloads\logstash-7.14.0-windows-x86_64\logstash-7.14.0>**
Iniciar o arquivo com o **Logstash**. Deu ruim :x: Não conectou.
```json=
.\bin\logstash.bat -f .\config\mysql.conf
```
.
Para consultar a partir do banco MySql. Deu ruim :x:
```json=
curl -H "Content-Type: application/json" -XGET
"localhost:9200/movielens-sql/_search?pretty"
```
.
### Importando CSV
Entrar no diretorio dos seus arquivos, criar uma pasta chamda `CSV` e abrir o terminal dentro dela. Digitar o comando para baixar o arquivo em CSV.
```json=
curl -H "Content-Type: application/json" -O
"https://raw.githubusercontent.com/coralogix-resources/elk-course-samples/master/csv-schema-short-numerical.csv"
```
Arquivos baixados.
.
Conteúdo do arquivo `csv`.
.
Abrir conteúdo do arquivo na web e copiar.
```
https://raw.githubusercontent.com/coralogix-resources/elk-course-samples/master/csv-read.conf
```
.
:floppy_disk: Salvar o arquivo com o nome de `csv-read.conf` dentro da pasta do **logstash/conf**
:warning: Atenção para o caminho do arquivo csv, na linha `3`.
:warning: Para o cabeçalho do arquivo na linha `12`
```json=
input {
file {
path => "C:/tempelastic/csv/csv-schema-short-numerical.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
skip_header => "true"
columns => ["id","timestamp","paymentType","name","gender","ip_address","purpose","country","age"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "demo-csv"
}
stdout {}
}
```
.
:eye: Visualizando no explorer do windows. Arquivo criado na pasta **logstash/config**
.
Dentro desta pasta abrir um terminal e digitar o comando:
```json
.\bin\logstash.bat -f .\conf\csv-read.conf
```
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
---
## 100 - Kibana `http://localhost:5601`
Ler o log do Shakespeare.
Kibana -> Discovery
.
Digitar uma palavra para filtrar.
"die" -> clicar em UPDATE
.
Clicar em PLAY_NAME
.
Clicar em VISUALIZE
.
Mostrando uma outra forma de visualização.
.
## Fazer uma `tag cloud` de palavras mais usadas
Clicar em Analytics -> Visualize Library
.
Criar uma visualização
.
Selecionar o tipo de gráfico agregation based -> explore options
.
Clicar em `Tag Cloud`
.
Selecionar a fonte de dados
.
.
Do lado direito selecionar os campos para mostrar. Count para contar so termos
.
Em aggregation, selecionar `Terms`, `text_entry.keyword`, `Metric: Count`, Size = 5 (as 5 palavras que mais aparecem nos textos)
.
Clicar em UPDATE
.
Mudar Count para **20** (as 20 palavras que mais aparecem)
.
### Criar um espaço no Kibana.
.
### Fazer acesso pelo terminal
```
curl -H "Content-Type: application/json"
'localhost:5601/api/spaces/space/devops'
```
.
#### :x: DEU ERRO:
`https://raw.githubusercontent.com/elastic/uptime-contrib/master/dashboards/http_dashboard.json`
## Elastic Search com Sql
### Consultando como no banco de dados
```
curl -H "Content-Type: application/json" -XPOST 127.0.0.1:9200/_xpack/sql -d '
{
"query": "describe movies"
}
'
```
.
### Descrevendo os campos e apresentando de uma forma melhor
```
curl -H "Content-Type: application/json" -XPOST
127.0.0.1:9200/_xpack/sql?format=txt -d '
{
"query": "describe movies"
}
'
```
.
### Pesquisando com SELECT os campos e apresentando de uma forma melhor
```
curl -H "Content-Type: application/json" -XPOST
127.0.0.1:9200/_xpack/sql?format=txt -d '
{
"query": "select title from movies limit 10"
}
'
```
.
### Pesquisando com SELECT filmes anteriores do ano 2000
```
curl -H "Content-Type: application/json" -XPOST
127.0.0.1:9200/_xpack/sql?format=txt -d '
{
"query": "select title, year from movies where year<2000 order by year"
}
'
```
.
```
curl -H "Content-Type: application/json" -XPOST
127.0.0.1:9200/_xpack/sql/translate?pretty -d '
{
"query": "select title, year from movies where year<2000 order by year"
}
'
```
.
## Inicializar o elasticsearch-sql-cli
Clicar em `elasticsearch-sql-cli.bat`
.
```
describe movies;
```
.
```
select title, year from movies where year> 1920 order by year
```
.
## 3-Ruby
### Fazer a instalação do Ruby
:warning: O Logstash só funciona com o Ruby instalado na maquina...
`https://rubyinstaller.org/`
.
## 3-Kibana
### Um guia para o Kibana
https://coralogix.com/log-analytics-blog/advanced-guide-to-kibana-timelion-functions
.
### Baixar os logs de exemplo pelo site:
`media.sundog-soft.com/es/canvas.txt`
.
Salvar o arquivo sem extensão
```
wget https://raw.githubusercontent.com/elastic/examples/master/
Common%20Data%20Formats/nginx_json_logs/nginx_json_logs
```
.
Converter o arquivo.
#### :x: DEU ERRO:
```
awk '{print "{\"index\":{}}\n" $0}' nginx_json_logs > nginx_json_logs_bulk
```
Converter o arquivo.
```
curl -H "Content-Type: application/json" --request PUT
"http://localhost:9200/nginx" -d '
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"properties": {
"time": {"type":"date","format":"dd/MMM/yyyy:HH:mm:ss Z"},
"response": {"type":"keyword"}
}
}
}'
```
.
#### :x: DEU ERRO:
Instalar o JQ
```
curl -H "Content-Type: application/json" --request POST 'http://localhost:9200/nginx/_doc/_bulk' --data-binary '@nginx_json_logs_bulk'
```
Abrir o Kibana novamente, selecionar o SPACE DEVOPS
.
Ir para o console ==Dev Tools==
```
POST /_sql?format=txt
{
"query": "select sum(bytes) as total_transferred from nginx2 group by remote_ip order by total_transferred desc nulls last limit 5"
}
```
###### tags: `elasticsearch` `logstash` `kibana`
Google Drive
Gist
Clipboard
Sign in with Wallet
