InfluxDB + OpenTelemetry

本文紀錄如何把基礎的 Observability Tools 架設起來
主要是 influxDB + OpenTelemetry

0x00 目標

我們的目標是在一個分散式系統上架設效能觀測工具，如下圖所示，在 Monitiored System [1..n] 上架設我們效能工具 (ddshark) 以及一個現成的觀測工具 (Node Exporter)，兩種工具是用不同的協定發送觀測資料，因此有兩個 data storage 分別儲存效能資料。最後用 Grafana UI 視覺畫工具分析效能資料。

0x01 確認運作正常

otel-collector

圖中，我們使用 ddshark 當作示範的客製化效能工具。ddshark 使用 opentelemetry 的協定發送 trace 資料。我們的目標是把 opentelmetry 格式的資料儲存到 InfluxDB 裡，但是 InfluxDB 是沒辦法直接接收 opentelmetry 的資料，因此我們需要另一個工具叫 Opentelemetry Collector

首先，下載 opentelemetry-collector-contrib，你可能會疑問，為何不下載官方版本的 opentelemetry-collector-releases 呢？這是因為能夠支援 trace 轉換成 InfluxDB 格式資料的 exporter 已經包含在 contrib 裡面，而官方版本並沒有這個功能。

移動到 opentelemetry-collector-contrib 的目錄底下，build 之後會有 otelcontribcol_linux_amd64 這個執行檔，首先，先確認我們的效能工具丟出的 opentelemetry trace 能夠被 otel-collector 接收。

修改 otel-collector 的 yaml 設定檔，位於 examples/demo/otel-collector-config.yaml，把接收到的 opentelemetry trace 顯示在終端機上。

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      label1: value1

  logging:
    verbosity: detailed

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, prometheus]

啟動 otel-collector 之後，執行我們的效能工具 (ddshark)，並且丟出幾個測試用的 OpenTelemetry trace，可以看到 otel-collector 的確有接到到我們發送的資料。

❯ ./bin/otelcontribcol_linux_amd64 --config ./examples/demo/otel-collector-config.yaml
2024-05-12T17:40:20.668+0800    info    service/telemetry.go:81 Setting up own telemetry...
2024-05-12T17:40:20.669+0800    info    service/telemetry.go:104        Serving Prometheus metrics      {"address": ":8888", "level": "Basic"}
2024-05-12T17:40:20.669+0800    info    exporter@v0.79.1-0.20230609201858-ed8547a8e5d6/exporter.go:275  Development component. May change in the future.      {"kind": "exporter", "data_type": "traces", "name": "logging"}
2024-05-12T17:40:20.670+0800    info    exporter@v0.79.1-0.20230609201858-ed8547a8e5d6/exporter.go:275  Development component. May change in the future.      {"kind": "exporter", "data_type": "metrics", "name": "logging"}
2024-05-12T17:40:20.672+0800    info    service/service.go:131  Starting otelcontribcol...      {"Version": "0.79.0-dev", "NumCPU": 24}
2024-05-12T17:40:20.672+0800    info    extensions/extensions.go:30     Starting extensions...
2024-05-12T17:40:20.672+0800    warn    internal/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks      {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-05-12T17:40:20.673+0800    info    otlpreceiver@v0.79.1-0.20230609201858-ed8547a8e5d6/otlp.go:83   Starting GRPC server  {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4317"}
2024-05-12T17:40:20.673+0800    warn    internal/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks      {"kind": "exporter", "data_type": "metrics", "name": "prometheus", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-05-12T17:40:20.673+0800    info    service/service.go:148  Everything is ready. Begin running and processing data.
2024-05-12T17:48:43.665+0800    info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 4, "spans": 4}
2024-05-12T17:48:43.665+0800    info    ResourceSpans #0
Resource SchemaURL: 
Resource attributes:
     -> service.name: Str(dds.traffic)
     -> host.name: Str(st9540808-i9-13900K)
ScopeSpans #0
ScopeSpans SchemaURL: 
InstrumentationScope opentelemetry-otlp 0.12.0
Span #0
    Trace ID       : 3e62d9d7d5e8ce2d1c22922b0aee5d09
    Parent ID      : 
    ID             : 5bc761f206f2aee8
    Name           : DATA_FRAG
    Kind           : Internal
    Start time     : 2023-05-16 06:49:44.611149 +0000 UTC
    End time       : 2023-05-16 06:49:44.611192008 +0000 UTC
    Status code    : Unset
    Status message : 
Attributes:
     -> event.name: Str(eno2)
     -> traffic_type: Str(USER_DEFINED)
     -> data_size: Int(3537316)
     -> payload_size: Int(13440)
     -> topic_name: Str(<none>)
     -> fragment_starting_num: Int(1541)
     -> pcp: Int(0)
     -> submsg_type: Str(DATA_FRAG)
     -> writer_id: Str(01:10:59:06:e3:25:a3:7e:93:91:ab:42:00:00:37:03)
     -> sn: Int(107)

如果需要用測試 otel-collector 跑在 docker container 可用以下命令

 docker run -p 4317:4317 -v $(pwd)/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml otel/opentelemetry-collector-contrib:0.79.0

InfluxDB

otel-collector 測試可以使用之後，接下來要加入 InfluxDB，確認我們工具丟出來的 trace 可以經由 otel-collector 最終寫入 InfluxDB 資料庫裡。

這裡我們直接使用 InfluxDB 官方提供的 docker image，省去自己準備環境的麻煩。使用以下命令啟動 InfluxDB，並將 container 中的連接埠號 8086 映射到主機的 8086。

❯ docker run --rm -p 8086:8086 --name influxdb influxdb:latest
2024-05-13T15:48:47.248103400Z  warn    boltdb not found at configured path, but DOCKER_INFLUXDB_INIT_MODE not specified, skipping setup wrapper      {"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt"}
2024-05-13T15:48:47.333610215Z  warn    boltdb not found at configured path, but DOCKER_INFLUXDB_INIT_MODE not specified, skipping setup wrapper      {"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt"}
ts=2024-05-13T15:48:47.478721Z lvl=info msg="Welcome to InfluxDB" log_id=0p8f78yG000 version=v2.7.1 commit=407fa622e9 build_date=2023-04-28T13:24:27Z log_level=info
ts=2024-05-13T15:48:47.478743Z lvl=warn msg="nats-port argument is deprecated and unused" log_id=0p8f78yG000
ts=2024-05-13T15:48:47.480756Z lvl=info msg="Resources opened" log_id=0p8f78yG000 service=bolt path=/var/lib/influxdb2/influxd.bolt
...

在測試之前，還必須要設定 InfluxDB，當第一次啟動時，InfluxDB 會需要你設定基本資訊，以及 Organization Name, Bucket Name，這兩個資訊相當重要，等等在設定 otel-collector 時需要用到。

輸入完基本資訊，InfluxDB 就算設定好了，接下來會給你最高權限的 token，讓你可以任意寫入讀取 InfluxDB 裡的資料。

接下來回到 otel-collector 這邊，我們需要把剛剛的資訊填 otel-collector 的 yaml 設定檔。這裡我們中間 exporters 新增一個 InfluxDB 的 exporter，以及對應的 org, bucket, token，讓 exporter 知道要把 OpenTelemtry trace 轉送到哪個地方。

同時，將最下面的 service 裡面 exporters 欄位改成 [logging, influxdb]，讓 otel-collector 街收到 OpenTelemtry trace 的時候，把資訊送往終端機以及 InfluxDB。

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  influxdb:
    endpoint: "http://localhost:8086"
    timeout: 500ms
    org: my-org
    bucket: my-bucket
    token: twYP7TRYD5jNB9MtUnJSNMxelc-s1jZCsOMEFQUYIs0UPZWq1A8QSIxFiOMV99ehORcvRvhwNAOboQYMB5YxZA==

  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      label1: value1

  logging:
    verbosity: detailed

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, influxdb]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, prometheus]

在 InfluxDB 還正在跑的時候，我們重新啟動 otel-collector，並且用我們的效能工具 (ddshark) 丟出幾個測試用的 OpenTelemetry trace，如果一切運作正常，就能看到 trace 成功出現在 InfluxDB 的 data exporter 裡面，如下圖所示。

圖中是將界面切換成 "QUERY BUILDER" 之後，使用以下 InfluxDB Query 尋找資料。

from(bucket: "my-bucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "spans")
  |> filter(fn: (r) => r["_field"] == "attributes")
  |> filter(fn: (r) => r["service.name"] == "dds.traffic")

0x02 架設工具

如文中一開始的圖所示，這個觀測系統的必要基礎設施為圖中正中間框起來的三個工具 Opentelemetry Collector, InfluxDB, Prometheus。

我們遇到的一個問題是，每次把這些工具一個一個跑起來十分麻煩，有沒有可能一鍵把所有必要基礎設施架設起來？這裡我們使用 docker compose 這個命令將所有工具，啟動在個別的 docker container 裡面。

同時我們希望只需要設定 InfluxDB 一次就好，另外，當停止所有 container 時能夠把當前狀態以及資料庫，都存到本機上，等到下次再執行 docker compose，所有的狀態都還存在。

使用的腳本參考這個 repo https://github.com/build-on-aws/instrumenting-java-apps-using-opentelemetry

以下為架設工具所需要的檔案及其結構，o11y-backend 目錄底下的資料不需要更改，有需要調校 InfluxDB 的效能或行為時再去修改即可。

.
├── collector-config-local.yaml
├── docker-compose.yaml
├── influxdb2
│   ├── engine
│   ├── influxd.bolt
│   └── influxd.sqlite
└── o11y-backend
    ├── influxdb-config.yaml
    └── prometheus.yaml

原始 repo 中沒有提供 influxdb-config.yaml，以下列出我用的設定檔當作參考。

assets-path: ""
bolt-path: /var/lib/influxdb2/influxd.bolt
e2e-testing: false
engine-path: /var/lib/influxdb2/engine
feature-flags: {}
flux-log-enabled: false
hardening-enabled: false
http-bind-address: :8086
http-idle-timeout: 3m0s
http-read-header-timeout: 10s
http-read-timeout: 0s
http-write-timeout: 0s
influxql-max-select-buckets: 0
influxql-max-select-point: 0
influxql-max-select-series: 0
instance-id: ""
key-name: ""
log-level: info
metrics-disabled: false
nats-max-payload-bytes: 0
nats-port: 4222
no-tasks: false
pprof-disabled: false
query-concurrency: 1024
query-initial-memory-bytes: 0
query-max-memory-bytes: 0
query-memory-bytes: 0
query-queue-size: 1024
reporting-disabled: false
secret-store: bolt
session-length: 1440
session-renew-disabled: false
sqlite-path: ""
storage-cache-max-memory-size: 1073741824
storage-cache-snapshot-memory-size: 26214400
storage-cache-snapshot-write-cold-duration: 10m0s
storage-compact-full-write-cold-duration: 4h0m0s
storage-compact-throughput-burst: 50331648
storage-max-concurrent-compactions: 0
storage-max-index-log-file-size: 1048576
storage-no-validate-field-size: false
storage-retention-check-interval: 30m0s
storage-series-file-max-concurrent-snapshot-compactions: 0
storage-series-id-set-cache-size: 0
storage-shard-precreator-advance-period: 30m0s
storage-shard-precreator-check-interval: 10m0s
storage-tsm-use-madv-willneed: false
storage-validate-keys: false
storage-wal-fsync-delay: 0s
storage-wal-max-concurrent-writes: 0
storage-wal-max-write-delay: 10m0s
storage-write-timeout: 10s
store: disk
testing-always-allow-setup: false
tls-cert: ""
tls-key: ""
tls-min-version: "1.2"
tls-strict-ciphers: false
tracing-type: ""
ui-disabled: false
vault-addr: ""
vault-cacert: ""
vault-capath: ""
vault-client-cert: ""
vault-client-key: ""
vault-client-timeout: 0s
vault-max-retries: 0
vault-skip-verify: false
vault-tls-server-name: ""
vault-token: ""

先看 docker-compose.yaml 這個設定檔，這個檔案能夠決定有多少 docker container 需要啟動，以及啟動他們的必要參數，如 container 名稱，連接埠號等。以下我們關注 otel-collector (collector) 以及 InfluxDB (influxdb)。

• collector
  • container_name, hostname 都設成 collector
  • volume: 將本機上的 ./collector-config local.yaml 檔案掛載到 container 內部的 /etc/collector-config.yaml
  • ports: otel-collector 預設埠號為 4317，因此將本機的 4317 port 映射到 container 內部的 4317 port
• influxdb
  • container_name, hostname 都設成 influxdb
  • volume: 將本機上 InfluxDB 設定檔 ./o11y-backend/influxdb-config.yaml 檔案掛載到 container 內部的 /etc/influxdb2/config.
  • 另外一個掛載的目錄為 ./influxdb2:/var/lib/influxdb2，這樣能夠把 InfluxDB 的設定以及資料庫都保存到本機上。
  • ports: InfluxDB 預設埠號為 8086，因此將本機的 8086 port 映射到 container 內部的 8086 port

docker-compose.yaml

version: '3.0'

services:
  collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: collector
    hostname: collector
    depends_on:
      prometheus:
        condition: service_healthy
    command: [ "--config=/etc/collector-config.yaml" ]
    volumes:
      - ./collector-config-local.yaml:/etc/collector-config.yaml
    ports:
      - "4317:4317"
      - "6666:6666"

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    hostname: prometheus
    command:
      - --config.file=/etc/prometheus.yaml
      - --web.enable-remote-write-receiver
      - --enable-feature=exemplar-storage
    volumes:
      - ./o11y-backend/prometheus.yaml:/etc/prometheus.yaml
      - ./prometheus:/prometheus
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "9090:9090"
      # - "8081:8080"
    healthcheck:
      interval: 5s
      retries: 10
      test: wget --no-verbose --tries=1 --spider http://localhost:9090/status || exit 1

  influxdb:
    image: influxdb:latest
    container_name: influxdb
    hostname: influxdb
    volumes:
      - ./o11y-backend/influxdb-config.yaml:/etc/influxdb2/config.yml
      - ./influxdb2:/var/lib/influxdb2
    ports:
      - "8086:8086"

networks:
  default:
    name: inst-java-apps-using-otel

接下來看 otel-collector 的設定檔，跟之前一樣把所有必要資訊都填進去，這邊要注意的是 influxDB 的 endpoint 改成 http://influxdb:8086，因為我們剛剛有設定執行 InfluxDB 的 container 的名稱。

另外，在還沒有開始執行前，因為還沒設定 InfluxDB，所以一開始還不會知道 org, bucket，token 這三個資訊，因此，我們需要執行一次 docker compose，進到 InfluxDB 裡面做第一次設定，把這些資訊都記錄下來，順便測試必要的工具都能啟動。

同時，因為我們把 InfluxDB 的狀態都紀錄到本機，所以不用擔心重新啟動 docker compose 之後資料會不見，第二次啟動之後也不需要再到 InfluxDB 裡面再做設定。

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 20s
    send_batch_size: 8192

exporters:
  prometheus:
    endpoint: collector:6666
    namespace: default
  influxdb:
    endpoint: "http://influxdb:8086"
    timeout: 500ms
    org: my-org
    bucket: my-bucket
    token: tFGkjYo3fgfhnroaLx_vug8c95fM3Plgn4gayZzgwD34Rxhwp2po5BriKA8h-z_ceqY-f0jWqMUDgPknJNzobg==

service:
  pipelines:
    # metrics:
    #   receivers: [otlp]
    #   processors: [batch]
    #   exporters: [prometheus]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [influxdb]
  telemetry:
    logs:
      level: debug

接下來試著用 docker compose 啟動所有工具。

❯ docker compose -f docker-compose.yaml up --build
WARN[0000] /home/st9540808/Desktop/VSCode/instrumenting-java-apps-using-opentelemetry-modified/docker-compose.yaml: `version` is obsolete 
[+] Running 3/0
 ✔ Container prometheus  Created                                                                                 0.0s 
 ✔ Container collector   Created                                                                                 0.0s 
 ✔ Container influxdb    Created                                                                                 0.0s 
Attaching to collector, influxdb, prometheus

到了這邊所有工具就都啟動起來了，啟動之後我們同樣用我們的效能工具 (ddshark) 丟出幾個測試用的 OpenTelemetry trace，看看是否運作正常。可以看到 Query 得到的資料跟一開始單獨測試時一樣。

小技巧

以下為操作 InfluxDB 的幾個實用命令，如何用 docker 控制跑在 container 裡的 InfluxDB

刪除一個 bucket

docker exec influxdb influx bucket delete -n my-bucket -o my-org -t tFGkjYo3fgfhnroaLx_vug8c95fM3Plgn4gayZzgwD34Rxhwp2po5BriKA8h-z_ceqY-f0jWqMUDgPknJNzobg==

建立一個 bucket

docker exec influxdb influx bucket create -n my-bucket -o my-org -t tFGkjYo3fgfhnroaLx_vug8c95fM3Plgn4gayZzgwD34Rxhwp2po5BriKA8h-z_ceqY-f0jWqMUDgPknJNzobg==

刪除 bucket 內某個特定範圍的資料，注意這類需要查詢的刪除會耗費大量時間

docker exec influxdb influx delete --org my-org --bucket my-bucket --token tdb5m8gTHvK1aY-GBnCEFNtCo0xhxNanw0Wv_bcyLMyhBypwWHtqVS5TzaO2PljhEgnIG9wWM1cL9a28wXoT8w==  --start 2023-01-01T00:00:00Z --stop 2023-07-31T00:00:00Z  --predicate '_measurement="spans"'

0x03 InfluxDB 查詢語法

我們已經架起 InfluxDB，而且 Opentelemetry trace 已經存到資料庫裡面了，現在只剩下寫 Query 撈出我們想要的資料。如果是要用 Grafana 視覺化資料，學習 InfluxDB 查詢語法也是必須要的。

我目前使用的 InfluxDB 版本是 v2，使用的是 InfluxQL 的查詢語法，未來當 v3 開源之後，就會重新使用 SQL 的語法，但 v3 開源的時間未知，因此目前用 InfluxDB v2 必須學會使用 InfluxQL。

重要工具

map

https://docs.influxdata.com/flux/v0/stdlib/universe/map/

Output tables are the result of applying the map function (fn) to each record of the input tables.

例如以下範例用來處理 ddshark 的效能資料，因為我的資料存在 attribute 裡面，opentelemtry 會把 attribute 包裝成 json 格式給你，因此用以下 fn 將需要的 attribute 取出。

data
    |> map(
        fn: (r) => {
            jsonData = json.parse(data: bytes(v: r._value))

            return {
                _measurement: r._measurement,
                _time: r._time,
                _field: r._field,
                _value: jsonData.payload_size,
                topic_name: jsonData.topic_name,
                sn: jsonData.sn,
                writer_id: jsonData.writer_id,
                fragment_starting_num: jsonData.fragment_starting_num,
            }
        },
    )

elapsed

https://docs.influxdata.com/flux/v0/stdlib/universe/elapsed/

elapsed() returns the time between subsequent records.

For each input table, elapsed() returns the same table without the first row (because there is no previous time to derive the elapsed time from) and an additional column containing the elapsed time.

應用以上

import "experimental/json"

data = from(bucket: "my-bucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "spans")
  |> filter(fn: (r) => r["_field"] == "attributes")
  |> filter(fn: (r) => r["service.name"] == "dds.traffic")

groupedData = data
    |> map(
        fn: (r) => {
            jsonData = json.parse(data: bytes(v: r._value))

            return {
                _measurement: r._measurement,
                _time: r._time,
                _field: r._field,
                _value: jsonData.payload_size,
                host_name: jsonData["host.name"],
                topic_name: jsonData.topic_name,
                sn: jsonData.sn,
                writer_id: jsonData.writer_id,
                fragment_starting_num: jsonData.fragment_starting_num,
            }
        },
    )
    |> sort(columns: ["_time"])
    |> group(columns: ["topic_name"])
    |> filter(fn: (r) => r.topic_name == "rt/topic2")
    |> group(columns: ["sn", "fragment_starting_num", "writer_id"])
    |> yield()
    

注意

    |> group(columns: ["sn", "fragment_starting_num", "writer_id"])

    |> group(columns: ["sn", "writer_id"])

Backup Text

❯ docker run --rm -p 8086:8086 --name influxdb influxdb:latest
2023-06-19T14:17:27.285295308Z  warn    boltdb not found at configured path, but DOCKER_INFLUXDB_INIT_MODE not specified, skipping setup wrapper      {"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt"}
2023-06-19T14:17:27.301633682Z  warn    boltdb not found at configured path, but DOCKER_INFLUXDB_INIT_MODE not specified, skipping setup wrapper      {"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt"}
ts=2023-06-19T14:17:27.355530Z lvl=info msg="Welcome to InfluxDB" log_id=0iX07jEl000 version=v2.7.1 commit=407fa622e9 build_date=2023-04-28T13:24:27Z log_level=info
ts=2023-06-19T14:17:27.355551Z lvl=warn msg="nats-port argument is deprecated and unused" log_id=0iX07jEl000
ts=2023-06-19T14:17:27.358732Z lvl=info msg="Resources opened" log_id=0iX07jEl000 service=bolt path=/var/lib/influxdb2/influxd.bolt
ts=2023-06-19T14:17:27.358807Z lvl=info msg="Resources opened" log_id=0iX07jEl000 service=sqlite path=/var/lib/influxdb2/influxd.sqlite
ts=2023-06-19T14:17:27.359453Z lvl=info msg="Bringing up metadata migrations" log_id=0iX07jEl000 service="KV migrations" migration_count=20
ts=2023-06-19T14:17:27.428741Z lvl=info msg="Bringing up metadata migrations" log_id=0iX07jEl000 service="SQL migrations" migration_count=8
ts=2023-06-19T14:17:27.446524Z lvl=info msg="Using data dir" log_id=0iX07jEl000 service=storage-engine service=store path=/var/lib/influxdb2/engine/data
ts=2023-06-19T14:17:27.446548Z lvl=info msg="Compaction settings" log_id=0iX07jEl000 service=storage-engine service=store max_concurrent_compactions=16 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2023-06-19T14:17:27.446552Z lvl=info msg="Open store (start)" log_id=0iX07jEl000 service=storage-engine service=store op_name=tsdb_open op_event=start
ts=2023-06-19T14:17:27.446575Z lvl=info msg="Open store (end)" log_id=0iX07jEl000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=0.024ms
ts=2023-06-19T14:17:27.446584Z lvl=info msg="Starting retention policy enforcement service" log_id=0iX07jEl000 service=retention check_interval=30m
ts=2023-06-19T14:17:27.446590Z lvl=info msg="Starting precreation service" log_id=0iX07jEl000 service=shard-precreation check_interval=10m advance_period=30m
ts=2023-06-19T14:17:27.446825Z lvl=info msg="Starting query controller" log_id=0iX07jEl000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
ts=2023-06-19T14:17:27.447841Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0iX07jEl000 max_select_point=0 max_select_series=0 max_select_buckets=0
ts=2023-06-19T14:17:27.451127Z lvl=info msg=Starting log_id=0iX07jEl000 service=telemetry interval=8h
ts=2023-06-19T14:17:27.451144Z lvl=info msg=Listening log_id=0iX07jEl000 service=tcp-listener transport=http addr=:8086 port=8086
ts=2023-06-19T14:21:51.388812Z lvl=info msg=Unauthorized log_id=0iX07jEl000 error="token required"
ts=2023-06-19T14:22:58.916436Z lvl=info msg="index opened with 8 partitions" log_id=0iX07jEl000 service=storage-engine index=tsi
ts=2023-06-19T14:22:58.916595Z lvl=info msg="loading changes (start)" log_id=0iX07jEl000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
ts=2023-06-19T14:22:58.916619Z lvl=info msg="loading changes (end)" log_id=0iX07jEl000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.026ms
ts=2023-06-19T14:22:58.917005Z lvl=info msg="Reindexing TSM data" log_id=0iX07jEl000 service=storage-engine engine=tsm1 db_shard_id=1
ts=2023-06-19T14:22:58.917010Z lvl=info msg="Reindexing WAL data" log_id=0iX07jEl000 service=storage-engine engine=tsm1 db_shard_id=1
ts=2023-06-19T14:22:58.920440Z lvl=info msg="saving field index changes (start)" log_id=0iX07jEl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=start
ts=2023-06-19T14:22:58.921172Z lvl=info msg="saving field index changes (end)" log_id=0iX07jEl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=end op_elapsed=0.739ms