Note of Storage of Prometheus & thanos

tags: IT

The document might jump around topics. This is more like a record of investigating behaviors about how prometheus and thanos store data.

Thanos sidecar

By default, Thanos sidecar sends data to thanos storage every 2 hours
https://thanos.io/tip/components/sidecar.md/#sidecar

The default of 2h is recommended.

See following pic

The amount of data transferred depends on the data output of prometheus.
Prometheus is also set to flush every two hours

--storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h

We can find a proper memory usage based on how much data we save withing 2 hours.

Thanos downsampling

https://github.com/thanos-io/thanos/blob/main/docs/components/compact.md

​​​​Creating 5m downsampling for blocks older than 40 hours (2d)
​​​​Creating 1h downsampling for blocks older than 10 days (2w)

Thanos will not only save the original blocks, and will also save different resolutions.
It is divided into two, sampling every five minutes and sampling every hour.
Therefore, when calculating the storage space of thanos, it is necessary to add the space occupied by these two types of sampling. Otherwise the capacity won't be sufficient.

By default, Thanos will do 5m dowsampling every two days and downsampling every 14 days.
When doing downsampling, thanos create the dowsampling file and then delete the original file, so you can see the usage is ups and downs

storage.tsdb.min-block-duration

This parameter tells prometheus how long to keep the data in memory (default is 2hr).
If the parameter is too large, or prometheus manages too many targets, it may cause insufficient memory (because all the data is left in memory). If the parameters are set too small, prometheus will do read and write disk io often, which will affect performance
(because disk reading and writing will be slower, and some systems may use SSD to accelerate)

https://github.com/laszlocph/tsdbinfo

Calculate HDD usage

• https://www.robustperception.io/how-much-disk-space-do-prometheus-blocks-use/
• https://twgreatdaily.com/zh-hans/i_2hYG0BJleJMoPMUNRD.html
• https://www.zknow.top/2019/12/23/Prometheus_capacity/
• https://www.csdn.net/tags/NtTagg1sODYxNjEtYmxvZwO0O0OO0O0O.html
• https://www.hanktsai.com/2021/11/howcalculateprometheusneededspace.html

http_requests_total

• https://prometheus.io/docs/prometheus/latest/querying/examples/
• https://zhangguanzhang.github.io/2020/07/30/prometheus-rate-and-irate/
• https://timber.io/blog/promql-for-humans/#you-can-check-a-substring-using-regex-matching-
• https://yunlzheng.gitbook.io/prometheus-book/part-ii-prometheus-jin-jie/exporter/custom_exporter_with_java/custom_app_support_prometheus

Other Reference

• https://yunlzheng.gitbook.io/prometheus-book/part-ii-prometheus-jin-jie/readmd/prometheus-local-storage
• https://www.percona.com/blog/2018/09/20/prometheus-2-times-series-storage-performance-analyses/
• https://devops.stackexchange.com/questions/9298/how-to-calculate-disk-space-required-by-prometheus-v2-2
• https://source.coveo.com/2021/03/03/prometheus-memory/
• https://iximiuz.com/en/posts/prometheus-metrics-labels-time-series/
• http://www.vldb.org/pvldb/vol8/p1816-teller.pdf
• Writing a Time Series Database from Scratch (copy)
• https://nakabonne.dev/posts/write-tsdb-from-scratch/