# Main Record of Alerting As A Service

* **Date:** 2022-12-09
* **Status:** proposed
* **Deciders:** Enes Canca
* **Consulted:** Team Beholder
* **Informed:** Team Beholder

# Architecture Design Record

## Context and Problem Statement

We are currently using VictoriaMetrics for long-term storage of metrics and time-series data. VictoriaMetrics is a high-performance time-series database that has performed well over the last year, but it is no longer able to meet the needs of our rapidly growing scale. As the volume and velocity of data increases, VictoriaMetrics is becoming unreliable and is unable to provide the level of availability and performance that we require.

To address this problem, we need a scalable and cost-effective storage solution that can handle large amounts of time-series data with low latencies and high availability. The solution should be based on Prometheus, as we already use Prometheus for monitoring and alerting, and it will be easier to integrate with our existing systems.

One of the main challenges with VictoriaMetrics is the amount of memory and disk space required by the `vm-storage` units. VictoriaMetrics is designed to be highly efficient in terms of disk space usage, but it still requires a significant amount of memory and disk space to store and index large amounts of data. For example, to store 1 year of data, we currently use 70 `vm-storage` units, each with 32 CPU cores, 128GB of memory, and 4TB of disk space. This is a huge scale, and the cost of maintaining this setup is becoming unsustainable.

In addition to the resource usage, there are also issues with redundancy and cost. VictoriaMetrics does not provide built-in support for redundancy or data replication, which means that we have to implement these features manually. This adds complexity and increases the cost of maintaining the system.

Given these challenges, we need to find a storage solution that is more scalable, cost-effective, and redundant than VictoriaMetrics. The best candidate for this is Grafana Mimir, which is a Prometheus-based storage system that uses object storage for long-term data retention. Grafana Mimir is part of the Grafana ecosystem, which we already use for monitoring and alerting, so it will be easier to integrate with our existing systems.

## Decision Drivers

The decision drivers for replacing VictoriaMetrics with Grafana Mimir for long-term storage are:

* Scalability: VictoriaMetrics is no longer able to meet the needs of our rapidly growing scale. Grafana Mimir uses object storage, which is more scalable than the block storage used by VictoriaMetrics. This will enable us to store and analyze large amounts of time-series data more efficiently and cost-effectively.

* Cost-effectiveness: The cost of maintaining VictoriaMetrics is becoming unsustainable.Grafana Mimir uses object storage, which is much cheaper than the block storage used by VictoriaMetrics. This will enable us to reduce the overall cost of storing and analyzing time-series data.

* Reliability and availability: VictoriaMetrics does not provide built-in support for redundancy or data replication. Grafana Mimir uses object storage and provides built-in support for redundancy and data replication. This will improve the reliability and availability of the storage system, and it will reduce the complexity and cost of maintaining the system.

* Integration with existing systems: We already use Grafana for monitoring and alerting, and Grafana Mimir is part of the Grafana ecosystem. This will make it easier to integrate Grafana Mimir with our existing systems and to maintain a consistent monitoring and observability stack.

## Considered Options

* VictoriaMetrics: VictoriaMetrics is a fast, scalable, and cost-effective time-series database that uses block storage for data storage. However, it does not have built-in support for redundancy or data replication, which can affect the reliability and availability of the storage system.

* Grafana Mimir: Grafana Mimir is a Prometheus-based storage system that uses object storage for data storage. It provides built-in support for redundancy and data replication, which improves the reliability and availability of the storage system. However, it has higher data access latencies than VictoriaMetrics, which can be an issue for applications that require fast access to data.

* InfluxDB: InfluxDB is a popular open-source time-series database that has good scalability and reliability. However, it can be more expensive than VictoriaMetrics and Grafana Mimir due to its use of block storage, and it requires more complex maintenance and operations compared to Grafana Mimir.

* TimescaleDB: TimescaleDB is an open-source time-series database that is built on top of PostgreSQL. It has good scalability and reliability, and it uses block storage for data storage. However, it can be more expensive than VictoriaMetrics and Grafana Mimir, and it requires more complex maintenance and operations compared to Grafana Mimir.

* Cortex: Cortex is an open-source, horizontally scalable, multi-tenant time-series database that is built on top of Prometheus. It uses object storage for data storage, which provides good scalability and cost-effectiveness. However, it does not have built-in support for redundancy or data replication, which can affect the reliability and availability of the storage system, and it requires more complex maintenance and operations compared to Grafana Mimir.

## Decision Outcome

After considering the options and conducting performance tests, we have decided to use Grafana Mimir for long-term storage. This decision is based on the following factors:

* Grafana Mimir provides better scalability and cost-effectiveness than VictoriaMetrics. We can store more data at a lower cost, and we can easily scale the system as the volume and velocity of data increases.
* Grafana Mimir has built-in support for redundancy and data replication. This will improve the reliability and availability of the storage system, and it will reduce the complexity and cost of maintaining the system.
* While Grafana Mimir may have higher latencies for data access than VictoriaMetrics, it is still suitable for most use cases in the field of monitoring and observability.

In summary, we believe that switching to Grafana Mimir for long-term storage will provide significant benefits in terms of scalability, cost-effectiveness, and reliability. We will implement this solution as soon as possible to improve the performance and reliability of our storage system.

### Proposed Solution

The proposed solution is to replace VictoriaMetrics with Grafana Mimir for long-term storage of metrics and time-series data. Grafana Mimir is a Prometheus-based storage system that uses object storage for data retention, and it provides built-in support for redundancy and data replication. This will address the issues with scalability, cost, and redundancy that we are currently experiencing with VictoriaMetrics.

## Architecture


## Benefits and Drawbacks

The main benefit of using Grafana Mimir for long-term storage is the improved scalability and cost-effectiveness of the system. Grafana Mimir uses object storage, which is much cheaper than the block storage used by VictoriaMetrics. This means that we can store more data at a lower cost, and we can scale the system more easily as the volume and velocity of data increases.

Another benefit of using Grafana Mimir is the built-in support for redundancy and data replication. This will improve the reliability and availability of the storage system, and it will reduce the complexity and cost of maintaining the system.

One potential drawback of using Grafana Mimir is the higher latencies for data access compared to VictoriaMetrics. Grafana Mimir uses object storage, which has higher latencies for data access than block storage. This may make Grafana Mimir less suitable for applications that require fast access to data, but it is still acceptable for most use cases in the field of monitoring and observability.

Overall, the proposed solution of replacing VictoriaMetrics with Grafana Mimir for long-term storage offers significant benefits in terms of scalability, cost-effectiveness, and reliability. While there may be some trade-offs in terms of data access latencies, the overall performance and reliability of the storage system will be improved. This will enable us to store and analyze large amounts of time-series data more efficiently and cost-effectively, and to support our rapidly growing scale.

The main drawbacks of choosing Grafana Mimir as our long-term storage solution are:

* Limited support for certain features: Grafana Mimir may not support some advanced features in VictoriaMetrics, such as custom retention policies or query functions. We will need to evaluate the compatibility of Grafana Mimir with our existing systems and make any necessary changes or adjustments.

* Performance limitations: Grafana Mimir may not be as fast or as efficient as other options, such as InfluxDB or Elasticsearch, in certain scenarios. We will need to monitor the performance of Grafana Mimir in our environment and make any necessary adjustments or optimizations to ensure that it meets our requirements for long-term storage.

### Positive Consequences

* Grafana Mimir uses object storage, which is much cheaper than the block storage used by VictoriaMetrics. This means that we can store more data at a lower cost, and we can scale the system more easily as the volume and velocity of data increases.

* Grafana Mimir has built-in support for redundancy and data replication. This will improve the reliability and availability of the storage system, and it will reduce the complexity and cost of maintaining the system.

### Negative Consequences

* Grafana Mimir uses object storage, which has higher latencies for data access than block storage. This may make Grafana Mimir less suitable for applications that require fast access to data.

* Grafana Mimir does not support the MetricsQL language used by Victoria Metrics, which means that some functions that we are currently using may not be available. However, this issue can be solved by using PromQL functions. PromQL is a language supported by Victoria Metrics and also used by Grafana Mimir, which allows us to access and visualize Victoria Metrics data through Mimir.