# VictoriaMetrics
## Ways to send metrics to VictoriaMetrics for multi-tenancy setup
### Option #1: Using Remote Write URL
Example metrics data:
```
# HELP dummy_bar dummy counter
# TYPE dummy_bar counter
dummy_bar{id="0",l1="one",l2="aaa"} 3
dummy_bar{id="1",l1="two",l2="aaa"} 3
dummy_bar{id="2",l1="three",l2="aaa"} 3
dummy_bar{id="3",l1="one",l2="aaa"} 3
dummy_bar{id="4",l1="two",l2="aaa"} 3
dummy_bar{id="5",l1="three",l2="aaa"} 3
dummy_bar{id="6",l1="one",l2="aaa"} 3
dummy_bar{id="7",l1="two",l2="aaa"} 3
dummy_bar{id="8",l1="three",l2="aaa"} 3
dummy_bar{id="9",l1="one",l2="aaa"} 3
```
By setting...
```yaml
# https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-agent/values.yaml
remoteWriteUrls:
- http://victoria-metrics-victoria-metrics-cluster-vminsert:8480/insert/2/prometheus
# The rest of the config ...
config:
global:
scrape_interval: 10s
scrape_configs:
- job_name: dummy-exporter
scrape_interval: 10s
static_configs:
- targets: ["dummy-exporter.dummy-exporter.svc.cluster.local:80"]
```
...where `2` in the `remoteWriteUrls` is the tenant ID
Pod manifest:
```yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
checksum/config: e8d02f03e02dd9e49c3d6e3bdcfd146b29c13b1d2bf7431a4007a2729a0b0895
labels:
app.kubernetes.io/instance: vmagent
app.kubernetes.io/name: victoria-metrics-agent
pod-template-hash: 7b9c96c864
name: vmagent-victoria-metrics-agent-7b9c96c864-lsd7m
namespace: observability
spec:
containers:
- args:
- -promscrape.config=/config/scrape.yml
- -remoteWrite.tmpDataPath=/tmpData
- -remoteWrite.url=http://victoria-metrics-victoria-metrics-cluster-vminsert:8480/insert/2/prometheus
- -envflag.enable=true
- -envflag.prefix=VM_
- -loggerFormat=json
image: victoriametrics/vmagent:v1.83.0
livenessProbe:
initialDelaySeconds: 5
periodSeconds: 15
tcpSocket:
port: http
timeoutSeconds: 5
name: victoria-metrics-agent
ports:
- containerPort: 8429
name: http
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 5
periodSeconds: 15
volumeMounts:
- mountPath: /tmpData
name: tmpdata
- mountPath: /config
name: config
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-7plsg
readOnly: true
workingDir: /
preemptionPolicy: PreemptLowerPriority
priority: 0
serviceAccountName: vmagent-victoria-metrics-agent
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: tmpdata
- configMap:
name: vmagent-victoria-metrics-agent-config
name: config
- name: kube-api-access-7plsg
projected:
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
fieldPath: metadata.namespace
path: namespace
```
Data in `vmagent-victoria-metrics-agent-config` ConfigMap:
```
# cat /config/scrape.yml
global:
scrape_interval: 10s
scrape_configs:
- job_name: dummy-exporter
scrape_interval: 10s
static_configs:
- targets:
- dummy-exporter.dummy-exporter.svc.cluster.local:80
```
---
### Option #2: Using Multi Tenant URL
Example metrics data:
```
# HELP dummy_bar dummy counter
# TYPE dummy_bar counter
dummy_bar{id="0",l1="one",l2="aaa"} 3
dummy_bar{id="1",l1="two",l2="aaa"} 3
dummy_bar{id="2",l1="three",l2="aaa"} 3
dummy_bar{id="3",l1="one",l2="aaa"} 3
dummy_bar{id="4",l1="two",l2="aaa"} 3
dummy_bar{id="5",l1="three",l2="aaa"} 3
dummy_bar{id="6",l1="one",l2="aaa"} 3
dummy_bar{id="7",l1="two",l2="aaa"} 3
dummy_bar{id="8",l1="three",l2="aaa"} 3
dummy_bar{id="9",l1="one",l2="aaa"} 3
```
By setting...
```yaml
# https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-agent/values.yaml
multiTenantUrls:
- http://victoria-metrics-victoria-metrics-cluster-vminsert:8480
# The rest of the config ...
config:
global:
scrape_interval: 10s
scrape_configs:
- job_name: dummy-exporter
scrape_interval: 10s
static_configs:
- targets: ["dummy-exporter.dummy-exporter.svc.cluster.local:80"]
relabel_configs:
- source_labels: [__address__]
target_label: __tenant_id__
replacement: "1" # tenant ID
```
Pod manifest:
```yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
checksum/config: 560092ef2eef37de8b945ecd46a2129088796e64b2ba0edba1cfa46b3b54148b
labels:
app.kubernetes.io/instance: vmagent
app.kubernetes.io/name: victoria-metrics-agent
pod-template-hash: 56995d9c5b
name: vmagent-victoria-metrics-agent-56995d9c5b-f5zfs
namespace: observability
spec:
containers:
- args:
- -promscrape.config=/config/scrape.yml
- -remoteWrite.tmpDataPath=/tmpData
- -remoteWrite.multitenantURL=http://victoria-metrics-victoria-metrics-cluster-vminsert:8480
- -envflag.enable=true
- -envflag.prefix=VM_
- -loggerFormat=json
image: victoriametrics/vmagent:v1.83.0
livenessProbe:
initialDelaySeconds: 5
periodSeconds: 15
tcpSocket:
port: http
timeoutSeconds: 5
name: victoria-metrics-agent
ports:
- containerPort: 8429
name: http
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 5
periodSeconds: 15
volumeMounts:
- mountPath: /tmpData
name: tmpdata
- mountPath: /config
name: config
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-d6tk2
readOnly: true
workingDir: /
preemptionPolicy: PreemptLowerPriority
priority: 0
serviceAccountName: vmagent-victoria-metrics-agent
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: tmpdata
- configMap:
name: vmagent-victoria-metrics-agent-config
name: config
- name: kube-api-access-d6tk2
projected:
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
fieldPath: metadata.namespace
path: namespace
```
Data in `vmagent-victoria-metrics-agent-config` ConfigMap:
```
# cat config/scrape.yml
global:
scrape_interval: 10s
scrape_configs:
- job_name: dummy-exporter
relabel_configs:
- replacement: "1"
source_labels:
- __address__
target_label: __tenant_id__
scrape_interval: 10s
static_configs:
- targets:
- dummy-exporter.dummy-exporter.svc.cluster.local:80
```
---
### Option #3: Using `vm_account_id`
Example metrics data:
```
# HELP dummy_bar dummy gauge
# TYPE dummy_bar gauge
dummy_bar{id="0",vm_account_id="1"} 0.10119738869055629
dummy_bar{id="1",vm_account_id="1"} 0.7877132336827702
dummy_bar{id="2",vm_account_id="1"} 0.8482502886262785
dummy_bar{id="3",vm_account_id="1"} 0.7474721768346807
dummy_bar{id="4",vm_account_id="1"} 0.7006719919391945
```
By setting...
```yaml
# https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-agent/values.yaml
remoteWriteUrls:
- http://victoria-metrics-victoria-metrics-cluster-vminsert:8480/insert/multitenant/prometheus/api/v1/write
# The rest of the config ...
config:
global:
scrape_interval: 10s
scrape_configs:
- job_name: dummy-exporter
scrape_interval: 10s
static_configs:
- targets: ["dummy-exporter.dummy-exporter.svc.cluster.local:80"]
```
Pod manifest:
```yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
checksum/config: e8d02f03e02dd9e49c3d6e3bdcfd146b29c13b1d2bf7431a4007a2729a0b0895
labels:
app.kubernetes.io/instance: vmagent
app.kubernetes.io/name: victoria-metrics-agent
pod-template-hash: 846ccc9bc4
name: vmagent-victoria-metrics-agent-846ccc9bc4-9j8pr
namespace: observability
spec:
containers:
- args:
- -promscrape.config=/config/scrape.yml
- -remoteWrite.tmpDataPath=/tmpData
- -remoteWrite.url=http://victoria-metrics-victoria-metrics-cluster-vminsert:8480/insert/multitenant/prometheus/api/v1/write
- -envflag.enable=true
- -envflag.prefix=VM_
- -loggerFormat=json
image: victoriametrics/vmagent:v1.83.0
livenessProbe:
initialDelaySeconds: 5
periodSeconds: 15
tcpSocket:
port: http
timeoutSeconds: 5
name: victoria-metrics-agent
ports:
- containerPort: 8429
name: http
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 5
periodSeconds: 15
volumeMounts:
- mountPath: /tmpData
name: tmpdata
- mountPath: /config
name: config
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-sn9x4
readOnly: true
workingDir: /
preemptionPolicy: PreemptLowerPriority
priority: 0
serviceAccountName: vmagent-victoria-metrics-agent
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: tmpdata
- configMap:
name: vmagent-victoria-metrics-agent-config
name: config
- name: kube-api-access-sn9x4
projected:
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
fieldPath: metadata.namespace
path: namespace
```
Data in `vmagent-victoria-metrics-agent-config` ConfigMap:
```
# cat config/scrape.yml
global:
scrape_interval: 10s
scrape_configs:
- job_name: dummy-exporter
scrape_interval: 10s
static_configs:
- targets:
- dummy-exporter.dummy-exporter.svc.cluster.local:80
```
Sign in with Wallet
Connect another wallet