# Grafana Agent: Checking Whether Grafana Agent Picked Up the PodMonitor and ServiceMonitor and Actually Scrapes Metrics
> This guide works with Prometheus ("Prometheus Agent" mode) and with the Grafana Agent Operator.
Imagine that you have installed the Grafana Agent Operator in the `grafana` namespace with Helm:
```bash
helm upgrade --install -n grafana --create-namespace grafana-agent-operator grafana/grafana-agent-operator
```
You have created a PodMonitor or a ServiceMonitor but nothing is showing in Grafana.
Unfortunately, the Grafana Agent Operator and the Grafana Agent don't show anything in the logs that will help you diagnose if your ServiceMonitor is working.
## Check 1: Has the Grafana Agent Operator loaded the PodMonitor or ServiceMonitor into a ConfigMap?
```bash
kubectl get secret -n grafana grafana-agent-config -ojson | jq '.data."agent.yml"' -r \
| base64 -d \
| grep job_name:
```
It should show the list of monitors that the Grafana Agent Operator has found and loaded into the ConfigMap:
```text
job_name: podMonitor/default/example/0
job_name: serviceMonitor/cert-manager/cert-manager/0
```
If your PodMonitor or ServiceMonitor doesn't show in the list:
1. Try restarting the Grafana Agent Operator:
```bash
kubectl rollout restart -n grafana deploy grafana-agent-operator
```
2. Check that the labels match in the MetricsInstance:
```bash
kubectl get metricsinstance -A -oyaml
```
For example, if you have:
```yaml
kind: MetricsInstance
spec:
podMonitorNamespaceSelector: {}
podMonitorSelector:
matchLabels:
instance: primary
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
matchLabels:
instance: primary
```
Then, look at your PodMonitors and ServiceMonitor and check that they have that label among their labels:
```bash
kubectl get servicemonitor,podmonitor -A -ojson | jq '.items[].metadata | {"name": .name, "namespace": .namespace, "labels": .labels}'
```
## Check 2: Has Grafana Agent Loaded the PodMonitor or ServiceMonitor?
The first thing you will want to do is to create a Service to access the Grafana Agent's API. Imagining that your agent is in the default namespace, the command will be:
```bash
kubectl expose pod grafana-agent-0
```
Then, query the Grafana Agent's API:
```bash
kubectl get --raw /api/v1/namespaces/grafana/services/grafana-agent-0:8080/proxy/agent/api/v1/metrics/targets \
| jq -r '.data[] | "\(.target_group)\t\(.state)\t\(.scrape_error)"' \
| column -t
```
That command will list each service monitor and show their state:
```
JOB NAME STATE
serviceMonitor/cert-manager/cert-manager/0 up
serviceMonitor/default/cadvisor-monitor/0 up
serviceMonitor/default/kubelet-monitor/0 up
serviceMonitor/cert-manager/cert-manager/0 up
serviceMonitor/default/foo/0 down Get "http://10.76.0.20:8080/metrics": dial tcp 10.76.0.20:8080: connect: connection refused
```
As you can see, the ServiceMonitor `foo` in the namespace `default` can't be reached. If the Service is also named `foo`, we can check whether the `/metrics` endpoint is reachable with the following command:
```bash
kubectl get --raw /api/v1/namespaces/default/services/foo:8080/proxy/metrics
```
That should help you debug the issue further.
I have found that when I forget to set the correct labels, the Grafana Agent Operator doesn't create the corresponding entry in the `grafana-agent-config`:
```bash
kubectl view-secret -n default grafana-agent-config
```
When a service monitor doesn't show in the Grafana Agent API:
1. The ServiceMonitor references a Service (not a Deployment, or a Pod), by labels and **by the port name** in the Service. This port name is optional in Kubernetes, but must be specified for the ServiceMonitor to work. It is not the same as the port name on the Pod or container, although it can be. (source: [Prometheus Troubleshooting](https://prometheus-operator.dev/docs/operator/troubleshooting/))
Sign in with Wallet
Connect another wallet