---
# System prepended metadata

title: Loki

---

# Loki

[Official LogQL Guide](https://grafana.com/docs/loki/latest/logql/)

## Finding the cluster

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade/1367068241817505792
Check `ipi-install-hosted-loki` test contents

`{invoker="openshift-internal-ci/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade/1367068241817505792"} | unpack`

### Filtering by namespace
`{...} | unpack | namespace="openshift-cluster-version"`

### Filtering out throttling messages and leader election
`{...} | unpack | namespace="openshift-cluster-version" !~ "Throttling" !~ "leaderelection.go"`

### Filtering by "Sync"
`{..} | unpack | namespace="openshift-cluster-version" |~ "Running sync"`

### Parsing sync messages
`{... } | unpack | namespace="openshift-cluster-version" |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\""`

for line
```
I0303 05:30:59.744987       1 sync_worker.go:765] Running sync for rolebinding "kube-system/insights-operator-auth" (459 of 668)
```
gives additional labels:


| Label     | Value    |
| --------  | -------- |
| s_msgtype | I0303    |
| s_ts | 05:30:59.744987    |
| s_line | sync_worker.go:765    |
| s_objtype | rolebinding    |
| s_objname | kube-system/insights-operator-auth    |

This allows us to see the breakdown of objects we sync:
| s_objtype | Encountered | Percentage | 
| --------  | --------    | --------   |
| customresourcedefinition  | 131    | 13%   |
| rolebinding  | 95    | 10%   |
| role  | 90    | 9%   |
| clusterrole  | 71    | 7%   |
| configmap  | 64    | 6%   |

### Additional filtering:
```
{... } | unpack | namespace="openshift-cluster-version" |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\"" | s_objtype = "rolebinding"
```

Print just the object name:
```
{...} | unpack | namespace="openshift-cluster-version" |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\"" | s_objtype="rolebinding" | line_format "{{.s_objname}}"
```

## Metrics

```
count_over_time({...} | unpack | namespace="openshift-cluster-version" |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\"" | s_objtype="rolebinding"[5m])
```
shows amount of rolebindings we sync per 5m

```
sum by (s_objtype) (count_over_time({invoker="openshift-internal-ci/periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade/1366956172711563264",namespace="openshift-cluster-version"} |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\"" [5m]))
```

shows object type sums.

### Throttling histogram
```
{...} | unpack | namespace="openshift-cluster-version" |~ "Throttling request" | regexp "took (?P<s_ts>.*), "
```
sets throttling time label

95% quantile of throttling
```
quantile_over_time(0.99,
{...} | unpack | namespace="openshift-cluster-version" |~ "Throttling request" |regexp "took (?P<s_ts>.*), " | unwrap duration_seconds(s_ts)[1m])
```
(time: 12:00 -> 14:00)
