# Loki

[Official LogQL Guide](https://grafana.com/docs/loki/latest/logql/)

## Finding the cluster

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade/1367068241817505792
Check `ipi-install-hosted-loki` test contents

`{invoker="openshift-internal-ci/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade/1367068241817505792"} | unpack`

### Filtering by namespace
`{...} | unpack | namespace="openshift-cluster-version"`

### Filtering out throttling messages and leader election
`{...} | unpack | namespace="openshift-cluster-version" !~ "Throttling" !~ "leaderelection.go"`

### Filtering by "Sync"
`{..} | unpack | namespace="openshift-cluster-version" |~ "Running sync"`

### Parsing sync messages
`{... } | unpack | namespace="openshift-cluster-version" |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\""`

for line

I0303 05:30:59.744987       1 sync_worker.go:765] Running sync for rolebinding "kube-system/insights-operator-auth" (459 of 668)

gives additional labels:

| Label | Value |
| -------- | -------- |
| s_msgtype | I0303 |
| s_ts | 05:30:59.744987 |
| s_line | sync_worker.go:765 |
| s_objtype | rolebinding |
| s_objname | kube-system/insights-operator-auth |

This allows us to see the breakdown of objects we sync:
| s_objtype | Encountered | Percentage |
| -------- | -------- | -------- |
| customresourcedefinition | 131 | 13% |
| rolebinding | 95 | 10% |
| role | 90 | 9% |
| clusterrole | 71 | 7% |
| configmap | 64 | 6% |

### Additional filtering:

{... } | unpack | namespace="openshift-cluster-version" |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\"" | s_objtype = "rolebinding"

Print just the object name:

{...} | unpack | namespace="openshift-cluster-version" |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\"" | s_objtype="rolebinding" | line_format "{{.s_objname}}"

## Metrics

count_over_time({...} | unpack | namespace="openshift-cluster-version" |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\"" | s_objtype="rolebinding"[5m])

shows amount of rolebindings we sync per 5m

sum by (s_objtype) (count_over_time({invoker="openshift-internal-ci/periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-aws-upgrade/1366956172711563264",namespace="openshift-cluster-version"} |~ "Running sync" | regexp "(?P<s_msgtype>\\w+) (?P<s_ts>.+) 1 (?P<s_line>.+)\\] Running sync for (?P<s_objtype>\\w+) \"(?P<s_objname>.*)\"" [5m]))

shows object type sums.

### Throttling histogram

{...} | unpack | namespace="openshift-cluster-version" |~ "Throttling request" | regexp "took (?P<s_ts>.*), "

sets throttling time label

95% quantile of throttling

quantile_over_time(0.99,
{...} | unpack | namespace="openshift-cluster-version" |~ "Throttling request" |regexp "took (?P<s_ts>.*), " | unwrap duration_seconds(s_ts)[1m])

(time: 12:00 -> 14:00)