# Loki Setup for httplog

three components:
- promtail: it forwards the log to loki
- loki
- grafana

first, we need to extract the httplog entries from the kube-apiserver log file.

## Prepare httplog

extract all httplog entries from kube-apiserver log into a file:

cat kube-apiserver.log | grep 'httplog.go' > httplog.log

httplog entry does not conform to a `logfmt` format, so i had to do some `sed` magic:

cat httplog.log | \
sed -E 's/^I//; s/([0-9]{2})([0-9]{2}) ([0-9]{2}:[0-9]{2}:[0-9]{2}).([0-9]{6})(.*"HTTP")/ts="\1\2 \3.\4"/'\
> httplog.logfmt

httplog entries are out of order, so sort the file by log timestamp

sort -k 1 httplog.log > http.log
rm httplog.log

## Prepare audit
collect audit logs from all master nodes and cat to one file

oc adm node-logs {server url} --path=/kube-apiserver/audit.log > audit-0.log
cat audit-0.log audit-1.log audit-2.log > audit.log

for file in *; do cat $file >> audit.log; done

// 
by received timestamp
cat audit.log | jq '[.] | sort_by(.requestReceivedTimestamp) | .[]' | jq -c . > audit-sorted.log

jq -sc 'sort_by(.requestReceivedTimestamp)|.[]' audit.jq

## Loki Setup
i prepared a custom image for `loki` since i wanted to have a custom value for `max_entries_limit_per_query`. the loki query i have retuns more than `5000` lines.

extend the official docker image with your own:

FROM grafana/loki:2.3.0

COPY my-loki-local-config.yaml /etc/loki/my-local-config.yaml

ENTRYPOINT [ "/usr/bin/loki" ]
CMD ["-config.file=/etc/loki/my-local-config.yaml"]

`my-loki-local-config.yaml` is a copy of the official with the following change:

limits_config:
  max_entries_limit_per_query: 0

build the loki image

docker build -t tohinkashem/myloki:dev -f Dockerfile .

## PromTail Setup

This is the promtail config file to ingest the httplog entries

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: httplog
  static_configs:
  - targets:
    - localhost
    labels:
      job: httplogs
      __path__: /httplog/*log
  pipeline_stages:
  - regex:
      expression: '^ts="(?P<arrivaltime>\d{4} \d{2}:\d{2}:\d{2}\.\d{6})".*$'
  - timestamp:
      source: arrivaltime
      format: '0102 15:04:05.000000'
      
- job_name: audit
  static_configs:
  - targets:
    - localhost
    labels:
      job: audit
      __path__: /audit/*log
  pipeline_stages:
  - json:
      expressions:
        timestamp: requestReceivedTimestamp
  - timestamp:
      source: timestamp
      format: '2006-01-02T15:04:05.999999Z'

note that i am using `pipeline_stages` to read the timestamp from the httplog entry, otherwise promtail will assign the timestamp when it forwards the log line.

we also need to create my own promtail image

FROM grafana/promtail:2.3.0

COPY promtail-docker-config.yaml /etc/promtail/my-config.yml

ENTRYPOINT ["/usr/bin/promtail"]
CMD ["-config.file=/etc/promtail/my-config.yml"]

build promtail docker image:

docker build -t tohinkashem/mypromtail:dev -f Dockerfile .

## Run Grafana/Loki/PromTail

I used the following docker-compose file to run these components:

version: "3"

networks:
  loki:

services:
  loki:
    image: tohinkashem/myloki:dev
    restart: always
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/my-local-config.yaml
    networks:
      - loki

  promtail:
    image: tohinkashem/mypromtail:dev
    restart: always
    volumes:
      - /home/akashem/httplog:/httplog
    command: --inspect -config.file=/etc/promtail/my-config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_DISABLE_LOGIN_FORM=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    ports:
      - "3000:3000"
    networks:
      - loki

`home/akashem/httplog` is the directory where i stored the processed httplog file.

- run the suite `docker-compose up`.
- open a browser window, type in `http://localhost:3000` to open grafana.
- go to settings, and create a new data soure for `Loki`, provide the url `http://loki:3100`
- go to explore and choose Loki data source

## Loki Query

$ curl -v -o output.log "http://localhost:3000/api/datasources/proxy/1/loki/api/v1/query_range?direction=BACKWARD&limit=1000000&start=1634817600000000000&end=1634842800000000000" --data-urlencode 'query={job="httplogs"} | logfmt | verb="DELETE" and apf_f_seatss > 10'


cat output.log | jq '.data.result | .[]' | jq -r '.stream.apf_f_seatss + " " + .stream.apf_pl + " " + .stream.apf_fs + " " + .stream.verb +  " " + .stream.URI + " " + .stream.userAgent' | sort | uniq -c > delete.txt