---
title: 'Resolved - Auth via Keycloak & traefik forwar-auth proxy eng'
disqus: hackmd
---

Auth via Keycloak & traefik forwar-auth proxy
===
---

[TOC]

## Resolved

resolved using the configuration from https://github.com/sleighzy/k3s-traefik-forward-auth-openid-connect

## Overview

If an application isn't able to authenticate users itself it should be done via OIDC applications.

## The proposed scheme
---

sequenceDiagram
    participant client
    participant proxy
    participant kc
    participant app

    client ->> app: 
    app ->> proxy: redirect via middleware if not auth-ed
    proxy ->> kc: ask login/pw
    kc ->> proxy: check if user login succesfully
    proxy ->> client: 
    client ->> app: 

    note over proxy: traefik forward-auth proxy
    note over kc: keycloak

## Resources

| component | url | details |
|:---------------------------:| ------------------------------------- | ------------------------------------------ |
| traefik forward-auth proxy | https://auth.team.stage.company.com/ | - |
| kc | keycloak.team.stage.company.com | realm - `internal-users`; client - `proxy` |
| app | https://myapp.team.stage.company.com/ | - |

### proxy manifests

<details>

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik-forward-auth
  namespace: stage
  labels:
    app: traefik-forward-auth
spec:
  selector:
    matchLabels:
      app: traefik-forward-auth
  template:
    metadata:
      labels:
        app: traefik-forward-auth
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: mesosphere/traefik-forward-auth:3.0.3
        name: traefik-forward-auth
        ports:
        - containerPort: 4181
          protocol: TCP
        env:
        - name: PROVIDER_URI
          value: https://keycloak.team.stage.company.com/auth/realms/internal-users
        - name: CLIENT_ID
          value: proxy
        - name: CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: traefik-forward-auth
              key: CLIENT_SECRET
        - name: SECRET
          valueFrom:
            secretKeyRef:
              name: traefik-forward-auth
              key: SECRET
        # experimental
        - name: AUTH_HOST
          value: "auth.team.stage.company.com"
        - name: COOKIE_DOMAIN
          value: "team.stage.company.com"
        - name: LOG_LEVEL
          value: trace
        # resources: {}

---

apiVersion: v1
kind: Service
metadata:
  name: traefik-forward-auth
  namespace: stage
  labels:
    app: traefik-forward-auth
spec:
  type: ClusterIP
  selector:
    app: traefik-forward-auth
  ports:
  - name: auth-http
    port: 4181
    targetPort: 4181

---

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-forward-auth
  namespace: stage
spec:
  forwardAuth:
    address: http://traefik-forward-auth.stage:4181
    trustForwardHeader: true
    authResponseHeaders:
      - X-Forwarded-User

---

apiVersion: v1
kind: Service
metadata:
  name: traefik-forward-auth-ui-external
  namespace: stage
  annotations:
    external-dns.alpha.kubernetes.io/hostname: auth.team.stage.company.team
spec:
  type: ExternalName
  externalName: lb-traefik.team.stage.company.team

---

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-forward-auth-http
  namespace: stage
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`auth.team.stage.company.com`)
      middlewares:
        - name: redirect-https
          namespace: traefik
      services:
        - kind: Service
          name: traefik-forward-auth
          port: 4181

---

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-forward-auth-https
  namespace: stage
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`auth.team.stage.company.com`)
      services:
        - kind: Service
          name: traefik-forward-auth
          port: 4181
  tls:
    secretName: wildcard-stage-team-company-com

</details>

### app manifests

<details>

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: stage
spec:
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: nginx:latest
        ports:
        - containerPort: 80

---

apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: stage
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 80

---

apiVersion: v1
kind: Service
metadata:
  name: myapp-ui-external
    namespace: stage
  annotations:
    external-dns.alpha.kubernetes.io/hostname: myapp.team.stage.company.com
spec:
  type: ExternalName
  externalName: lb-traefik.team.stage.company.com

---

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: myapp-http
  namespace: stage
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`myapp.team.stage.company.com`)
      middlewares:
        - name: redirect-https
          namespace: traefik
      services:
        - kind: Service
          name: myapp
          port: 80

---

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: myapp-https
  namespace: stage
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`myapp.team.stage.company.com`)
      middlewares:
        - name: traefik-forward-auth
          namespace: stage
      services:
        - kind: Service
          name: myapp
          port: 80
  tls:
    secretName: wildcard-stage-team-company-com

</details>

### kc

|k |v |
|:-------------------:|--------------------------------------------|
| realm | internal-users |
| client | proxy |
| secret | MyClientAuthSecret |
| Valid Redirect URIs | https://auth.team.stage.company.com/_oauth |

## Desired State

1. Client sends a request to `app`'s url.

2. Middleware `proxy` is attached to `app`'s `IngressRoute`. It tweaks the client request, before it gets to `app`. The client is going to be redirected to `proxy`'s `Service`.

3. If client is not authenticated in OIDC (in our case it is Keycloak) `proxy` redirects the request to `kc` **realm** endpoint. If client can be identified via login / pass, and they have permissions to access `app` (keycloak's client scope), `kc` extends the request payload with some user info and cookies (csrf token).

4. After it -> `kc` will redirect the request back to `proxy`'s special path -> `/_oauth`.

5. `proxy` will see cookie in the request payload and will finally redirect the client to the `app`.

## Current State

I have an issue at the step **5** - **proxy** instead of redirection client to `app` **origin**, it redirects the request back to `kc`, and it gets to infinite loop till the request falls with `ERR_TO_MANY_REDIRECTS` error.

## Logs

### kc

No useful logs in `kc` with severity `INFO`

I have session in `kc`:

[Session](https://ah-public-pictures.hb.bizmrg.com/nx/kc-client-proxy.png)

### proxy

time="2022-03-03T15:29:41Z" level=debug msg="Authenticate request" headers="map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-us] Cookie:[_forward_auth_csrf=MySecretToken] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15] X-Forwarded-For:[94.X.X.X] X-Forwarded-Host:[auth.team.stage.company.com] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[traefik-7b758f795-j2x9w] X-Real-Ip:[94.X.X.X]]" rule=default source_ip=94.X.X.X
time="2022-03-03T15:29:41Z" level=debug msg="sending CSRF cookie and a redirect to OIDC login" source_ip=94.X.X.X

## browserdata

### kc -> proxy json

<details>

{
  "_initiator": {
    "type": "other"
  },
  "_priority": "VeryHigh",
  "_resourceType": "document",
  "cache": {},
  "connection": "926909",
  "request": {
    "method": "POST",
    "url": "https://keycloak.team.stage.company.com/auth/realms/internal-users/login-actions/authenticate?session_code=<SESSION_CODE>&execution=<EXECUTION>&client_id=proxy&tab_id=<TAB_ID>",
    "httpVersion": "HTTP/1.1",
    "headers": [
      {
        "name": "Host",
        "value": "keycloak.team.stage.company.com"
      },
      {
        "name": "Connection",
        "value": "keep-alive"
      },
      {
        "name": "Content-Length",
        "value": "68"
      },
      {
        "name": "Cache-Control",
        "value": "max-age=0"
      },
      {
        "name": "sec-ch-ua",
        "value": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"98\", \"Google Chrome\";v=\"98\""
      },
      {
        "name": "sec-ch-ua-mobile",
        "value": "?0"
      },
      {
        "name": "sec-ch-ua-team",
        "value": "\"macOS\""
      },
      {
        "name": "Upgrade-Insecure-Requests",
        "value": "1"
      },
      {
        "name": "Origin",
        "value": "null"
      },
      {
        "name": "Content-Type",
        "value": "application/x-www-form-urlencoded"
      },
      {
        "name": "User-Agent",
        "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.XX.XX.XX Safari/537.36"
      },
      {
        "name": "Accept",
        "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
      },
      {
        "name": "Sec-Fetch-Site",
        "value": "same-origin"
      },
      {
        "name": "Sec-Fetch-Mode",
        "value": "navigate"
      },
      {
        "name": "Sec-Fetch-User",
        "value": "?1"
      },
      {
        "name": "Sec-Fetch-Dest",
        "value": "document"
      },
      {
        "name": "Accept-Encoding",
        "value": "gzip, deflate, br"
      },
      {
        "name": "Accept-Language",
        "value": "en-US,en;q=0.9"
      },
      {
        "name": "Cookie",
        "value": "AUTH_SESSION_ID=<AUTH_SESSION_ID>.keycloak-0; AUTH_SESSION_ID_LEGACY=<AUTH_SESSION_ID>.keycloak-0; KC_RESTART=<KC_RESTART>; traefik-sticky=http://10.50.37.3:8080"
      }
    ],
    "queryString": [
      {
        "name": "session_code",
        "value": "<SESSION_CODE>"
      },
      {
        "name": "execution",
        "value": "<EXECUTION>"
      },
      {
        "name": "client_id",
        "value": "proxy"
      },
      {
        "name": "tab_id",
        "value": "<TAB_ID>"
      }
    ],
    "cookies": [
      {
        "name": "AUTH_SESSION_ID",
        "value": "<AUTH_SESSION_ID>.keycloak-0",
        "path": "/auth/realms/internal-users/",
        "domain": "keycloak.team.stage.company.com",
        "expires": "1969-12-31T23:59:59.000Z",
        "httpOnly": true,
        "secure": true,
        "sameSite": "None"
      },
      {
        "name": "AUTH_SESSION_ID_LEGACY",
        "value": "<AUTH_SESSION_ID>.keycloak-0",
        "path": "/auth/realms/internal-users/",
        "domain": "keycloak.team.stage.company.com",
        "expires": "1969-12-31T23:59:59.000Z",
        "httpOnly": true,
        "secure": true
      },
      {
        "name": "KC_RESTART",
        "value": "<KC_RESTART>",
        "path": "/auth/realms/internal-users/",
        "domain": "keycloak.team.stage.company.com",
        "expires": "1969-12-31T23:59:59.000Z",
        "httpOnly": true,
        "secure": true
      },
      {
        "name": "traefik-sticky",
        "value": "http://10.50.37.3:8080",
        "path": "/",
        "domain": "keycloak.team.stage.company.com",
        "expires": "1969-12-31T23:59:59.000Z",
        "httpOnly": false,
        "secure": true
      }
    ],
    "headersSize": 1892,
    "bodySize": 68,
    "postData": {
      "mimeType": "application/x-www-form-urlencoded",
      "text": "username=a.horbach%40company.com&password=PASSWORD&credentialId=",
      "params": [
        {
          "name": "username",
          "value": "a.horbach%40company.com"
        },
        {
          "name": "password",
          "value": "PASSWORD"
        },
        {
          "name": "credentialId",
          "value": ""
        }
      ]
    }
  },
  "response": {
    "status": 302,
    "statusText": "Found",
    "httpVersion": "HTTP/1.1",
    "headers": [
      {
        "name": "Cache-Control",
        "value": "no-store, must-revalidate, max-age=0"
      },
      {
        "name": "Content-Length",
        "value": "0"
      },
      {
        "name": "Content-Security-Policy",
        "value": "frame-src 'self'; frame-ancestors 'self'; object-src 'none';"
      },
      {
        "name": "Date",
        "value": "Mon, 07 Mar 2022 16:08:07 GMT"
      },
      {
        "name": "Location",
        "value": "https://auth.team.stage.company.com/_oauth?state=<STATE>%3Ahttps%3A%2F%2Fmyapp.team.stage.company.com%2F&session_state=<AUTH_SESSION_ID>&code=<CODE>.<AUTH_SESSION_ID>.<SOME_SECRET>"
      },
      {
        "name": "P3p",
        "value": "CP=\"This is not a P3P policy!\""
      },
      {
        "name": "Referrer-Policy",
        "value": "no-referrer"
      },
      {
        "name": "Set-Cookie",
        "value": "KEYCLOAK_LOCALE=; Version=1; Comment=Expiring cookie; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/internal-users/; Secure; HttpOnly"
      },
      {
        "name": "Set-Cookie",
        "value": "KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/internal-users/; Secure; HttpOnly"
      },
      {
        "name": "Set-Cookie",
        "value": "KEYCLOAK_IDENTITY=<KEYCLOAK_IDENTITY>; Version=1; Path=/auth/realms/internal-users/; SameSite=None; Secure; HttpOnly"
      },
      {
        "name": "Set-Cookie",
        "value": "KEYCLOAK_IDENTITY_LEGACY=<KEYCLOAK_IDENTITY>; Version=1; Path=/auth/realms/internal-users/; Secure; HttpOnly"
      },
      {
        "name": "Set-Cookie",
        "value": "KEYCLOAK_SESSION=internal-users/<KEYCLOAK_SESSION>/<AUTH_SESSION_ID>; Version=1; Expires=Thu, 10-Mar-2022 16:08:07 GMT; Max-Age=259200; Path=/auth/realms/internal-users/; SameSite=None; Secure"
      },
      {
        "name": "Set-Cookie",
        "value": "KEYCLOAK_SESSION_LEGACY=internal-users/<KEYCLOAK_SESSION>/<AUTH_SESSION_ID>; Version=1; Expires=Thu, 10-Mar-2022 16:08:07 GMT; Max-Age=259200; Path=/auth/realms/internal-users/; Secure"
      },
      {
        "name": "Set-Cookie",
        "value": "KEYCLOAK_REMEMBER_ME=; Version=1; Comment=Expiring cookie; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/internal-users/; Secure; HttpOnly"
      },
      {
        "name": "Strict-Transport-Security",
        "value": "max-age=31536000; includeSubDomains"
      },
      {
        "name": "X-Content-Type-Options",
        "value": "nosniff"
      },
      {
        "name": "X-Frame-Options",
        "value": "SAMEORIGIN"
      },
      {
        "name": "X-Robots-Tag",
        "value": "none"
      },
      {
        "name": "X-Xss-Protection",
        "value": "1; mode=block"
      }
    ],
    "cookies": [
      {
        "name": "KEYCLOAK_LOCALE",
        "value": "",
        "path": "/auth/realms/internal-users/",
        "domain": "keycloak.team.stage.company.com",
        "expires": "2022-03-07T16:08:07.690Z",
        "httpOnly": true,
        "secure": true,
        "sameSite": "None"
      }
    ],
    "content": {
      "size": 0,
      "mimeType": "x-unknown",
      "compression": 0
    },
    "redirectURL": "https://auth.team.stage.company.com/_oauth?state=<STATE>%3Ahttps%3A%2F%2Fmyapp.team.stage.company.com%2F&session_state=<AUTH_SESSION_ID>&code=<CODE>.<AUTH_SESSION_ID>.<SOME_SECRET>",
    "headersSize": 3284,
    "bodySize": 0,
    "_transferSize": 3284,
    "_error": null
  },
  "serverIPAddress": "34.251.109.144",
  "startedDateTime": "2022-03-07T16:08:07.683Z",
  "time": 253.26799997128546,
  "timings": {
    "blocked": 7.564999960236252,
    "dns": -1,
    "ssl": -1,
    "connect": -1,
    "send": 0.12899999999999995,
    "wait": 243.43899999404326,
    "receive": 2.13500001700595,
    "_blocked_queueing": 7.103999960236251
  }
}

</details>

### proxy -> kc json

<details>

{
  "_initiator": {
    "type": "other"
  },
  "_priority": "VeryHigh",
  "_resourceType": "document",
  "cache": {},
  "connection": "927096",
  "request": {
    "method": "GET",
    "url": "https://auth.team.stage.company.com/_oauth?state=<STATE>%3Ahttps%3A%2F%2Fmyapp.team.stage.company.com%2F&session_state=<SESSION_STATE>&code=<CODE>.<SESSION_STATE>.<SOME_SECRET>",
    "httpVersion": "HTTP/1.1",
    "headers": [
      {
        "name": "Host",
        "value": "auth.team.stage.company.com"
      },
      {
        "name": "Connection",
        "value": "keep-alive"
      },
      {
        "name": "Cache-Control",
        "value": "max-age=0"
      },
      {
        "name": "Upgrade-Insecure-Requests",
        "value": "1"
      },
      {
        "name": "User-Agent",
        "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.XX.XX.XX Safari/537.36"
      },
      {
        "name": "Accept",
        "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
      },
      {
        "name": "Sec-Fetch-Site",
        "value": "same-site"
      },
      {
        "name": "Sec-Fetch-Mode",
        "value": "navigate"
      },
      {
        "name": "Sec-Fetch-User",
        "value": "?1"
      },
      {
        "name": "Sec-Fetch-Dest",
        "value": "document"
      },
      {
        "name": "sec-ch-ua",
        "value": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"98\", \"Google Chrome\";v=\"98\""
      },
      {
        "name": "sec-ch-ua-mobile",
        "value": "?0"
      },
      {
        "name": "sec-ch-ua-team",
        "value": "\"macOS\""
      },
      {
        "name": "Accept-Encoding",
        "value": "gzip, deflate, br"
      },
      {
        "name": "Accept-Language",
        "value": "en-US,en;q=0.9"
      },
      {
        "name": "Cookie",
        "value": "_forward_auth_csrf=<STATE>"
      }
    ],
    "queryString": [
      {
        "name": "state",
        "value": "<STATE>%3Ahttps%3A%2F%2Fmyapp.team.stage.company.com%2F"
      },
      {
        "name": "session_state",
        "value": "<SESSION_STATE>"
      },
      {
        "name": "code",
        "value": "<CODE>.<SESSION_STATE>.<SOME_SECRET>"
      }
    ],
    "cookies": [
      {
        "name": "_forward_auth_csrf",
        "value": "<STATE>",
        "path": "/",
        "domain": ".team.stage.company.com",
        "expires": "2022-03-08T04:07:41.865Z",
        "httpOnly": true,
        "secure": true
      }
    ],
    "headersSize": 1044,
    "bodySize": 0
  },
  "response": {
    "status": 302,
    "statusText": "Found",
    "httpVersion": "HTTP/1.1",
    "headers": [
      {
        "name": "Content-Length",
        "value": "0"
      },
      {
        "name": "Date",
        "value": "Mon, 07 Mar 2022 16:08:08 GMT"
      },
      {
        "name": "Location",
        "value": "https://keycloak.team.stage.company.com/auth/realms/internal-users/protocol/openid-connect/auth?client_id=proxy&redirect_uri=https%3A%2F%2Fauth.team.stage.company.com%2F_oauth&response_type=code&scope=openid+profile+email+groups&state=<FORWARD_AUTH_CSRF>%3Ahttps%3A%2F%2Fauth.team.stage.company.com%2F"
      },
      {
        "name": "Set-Cookie",
        "value": "_forward_auth_csrf=<FORWARD_AUTH_CSRF>; Path=/; Domain=team.stage.company.com; Expires=Tue, 08 Mar 2022 04:08:08 GMT; HttpOnly; Secure"
      }
    ],
    "cookies": [
      {
        "name": "_forward_auth_csrf",
        "value": "<FORWARD_AUTH_CSRF>",
        "path": "/",
        "domain": "team.stage.company.com",
        "expires": "2022-03-08T04:08:08.000Z",
        "httpOnly": true,
        "secure": true
      }
    ],
    "content": {
      "size": 0,
      "mimeType": "x-unknown",
      "compression": 0
    },
    "redirectURL": "https://keycloak.team.stage.company.com/auth/realms/internal-users/protocol/openid-connect/auth?client_id=proxy&redirect_uri=https%3A%2F%2Fauth.team.stage.company.com%2F_oauth&response_type=code&scope=openid+profile+email+groups&state=<FORWARD_AUTH_CSRF>%3Ahttps%3A%2F%2Fauth.team.stage.company.com%2F",
    "headersSize": 585,
    "bodySize": 0,
    "_transferSize": 585,
    "_error": null
  },
  "serverIPAddress": "IP",
  "startedDateTime": "2022-03-07T16:08:07.939Z",
  "time": 364.34300001596284,
  "timings": {
    "blocked": 3.593000016462058,
    "dns": 54.203,
    "ssl": 95.025,
    "connect": 223.143,
    "send": 0.1769999999999925,
    "wait": 82.00700002304464,
    "receive": 1.2199999764561653,
    "_blocked_queueing": 3.251000016462058
  }
}

</details>

## Links

- https://doc.traefik.io/traefik/middlewares/http/forwardauth/
- https://brianturchyn.net/traefik-forwardauth-support-with-keycloak/
- https://geek-cookbook.funkypenguin.co.nz/ha-docker-swarm/traefik-forward-auth/
- https://itnext.io/how-to-implement-a-sso-middleware-for-traefik-v2-on-kubernetes-dcd9d45cc875
- https://github.com/thomseddon/traefik-forward-auth