# Handling of credentials in Renku
This document is a summary of the main authentication and authorization flows in Renku. It explains how user credentials are handled in Renku by the application and its user-facing clients. The last section also summarizes how deployment credentials are managed.
## Table of contents:
* Description of key services
* API Authorization
* Interactive session access
* DevOps flow
## Key services
These are the most important services in Renku that handle user credentials.
### Keycloak
[Keycloak](https://www.keycloak.org/) is an open source identity and access management application. Renku uses it as its main source of user identities and the Renku services use tokens provided by Keycloak to authenticate and authorize requests.
### GitLab
[GitLab](https://gitlab.com) is an open-source repository management solution. Renku relies on GitLab to manage user projects. GitLab can be provided either as a part of the Renku instance, or linked to a Renku instance as an external service. In either mode, Renku requires users to authorize Renku clients to use their GitLab credentials on their behalf for operations such as repository cloning and pushing.
### Gateway
The Renku Gateway is a component of the Renku platform, through which backend APIs are exposed. The Gateway takes care of ensuring that API requests are authenticated and that they contain the credentials needed by the backend to perform the required tasks.
## API authorization
Renku services expose a variety of APIs to external clients via the Renku Gateway. The API Gateway is itself a service that provides a unified interface for clients to access backends like GitLab, the Renku core service, the notebooks service etc. Its primary function is authorizing clients to access various backend services and authenticating clients with those services.
### Log in to Renku
Upon receiving a login request from a client, the gateway initiates an authentication request to Keycloak which prompts the user to log in if required. Some clients (e.g. Renku CLI client) might need to be authorized to access Keycloak; Keycloak prompts the user to authorize them. Upon success, Keycloak access/refresh/ID tokens are retrieved from Keycloak by the gateway and stored in a Redis storage. The gateway then redirects the client to initiate another authentication request to the Gitlab; it retrieves Gitlab access tokens and stores them in the Redis storage. Clients that do not have an active session to the server (e.g. Renku CLI client) need to store a Keycloak access token locally, to include it in future requests; those clients send a request to receive this token. For interactive clients (e.g. Renku UI) the access token will be saved in their session cookie by the gateway.
The following diagram shows different login steps for the Renku UI client:
```mermaid
sequenceDiagram
participant UI
participant Gateway
participant Keycloak
participant GitLab
participant Redis
UI ->>+ Gateway: /api/auth/login?reidrect_url
activate UI
Gateway ->> Redis: store state, redirect_url
Gateway -->>- UI: redirect
UI ->>+ Keycloak: /auth/realms/Renku/protocol/openid-connect/auth
Keycloak ->> Keycloak: login page if needed
Keycloak -->>- UI: redirect
UI ->>+ Gateway: /api/auth/token
Redis ->> Gateway: retreive state, redirect_url
Gateway ->>+ Keycloak: /auth/realms/Renku/protocol/openid-connect/token
Keycloak -->>- Gateway: access/refresh/ID tokens
Gateway ->> Redis: store Keycloak tokens
Gateway -->>- UI: redirect
UI ->>+ Gateway: /api/auth/gitlab/login?reidrect_url
Gateway ->> Redis: store state, redirect_url
Gateway -->>- UI: redirect
UI ->>+ GitLab: /gitlab/auth/authorize
GitLab ->> GitLab: redirect to Keycloak if needed
GitLab -->>- UI: redirect
UI ->>+ Gateway: /api/auth/gitlab/token
Redis ->> Gateway: retreive state, redirect_url
Gateway ->>+ GitLab: /gitlab/oauth/token
GitLab -->>- Gateway: access/refresh/ID tokens
Gateway ->> Redis: store GitLab tokens
Gateway -->>- UI: redirect to reidrect_url
deactivate UI
```
### Accessing the Renku API
Requests to access backend APIs are all routed through the gateway. When a client sends a request to use one of the backend APIs, the gateway uses the client's access token to find and inject the correct user credentials that are required by the target service. As mentioned in the previous section, access tokens for clients are either stored in the session (for interactive clients) or are included with the request (for non-interactive clients).
The gateway does the following steps to authenticate a request:
1. It looks for an access token, first in the request's authorization header and then in the session cookie.
2. If an access token is found, the gateway uses it as a key to the Redis store to get the Keycloak access token.
3. If the Keycloak access token is not found in Redis or if it's expired, then the session ended and the client cannot access the service.
4. If the requested backend service requires a GitLab access token, the gateway uses the Keycloak access token as a key to retrieve the GitLab access token from Redis.
6. Depending on the service that is being accessed, various request headers are updated and the access tokens are stored in them.
7. The request with the modified header is forwarded to the proper service.
The following diagram shows the sequence for a request that requires GitLab access:
```mermaid
sequenceDiagram
participant Client
participant Gateway
participant GitLab
participant Redis
Client ->>+ Gateway: /api/<resource> (session cookie/request parameter)
activate Client
Gateway ->> Gateway: extract access token from request parameters/session cookie
Redis ->> Gateway: retreive Keycloak tokens
Gateway ->> Gateway: validate Keycloak access token
Redis ->> Gateway: retreive GitLab tokens
Gateway ->> Gateway: include the GitLab access token in the request's Autorization header
Gateway ->>+ GitLab: /api/gitlab/<resource> (modified request parameters)
GitLab ->> GitLab: process request
GitLab -->>- Gateway: API response
Gateway -->>- Client: API response
deactivate Client
```
Similar flows are implemented for other backends, but only providing the required credentials.
### Logout
When a client sends a logout request, the gateway removes its associated access tokens from the Redis store and terminates the session.
## Interactive session access
There are two types of sessions:
- for anonymous users that are not logged in
- for normal registered (and logged in) users
These have a slightly different structure because anonymous users get assigned and are identified by a random ID, whereas registered users are all registered in Renku's `Keycloak` or any identity provider that Renku's `Keycloak` is connected to. Anonymous user sessions are optional and need to be explicitly enabled by an administrator. By default only registered and logged-in users can launch interactive sessions.
### Registered user sessions
Each session comes with its own ingress that leads only to the `Oauth2 proxy` container. Any requests for the session coming from outside the cluster have to go through the `Oauth2 proxy`. The `Oauth2 proxy` is a client to Renku's `Keycloak` and it can determine whether a user is logged in or not and whether they are authorized to access the session. The authorization is done based on the email contained in the user's identity token from `Keycloak`. If the request is coming from a logged-in and authorized user, the request is forwarded to the session container (running a JuptyerLab server). If the user is not logged in they are redirected to the login sequence; if they fail or are not authorized to access the session, the request is rejected.
```mermaid
flowchart LR
User[User]
subgraph Kubernetes[ ]
SI[Session<br>ingress]
subgraph Session statefulset
Jupyterlab[Jupyterlab]
auth[Oauth2 proxy]
end
Keycloak[Keycloak]
end
User --> |GET /sessions/session_name| SI
SI --> auth
auth --> Keycloak
Keycloak --> auth
auth --> Jupyterlab
```
### Anonymous user sessions
When anonymous users first access renku they are assigned an ID that is stored in a cookie in the user's browser. This id is used also as an access token in the session. In this case there is no need to use `Keycloak` or the `Oauth2 proxy`, Jupyterlab has a simple token based authentication that is utilized. If this token (which has the same value as the anonymous user ID) is not present or provided by the user they cannot access the session. We use a simple `traefik` proxy in every session to forward the request to the Jupyterlab container. The traefik proxy does not do any modifications to the request. This proxy may be removed in future versions because it is not strictly necessary.
Anonymous user sessions are quite limited relative to registered user sessions. We do not have any credentials for anonymous users and therefore they cannot do any git operations that require authentication (i.e. push, create branches, etc.).
```mermaid
flowchart LR
User[User]
subgraph Kubernetes[ ]
SI[Session<br>ingress]
subgraph Session statefulset
Jupyterlab[Jupyterlab<br>container]
proxy[Traefik proxy]
end
end
User --> |GET /sessions/session_name?token| SI
SI --> proxy
proxy --> Jupyterlab
```
### Git credentials in registered user sessions
The JupyterLab container in a user session is running an image created by the user. Since we do not have full control over this image we cannot trust the container to have direct access to credentials. However, registered users need to be able to access GitLab from within a session without being asked for credentials.
To solve this problem, all registered user sessions are configured to use a proxy for git requests. The proxy injects the user's git credentials for outgoing git requests, but only for the repository from which the session was started. This way we can avoid a user's credentials being hijacked to access other (potentially private or sensitive) repositories if a user is tricked into launching a sessions with a malicious image.
```mermaid
flowchart LR
subgraph Session statefulset
Jupyterlab[Jupyterlab<br>container]
gitProxy[Git proxy]
end
GitLab
Jupyterlab --> gitProxy
gitProxy --> GitLab
```
Cloning the repository is done in a Kubernetes `initContainer` that runs an image that we create. Therefore this container has the user git credentials. But the credentials are removed as soon as the clone operation finishes (regardless of whether it succeeded or failed).
### Network access to and from user sessions
Egress from user sessions is restricted to hosts that are outside of the cluster. This prevents users inside the interactive sessions from trying to access e.g. the Postgres database that is running in the same cluster.
Similarly, the ingress to user sessions is limited (via a network policy) to only occur through the `Oauth proxy`, in order to prevent attempts to bypass the proxy and access the running JupyterServer directly. In the case of anonymous sessions the only ingress to the session is through the Traefik proxy. This is applied to ingress that originates inside or outside of the cluster. Therefore, this network policy essentially forces all requests through a single "path" that always requires authentication.
### External storage access from interactive sessions
A recently added feature to Renku is the ability to mount S3 buckets to user sessions. This is possible with any S3 compatible storage (not just with AWS). This feature is optional and must be enabled by an administrator of the Renku instance.
The user is expected to provide credentials for every bucket they want to mount every time they lunch a session. If the bucket is public then credentials are not required, only the endpoint of the S3 storage service and the name of the bucket.
In a bit more detail, S3 credentials are handled as follows:
1. A user enters the credentials in their browser when launching a session (the credentials are not stored in the browser at all)
2. The credentials are sent with a POST request to the Renku notebook service, which is responsible for managing interactive sessions
3. The credentials are used to verify that the bucket exists in the notebook service (they are not stored in the notebook service)
4. The notebook service creates a manifest for a custom Kubernetes resource that is used by the controller process to create all the required resources needed by a session (the credentials are stored in this manifest)
5. The controller uses the manifest from the previous step to create a Kubernetes secret with the credentials (the credentials are stored in this secret). The secret is deleted when the user session that is using the S3 bucket that needs said credentials is deleted.
The S3 credentials are not stored anywhere in the user session because the S3 bucket is mounted by a [custom Kubernetes storage driver](https://datashim-io.github.io/datashim/) that operates outside of the user session and can handle S3 buckets.
## DevOps workflow
The Renku deployment is managed through Helm; the Helm chart and its values file are operated by Flux, our chosen Gitops tool. The values file contains the administrator credentials of the different services included in Renku: Keycloak, Postgres, GitLab (if deployed). Those credentials are encrypted using AES-256, through SOPS/GPG and stored in a private Git repository accessible by members of the Renku team. The private key used for the encryption is only available to the administrators within the Renku team and are rotated when a member leaves the team
Once deployed in the cluster the credentials are stored as Kubernetes secrets in the Renku namespace. Only the Renku-team administrators have access to the credentials to the Kubernetes cluster.
Google Drive
Gist
Clipboard
Sign in with Wallet
