# Infrastructure knowledge transfer to DESY/EGI/MAX IV

2022-02-04

Present: Michael Schuh (DESY), Andrii Salnikov (MAX IV), Dmitrii Ermakov (MAX IV), Zdenek Matej (MAX IV), Johannes Reppin (DESY), Tim Wetzel (DESY)
Not present: Andrea Manzi (EGI) - last minute issue, Jason Brudvik (MAX IV)

### Andrii
- starts speaking about k8s at MAX IV
- OKD (free version of OpenShift)
- 3 workers
- mainly information web system developers like that
- VMware infrastructure
- old Panosc demo-portal deployed on this (the 1st year of ExPaNDS)
- k8s with Rancher
- clusters:
- DAQ & DAQ-developemnt
- at MAX IV part of the DAQ is done in k8s
- ????? for high-bandwidth networks from detectors
- native GPFS mounts
- charts for pipelines
- example: SAXS beamline, Eiger detector, azimuthal integration, publishing visualisation
- jupyterhub
- before DockerSwarm, replaced now with k8s; beside this one there is another on SLURM
- logging/observability
- archive-db
- JupyterHub
- https://gitlab.maxiv.lu.se/scisw/jupyterhub/kubernetes-jupyterhub [internal, behind VPN]
- some things hardcoded: volumes
- several tools (hard to follow the list for a non-expert)
- LDAP periodic scans
- cpu maping with cgroups
- LXCFS customization
- nvidia-helper: nvidia-smi shows processes at GPU (not visible in docker outside of host pid namespace)
- ...
- images etc. is not here: MAX IV has Harbor at OKD for that
- cashing proxy to somehow bypass docker-limits (limitted ppoling)

### Michael
- Questions and comments (please correct here, tough to understand details for non-expert)
- some info exchange about Harbour could be usefull
- CIFS, S3
- docker hub limits: official dockerHub image as a client for one
- MAX IV jouerny from FLUX to Gitlab CI for application deployemnt (CD), at DESY rather opposite
- at MAX IV they are IT-infra and they provide tools for other subgroups
- FLUX appeared too complex for the web-developers (typical user-group), contrary they are more used to Gitlab CI
- OpenShift upgrade story/issue: legacy custom definition/resource left from TEKTON configuration
- educating users in working with k8s: quite some initial resitance, especially in DAQ
- mainly questions around cutomization
- profile list: i.e. user see specific list of images in JupyterHub
- accessible e.g. legacy images
- possibility to run a user defined container
- binder like function to e.g. from a git-link to run a container/jupyter

- sharing JupyterHub integrations
- would be interesting to excahnge something on this (some parental leaves on DESY side)

- next meeting:
- VISA on k8s ?

### Tim
- Questions and comments
- in relation to GPUs in JupyterHub
- DESY has GPUs in OpenStack cloud
- also GitLab runnursin

2022-01-28

### Andrii
- intro into MAX IV

### Michael
- Notes to EGI: for service too open, notes about profiles(?)
- first on JupyterHub level
- now more Keycloak and other services are using it
- DESY also part of the ~~Fraunhofer~~ Helmholtz Society
- dCache
- DESY is focusing on VISA now

### Andrea
- not only technical
- Easier for DESY as they are part of other German structures
- Questions as GDPR, polices
- Data controller
- Data ….
- Demo environment (!!!) could be useful
- multiple federation systems likely needed

### Johannes
* describes keycloak features ....
* GDPR notes
* [Keycloak Account Console](https://keycloak.desy.de/auth/realms/kubernetes/account/#/)
- manage federated identities
* discussion mostly
* EGI check in [Docs](https://docs.egi.eu/providers/check-in/sp/)

### Visa Demo
- https://eosc-pan-git.desy.de
- https://visa-demo.desy.de/
- example of starting instance
- mechanism to propagate user metainfo: userId, home-path, ...

- next meeting: Friday, Feb 4th, 14:00
- https://lu-se.zoom.us/j/64520866989?pwd=MUs0ZFZyVkNNVUVSaHBzUGJNcHBVUT09

### Keycloak docker compose File

version: "3.7"

services:    
  sso:
    image: quay.io/keycloak/keycloak:16.1.0
    container_name: "keycloak"
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - KEYCLOAK_USER=keycloak-admin
      - KEYCLOAK_PASSWORD=changeme
      - PROXY_ADDRESS_FORWARDING=true
      - DB_VENDOR=POSTGRES
      - DB_ADDR=database
      - DB_USER=keycloak
      - DB_PASSWORD=password
      - DB_DATABASE=keycloak
    networks:
      - internal
    ports:
      - 8080:8080
      - 8443:8443
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`keycloak.example.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=production"

  database:
    image: postgres
    container_name: "postgres"
    environment:
      - POSTGRES_USER=keycloak
      - POSTGRES_DATABASE=keycloak
      - POSTGRES_PASSWORD=password
    volumes:
     - database:/var/lib/postgreql/data
    networks:
      - internal

  traefik:
    image: "traefik:v2.6.0"
    container_name: "traefik"
    ports:
      - "443:443"
      - "80:80"
    expose:
      - "8080"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "./traefik.yml:/traefik.yaml"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - internal

networks:
  internal:
    driver: bridge

volumes:
  database: