---
# System prepended metadata

title: Infrastructure knowledge transfer to DESY/EGI/MAX IV

---

# Infrastructure knowledge transfer to DESY/EGI/MAX IV

2022-02-04

Present: Michael Schuh (DESY), Andrii Salnikov (MAX IV), Dmitrii Ermakov (MAX IV), Zdenek Matej (MAX IV), Johannes Reppin (DESY), Tim Wetzel (DESY)
Not present: Andrea Manzi (EGI) - last minute issue, Jason Brudvik (MAX IV)

### Andrii
- starts speaking about k8s at MAX IV
    - OKD (free version of OpenShift)
        - 3 workers
        - mainly information web system developers like that
    - VMware infrastructure
        - old Panosc demo-portal deployed on this (the 1st year of ExPaNDS)
    - k8s with Rancher
        - clusters:
            - DAQ & DAQ-developemnt
                - at MAX IV part of the DAQ is done in k8s
                - ????? for high-bandwidth networks from detectors
                - native GPFS mounts
                - charts for pipelines
                - example: SAXS beamline, Eiger detector, azimuthal integration, publishing visualisation
            - jupyterhub
                - before DockerSwarm, replaced now with k8s; beside this one there is another on SLURM
            - logging/observability
            - archive-db
        - JupyterHub
            - https://gitlab.maxiv.lu.se/scisw/jupyterhub/kubernetes-jupyterhub [internal, behind VPN]
                - some things hardcoded: volumes
            - several tools (hard to follow the list for a non-expert)
                - LDAP periodic scans
                - cpu maping with cgroups
                - LXCFS customization
                - nvidia-helper: nvidia-smi shows processes at GPU (not visible in docker outside of host pid namespace)
                - ...
        - images etc. is not here: MAX IV has Harbor at OKD for that
            - cashing proxy to somehow bypass docker-limits (limitted ppoling)


### Michael
- Questions and comments (please correct here, tough to understand details for non-expert)
    - some info exchange about Harbour could be usefull
    - CIFS, S3
    - docker hub limits: official dockerHub image as a client for one
    - MAX IV jouerny from FLUX to Gitlab CI for application deployemnt (CD), at DESY rather opposite
        - at MAX IV they are IT-infra and they provide tools for other subgroups
            - FLUX appeared too complex for the web-developers (typical user-group), contrary they are more used to Gitlab CI
            - OpenShift upgrade story/issue: legacy custom definition/resource left from TEKTON configuration 
        - educating users in working with k8s: quite some initial resitance, especially in DAQ
    - mainly questions around cutomization
        - profile list: i.e. user see specific list of images in JupyterHub
        - accessible e.g. legacy images
        - possibility to run a user defined container 
        - binder like function to e.g. from a git-link to run a container/jupyter

- sharing JupyterHub integrations
    - would be interesting to excahnge something on this (some parental leaves on DESY side)

- next meeting:
    - VISA on k8s ?


### Tim
- Questions and comments
    - in relation to GPUs in JupyterHub
    - DESY has GPUs in OpenStack cloud
        - also GitLab runnursin 



2022-01-28

### Andrii
- intro into MAX IV

### Michael
- Notes to EGI: for service too open, notes about profiles(?)
- first on JupyterHub level
- now more Keycloak and other services are using it
- DESY also part of the ~~Fraunhofer~~ Helmholtz Society 
- dCache
- DESY is focusing on VISA now

### Andrea
- not only technical
- Easier for DESY as they are part of other German structures
- Questions as GDPR, polices
    - Data controller
    - Data ….
- Demo environment (!!!) could be useful
- multiple federation systems likely needed

### Johannes 
* describes keycloak features ....
* GDPR notes
* [Keycloak Account Console](https://keycloak.desy.de/auth/realms/kubernetes/account/#/)
   - manage federated identities
* discussion mostly 
* EGI check in [Docs](https://docs.egi.eu/providers/check-in/sp/)
  
### Visa Demo
- https://eosc-pan-git.desy.de
- https://visa-demo.desy.de/
- example of starting instance
- mechanism to propagate user metainfo: userId, home-path, ...


- next meeting: Friday, Feb 4th, 14:00
- https://lu-se.zoom.us/j/64520866989?pwd=MUs0ZFZyVkNNVUVSaHBzUGJNcHBVUT09

### Keycloak docker compose File

```YAML=
version: "3.7"

services:    
  sso:
    image: quay.io/keycloak/keycloak:16.1.0
    container_name: "keycloak"
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - KEYCLOAK_USER=keycloak-admin
      - KEYCLOAK_PASSWORD=changeme
      - PROXY_ADDRESS_FORWARDING=true
      - DB_VENDOR=POSTGRES
      - DB_ADDR=database
      - DB_USER=keycloak
      - DB_PASSWORD=password
      - DB_DATABASE=keycloak
    networks:
      - internal
    ports:
      - 8080:8080
      - 8443:8443
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`keycloak.example.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=production"

  database:
    image: postgres
    container_name: "postgres"
    environment:
      - POSTGRES_USER=keycloak
      - POSTGRES_DATABASE=keycloak
      - POSTGRES_PASSWORD=password
    volumes:
     - database:/var/lib/postgreql/data
    networks:
      - internal

  traefik:
    image: "traefik:v2.6.0"
    container_name: "traefik"
    ports:
      - "443:443"
      - "80:80"
    expose:
      - "8080"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "./traefik.yml:/traefik.yaml"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - internal

networks:
  internal:
    driver: bridge

volumes:
  database:
```