---
# System prepended metadata

title: Guidelines for making EESSI available via CernVM-FS

---

# Guidelines for making EESSI available via CernVM-FS

## EESSI CernVM-FS repository `software.eessi.io`

status in Jan 2025:
- 9+2 supported CPU targets (6+1 x86_64, 3+1 aarch64)
- ~10,000 software installations in total (~1,000 per CPU target), excl. extensions
- ~19 million files in total (~1.8M per CPU target)
- full mirror of `software.eessi.io` repository currently requires ~300GB
    - thanks to deduplication + compression features of CernVM-FS

compared to typical `*.cern.ch` repositories:
- quicker turnaround time of software updates
- `sft.cern.ch`
- LHC experiments repos (lhcb, atlas, alice)
- grid sites typically do not set up their own Stratum 1
    - required disk space can be a hurdle (~120TB compressed, ~900 TBs uncompressed)

## Recommended approach

### Local Stratum 1 "mirror" server

- can be physical hardware or VM
- **fast storage (SSDs/NVME)**
    - support for maintaining a partial mirror of CernVM-FS repositories is planned, which would allow to drastically reduce required disk space (by ignoring parts of EESSI repository for irrelevant CPU targets);
    - => couple of TBs of disk space should be plenty for years to come (for only EESSI)...
- **low latency/high bandwidth network** (10Gb or better) to workernodes where EESSI is used
- handful of cores + ~2GB/core of RAM should be sufficient
- **active monitoring** of available disk space, network bandwidth usage, CPU load
    - full disk would be (really) problematic, so must be avoided;
    - startup performance of software provided through EESSI would suffer in case of high network/CPU load;
- CernVM-FS can be configured to use EFP Stratum 1 + EESSI public Stratum 1's as automatic fallback
- optional: proxy servers (Squid, maybe Varnish) to offload the Stratum 1 server
    - only if required based on load observed on Stratum 1


### Native installation of CernVM-FS client software on workernodes

- with local client cache
    - usually on local disk (ideally SSD/NVME), couple of GBs (~)
    - can also be ramdisk
    - can also be loopback device on parallel filesystem (less ideal, exposed to metadata latency of Lustre, ...)

## Alternative approaches

### Only proxy servers, no local Stratum 1

pros:
- ...
cons:
- ...

### Container image with CernVM-FS

pros:
- ...
cons:
- ...

### `cvmfsexec`

pros:
- ...
cons:
- ...

### `squashfs` images

pros:
- ...
cons:
- ...

### CernVM-FS alien cache

pros:
- ...
cons:
- ...

## References

### Vega

- ...

### Karolina

- ...

### Deucalion

- ...

### Digital Research Alliance of Canada (a.k.a. ComputeCanada)

- Stratum 1 servers
    - some OpenStack VMs (8 cores + 12 GB of RAM)
    - some (overpowered) bare metal servers
    - ~1.5TB of storage for `*.computecanada.ca` CernVM-FS repositories
    - storage decoupled from server via CEPH S3 object storage => fast + scalable

### CERN

typical Stratum 1:
- HA setup, bare metal servers
    - 32 cores, 64GB RAM, storage JBOD
    - to be replaced soon (64 cores, 128GB RAM, 3PB disk JBOD)
- 4 Squid proxies in front
    - 25 Squid proxies in front of each
        - 5 subclusters, each dedicated to each experiment + a public one
- 2 nginx proxies (HPC proxies)
    - mainly to separate the traffic
- exponential growth in size of repos
    - mostly due `unpacked.cern.ch`
        - unpacked container images hosted in CernVM-FS repository
        - any CERN user can request extra images to be ingested + service to auto-update the images
    - ~5k publishes/day for nightly builds experiments

### NERSC

### Fermilab