# Guidelines for making EESSI available via CernVM-FS ## EESSI CernVM-FS repository `software.eessi.io` status in Jan 2025: - 9+2 supported CPU targets (6+1 x86_64, 3+1 aarch64) - ~10,000 software installations in total (~1,000 per CPU target), excl. extensions - ~19 million files in total (~1.8M per CPU target) - full mirror of `software.eessi.io` repository currently requires ~300GB - thanks to deduplication + compression features of CernVM-FS compared to typical `*.cern.ch` repositories: - quicker turnaround time of software updates - `sft.cern.ch` - LHC experiments repos (lhcb, atlas, alice) - grid sites typically do not set up their own Stratum 1 - required disk space can be a hurdle (~120TB compressed, ~900 TBs uncompressed) ## Recommended approach ### Local Stratum 1 "mirror" server - can be physical hardware or VM - **fast storage (SSDs/NVME)** - support for maintaining a partial mirror of CernVM-FS repositories is planned, which would allow to drastically reduce required disk space (by ignoring parts of EESSI repository for irrelevant CPU targets); - => couple of TBs of disk space should be plenty for years to come (for only EESSI)... - **low latency/high bandwidth network** (10Gb or better) to workernodes where EESSI is used - handful of cores + ~2GB/core of RAM should be sufficient - **active monitoring** of available disk space, network bandwidth usage, CPU load - full disk would be (really) problematic, so must be avoided; - startup performance of software provided through EESSI would suffer in case of high network/CPU load; - CernVM-FS can be configured to use EFP Stratum 1 + EESSI public Stratum 1's as automatic fallback - optional: proxy servers (Squid, maybe Varnish) to offload the Stratum 1 server - only if required based on load observed on Stratum 1 ### Native installation of CernVM-FS client software on workernodes - with local client cache - usually on local disk (ideally SSD/NVME), couple of GBs (~) - can also be ramdisk - can also be loopback device on parallel filesystem (less ideal, exposed to metadata latency of Lustre, ...) ## Alternative approaches ### Only proxy servers, no local Stratum 1 pros: - ... cons: - ... ### Container image with CernVM-FS pros: - ... cons: - ... ### `cvmfsexec` pros: - ... cons: - ... ### `squashfs` images pros: - ... cons: - ... ### CernVM-FS alien cache pros: - ... cons: - ... ## References ### Vega - ... ### Karolina - ... ### Deucalion - ... ### Digital Research Alliance of Canada (a.k.a. ComputeCanada) - Stratum 1 servers - some OpenStack VMs (8 cores + 12 GB of RAM) - some (overpowered) bare metal servers - ~1.5TB of storage for `*.computecanada.ca` CernVM-FS repositories - storage decoupled from server via CEPH S3 object storage => fast + scalable ### CERN typical Stratum 1: - HA setup, bare metal servers - 32 cores, 64GB RAM, storage JBOD - to be replaced soon (64 cores, 128GB RAM, 3PB disk JBOD) - 4 Squid proxies in front - 25 Squid proxies in front of each - 5 subclusters, each dedicated to each experiment + a public one - 2 nginx proxies (HPC proxies) - mainly to separate the traffic - exponential growth in size of repos - mostly due `unpacked.cern.ch` - unpacked container images hosted in CernVM-FS repository - any CERN user can request extra images to be ingested + service to auto-update the images - ~5k publishes/day for nightly builds experiments ### NERSC ### Fermilab