EESSI CL - 2023.06

# EESSI CL - 2023.06 - effort to build a new compatability layer for EESSI - should address shortcomings identified with 2023.04 - OpenSSL 3.0 in CL causes troubles - purpose of this is also to document how to build (and how we built) a new version of the compatibility layer See additional preparation notes [below](#Preparation). We assume you have a machine/VM with sufficient disk space (min 25 GiB, recommended 50+ GiB or more if you want/need to retain several versions simultaneously) and Singularity/Apptainer, git and tmux/screen installed. ## Before you start ... Run everything in a `tmux` or `screen` session. Building the compat layer takes time, so it would be very annoying to loose results if your connection drops. Use your favourite tool. Below we use `tmux` just for illustration. ```bash= tmux new -s cl-2023-06 ``` Leave with <kbd>Ctrl</kbd>+<kbd>b</kbd> & <kbd>d</kbd>. Attach with `tmux a -t cl-2023-06`. Useful commands [tmux cheat sheet](https://kysuit.net/linux/learn-tmux-basic/) ## Keep different build trials separated Create different top-level directories for different attempts. ```bash= mkdir my_first_cl cd $_ ``` ## Clone EESSI/compatibility-layer from GitHub ```bash= git clone https://github.com/EESSI/compatibility-layer.git cd compatibility-layer ``` ## (optional) Create a new branch and check it out If you anticipate changes (as we do for CL v2023.06), create a new branch and check it out. The name of the branch below contains the short sha of the Gentoo commit that was used for the 2023.04 layer. We use that first, hoping it may still build (we may want to bump that to a more recent version later on). The Gentoo commit is explicitly given in `ansible/playbooks/roles/compatibility_layer/defaults/main.yml`, see value of `gentoo_git_commit`. ```bash= git checkout -b eessi-2023.06-gentoo-2949284 origin/main ``` ## Do changes and commit them locally We set version to `2023.06` and we mask the version of OpenSSL so only versions before v3.* are emerged by the Gentoo package manager. Changes via `git diff`. Note 1, the version has to be changed in three different files - `ansible/playbooks/roles/compatibility_layer/defaults/main.yml` - `bot/build.sh` - `install_compatibility_layer.sh` Note 2, the change of the `custom_overlays` setting `url` is only needed until a package set for the new version 2023.06 is added to EESSI/gentoo-overlay (see [EESSI/gentoo-overlay PR#92](https://github.com/EESSI/gentoo-overlay/pull/92)). See below for a description of [how to create a new package set](#Create-a-new-EESSI-package-set). ```diff= diff --git a/ansible/playbooks/roles/compatibility_layer/defaults/main.yml b/ansible/playbook s/roles/compatibility_layer/defaults/main.yml index 99501e2..847f8ac 100644 --- a/ansible/playbooks/roles/compatibility_layer/defaults/main.yml +++ b/ansible/playbooks/roles/compatibility_layer/defaults/main.yml @@ -1,11 +1,11 @@ # Defaults file for the compatibility layer role. --- -eessi_version: "2023.04" +eessi_version: "2023.06" custom_overlays: - name: eessi source: git - url: https://github.com/EESSI/gentoo-overlay.git + url: https://github.com/trz42/gentoo-overlay.git eclass-overrides: true cvmfs_repository: pilot.eessi-hpc.org @@ -26,6 +26,10 @@ prefix_mask_packages: | # stick to GCC 9.x; using a too recent compiler in the compat layer complicates stuff in the software layer, # see for example https://github.com/EESSI/software-layer/issues/151 >=sys-devel/gcc-10 + # mask OpenSSL 3.x, stick to OpenSSL 1.1.x for now to avoid problems with: + # - older versions of Rust (see https://github.com/EESSI/software-layer/issues/257) + # - older versions of cryptograhy in Python (see https://github.com/EESSI/software-layer/issues/258) + >=dev-libs/openssl-3 prefix_unmask_packages: | # unmask older GCC to make it installable =sys-devel/gcc-9* diff --git a/bot/build.sh b/bot/build.sh index 9995db5..0bc2584 100755 --- a/bot/build.sh +++ b/bot/build.sh @@ -81,7 +81,7 @@ host_arch=$(uname -m) eessi_arch=${cpu_target_arch:-${host_arch}} eessi_os=linux job_version=$(cfg_get_value "repository" "repo_version") -eessi_version=${job_version:-2023.04} +eessi_version=${job_version:-2023.06} job_repo=$(cfg_get_value "repository" "repo_name") eessi_repo=${job_repo:-pilot.eessi-hpc.org} tar_topdir=/cvmfs/${eessi_repo}/versions diff --git a/install_compatibility_layer.sh b/install_compatibility_layer.sh index 84e8ede..b621dcc 100755 --- a/install_compatibility_layer.sh +++ b/install_compatibility_layer.sh @@ -10,7 +10,7 @@ REPOSITORY="pilot.eessi-hpc.org" RESUME= RETAIN_TMP=0 STORAGE= -VERSION=2023.04 +VERSION=2023.06 VERBOSE= display_help() { ``` (optional) Commit changes locally ```bash git commit -am "bump version to 2023.06 + mask OpenSSL versions 3.*" ``` ## Create a new EESSI package set After the Gentoo Prefix has been bootstrapped additional packages needed for EESSI are installed. The packages are listed in a package set in the EESSI/gentoo-overlay repository. Clone that repository and create a new branch. ```bash= # get current state of EESSI/gentoo-overlay repository git clone https://github.com/EESSI/gentoo-overlay.git cd gentoo-overlay # checkout a new branch for your changes git checkout -b package_set_for_2023_06 ``` A package set is defined via the file `scripts/eessi_sets.yml`. The diff for the 2023.06 set looks as follows ```diff= diff --git a/scripts/eessi_sets.yml b/scripts/eessi_sets.yml index 5802054..111fd60 100644 --- a/scripts/eessi_sets.yml +++ b/scripts/eessi_sets.yml @@ -7,7 +7,7 @@ eessi_archs: # - macos-x86_64 eessi_sets: - - name: eessi-2023.04 + - name: eessi-2023.06 packages: - name: app-shells/bash-completion - name: dev-python/GitPython @@ -23,7 +23,7 @@ eessi_sets: - name: media-fonts/dejavu - name: media-fonts/liberation-fonts - name: sys-apps/archspec - version: 0.2.0 + version: 0.2.1 overlay: eessi - name: sys-apps/dbus - name: sys-cluster/lmod ``` The above diff shows that not only the name of the compat layer was changed to `eessi-2023.06` but also the version of `archspec`. This requires an ebuild file for the new version and then also an updated `Manifest` file. See output of `git diff --name-only` ```diff= scripts/eessi_sets.yml sys-apps/archspec/Manifest sys-apps/archspec/archspec-0.2.1.ebuild ``` The new ebuild file may include some changes, for example ```diff= diff sys-apps/archspec/archspec-0.2.0.ebuild sys-apps/archspec/archspec-0.2.1.ebuild 26,27d25 < RDEPEND=">dev-python/six-1.16.0[${PYTHON_USEDEP}] < =dev-python/click-8.1.3[${PYTHON_USEDEP}]" ``` So some dependencies have been dropped. Any change to a package (added file, changed file, removed file) requires an update of the `Manifest`. ### Updating the `Manifest` file The `Manifest` file of a package defined in a directory, say `sys-apps/archspec` can be updated by running the command (note the lowercase `manifest` is not a typo, but a "command" to `ebuild`) ```bash= ebuild archspec-0.2.1.ebuild manifest ``` in the directory `sys-apps/archspec`. You can get access to the `ebuild` command via a Gentoo installation such as the EESSI pilot. The below sequence shows the commands to update the Manifest. We assume that you have already cloned the EESSI/gentoo-overlay repository, created a new branch and that the EESSI pilot is already accessible on your system. If not you can get access to it via a container (see [EESSI container](https://www.eessi.io/docs/getting_access/eessi_container/)) ```bash= cd sys-apps/archspec # init the EESSI pilot source /cvmfs/pilot.eessi-hpc.org/latest/init/bash # update the Manifest DISTDIR=/tmp ebuild archspec-0.2.1.ebuild manifest # verify that it has changed git diff ``` Finally, commit the changes and push them to your fork of the EESSI/gentoo-overlay repository. ```bash= git commit -am "update manifest for archspec 0.2.1" git remote add YOUR_FORK https://github.com/YOUR_FORK/gentoo-overlay.git git push YOUR_FORK update_archspec_v0_2_1_manifest ``` If you merge it into the main branch of your fork, you can immediately use it for building a new compatibility layer. You only need to point the `url` in the `custom_overlays` to your fork. In the long term you may want that the EESSI/gentoo-overlay is updated with your changes. Just make a pull request to add your changes to EESSI/gentoo-overlay. ## Build compatibility layer Run the following command (to build for `x86_64`) ```bash! ./install_compatibility_layer.sh -a x86_64 -v 2023.06 -r pilot.eessi-hpc.org -c docker://ghcr.io/eessi/bootstrap-prefix:debian11 2>&1 | tee -a run.2023.06-jun7-1.log ``` or run the following to build for `aarch64` ```bash! ./install_compatibility_layer.sh -a aarch64 -v 2023.06 -r pilot.eessi-hpc.org -c docker://ghcr.io/eessi/bootstrap-prefix:debian11 2>&1 | tee -a run.2023.06-jun7-1.log ``` or run the command (if you set `$TEMP_STORAGE` in the [preparation](#Create-folder-for-temporary-storage)) ```bash! ./install_compatibility_layer.sh -a x86_64 -v 2023.06 -r pilot.eessi-hpc.org -g ${TEMP_STORAGE} -c docker://ghcr.io/eessi/bootstrap-prefix:debian11 2>&1 | tee -a run.2023.06-jun7-1.log ``` # Preparation Depending on your machine/VM you may have to do additional steps to prepare the build environment. ## Check temporary storage By default the install script uses `$TMPDIR` or `/tmp`. If neither of these is on a large enough disk, create a folder and point `$TEMP_STORAGE` to it. First check disk space on `$TMPDIR` and `/tmp` ```bash= [[ -z ${TMPDIR} ]] || df -h $TMPDIR && echo "# TMPDIR is not set, run: df -h /tmp" # TMPDIR is not set, run: df -h /tmp df -h /tmp ``` If output for either is like below, all is fine. If not, you need to create a temporary storage on a large enough disk (see [Section _Create folder for temporary storage_](#Create-folder-for-temporary-storage)). ``` Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p3 100G 18G 83G 18% / ``` ## Create folder for temporary storage First check available storage with `df -h .`, then (if result not satisfactory with `df -h`). Below we assume that your current directory (result of `df -h .`) is on a sufficiently large disk. ```bash= mkdir temp_storage export TEMP_STORAGE=${PWD}/temp_storage ``` Verify that temp storage is on a large enough disk (no worries, the script will do that too). ```bash= df -h ${TEMP_STORAGE} ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.