Package Management with Spack

Tuesday, May 18th, 2021 9:00-10:00

Anirban Pal, Assistant Professor West Texas A&M University, [e-mail]

Back to Resbaz HackMD Directory

What is Spack?

Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures, using libraries that do not have a standard ABI. Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system.

Most importantly, Spack is simple. It offers a simple spec syntax so that users can specify versions and configuration options concisely. Spack is also simple for package authors: package files are written in pure Python, and specs allow package authors to maintain a single file for many different builds of the same package. (https://spack.readthedocs.io/en/latest/)

Pre-requisites: Familiarity with Linux command line, and some basic exposure to compiling software on linux systems.

"The History of every major Galactic Civilization tends to pass through three distinct and recognizable phases, those of Survival, Inquiry and Sophistication, otherwise known as the How, Why, and Where phases. For instance, the first phase is characterized by the question 'How can we eat?' the second by the question 'Why do we eat?' and the third by the question 'Where shall we have lunch?"" (Douglas Adams, The Restaurant at the End of the Universe )

Spack assists and offers "Survival, Inquiry and Sophistication" in the business of building scientific packages on the HPC workspace both for users and administrators.

Why Spack?

Compiling, linking and building packages on HPC systems can be very challenging, especially if you have to build from source tarballs and the package has several dependencies. In this context, building from the source tarball with a source-compiler fails as

(1) a dependency was compiled with a different compiler from the source-compiler, OR
(2) a dependency was compiled with the same compiler as the source-compiler but different version, OR
(3) a dependency was compiled with the same compiler and version as the source-compiler but different compilation flags, OR
(4) a dependency has the wrong version

In linux workstations, apt/yum can usually figure out dependencies for software packages that exist in the official repositories and install them automatically when you install a package.

No such setup exists for advanced scientific sofware. Enter Spack.

Getting Started

0. Spack requirements

• Python3
• GNU C/C++
• make
• unzip/git/curl/wget

1. Download spack zip file (get .zip file link from github)

​​​​wget https://github.com/spack/spack/archive/refs/tags/v0.16.1.zip
​​​​unzip v0.16.1.zip
​​​​cd spack-0.16-1
​​​​source share/spack/setup-env.sh

Now spack is in your system path (echo $PATH) and you can run spack commands! Note that you have to do the source step each time you open a new terminal. (or you can add that line to .bashrc or .bash_profile)

A spack directory (.spack) will also be created in your home directory. This will store spack settings.

2. List packages available in spack

​​​​spack list

3. Get information about a certain package

​​​​spack info openmpi
​​​​spack versions openmpi
​​​​spack graph openmpi

4. Install a package

To install the preferred version of a package do: spack install package_name

​​​​spack install tcl

Wait for spack to install the package and its dependencies!

See where package is installed

​​​​spack location -i tcl

5. List installed packages

​​​​spack find
​​​​spack find --help
​​​​spack find -ldf

6. Load/Unload the package

​​​​spack load boost
​​​​spack find --loaded
​​​​spack unload boost

7. Uninstall/remove the package

​​​​spack find
​​​​spack uninstall boost
​​​​spack find

8. Uninstall remove ALL packages

​​​​spack uninstall -a

9. Install packages with sophistication

To install a specific version of a package from the list of versions available, do

​​​​spack install tcl@8.6.3

To install a specific version of a package with a specific dependency version, do

​​​​spack install tcl@8.6.3 ^zlib@1.2.8

To install a specific version of a package with a specific dependency version and a specific compiler

​​​​spack install tcl@8.6.3 ^zlib@1.2.8 %clang@6.0.0

See available compilers using spack compilers. If a compiler is not shown, use spack compiler add

​​​​spack compilers
​​​​spack compiler add
​​​​spack compilers

See what spack will install

​​​​spack spec tcl

10. Module files

​​​​ls share/spack/modules
​​​​cat ~/.spack/linux/modules.yaml
​​​​module avail

Edit/Create a modules.yaml file in .spack directory

​​​​modules:
​​​​  tcl:
​​​​    hash_length: 16
​​​​    projections:
​​​​      all:               '{name}/{version}/{compiler.name}/{compiler.version}'

And recreate the tcl module files

​​​​spack module tcl refresh

Introductions

Hola! I am Anirban Pal and I am leading an effort to bring high-performance computing to the West Texas A&M University campus, and facilitate campus-wide HPC use in research and academics.

I am teaching and developing graduate courses in Materials Science and HPC Applications for Engineers. I have used molecular dynamics, finite element methods and quantum chemistry calculations in my research. Currently, I am interested in developing computational methods for studying fiber networks and validating electrochemical models for battery performance.

Questions and Answers

In this section, you can post your questions and feel free to answer if you have it. Questions will be answered during or after the workshop.

1. Ask your question.
   • Here is an answer

---

Session Feedback

Use the link below to provide your feedback on the session:
Session Feedback Form