TrainTrack Containers talk

Steffen Bollmann and Thom Shaw

Table of Contents

How to access course material

https://github.com/thomshaw92/TrainTrack_containers_2020

git clone https://github.com/thomshaw92/TrainTrack_containers_2020.git

Prerequisites

Course code

We will be using Docker for Windows and Singularity on Linux.
This code may not work directly on your OS - so please ask the TAs or helpdesk if you run into issues.

1) Build a simple docker container

Docker pull and run

First let's try the simplest example and pull the hello-world of docker:

docker pull hello-world

and then try running it:

docker run hello-world

(just docker run on its own would pull the image as well if it's not there yet)

Images

where are the Docker images located?

docker images

in windows all docker images are stored in a single hyper-v virtual machine disk at: C:\ProgramData\DockerDesktop\vm-data\DockerDesktop.vhdx

Caching

Find image ID of hello world

remove the image

docker rmi ${image_id}

or if that doesn't work

docker rmi -f ${image_id}

fMRIprep example

First let us check out our example dataset:
The dataset is just a random Open Neuro dataset. Feel free to pull it via the web client or via AWS

https://openneuro.org/datasets/ds000205/versions/00001

https://aws.amazon.com/cli/

aws s3 sync --no-sign-request s3://openneuro.org/ds000205 ds000205-download/

Then we pull fMRIPrep (this will take a while!)

docker pull poldracklab/fmriprep:latest

NB you can use the docker fmriprep wrapper but we will be interfacing directly with the Docker image.
check in https://fmriprep.readthedocs.io/en/stable/docker.html
https://hub.docker.com/r/poldracklab/fmriprep/tags for the latest versions of fmriprep

Running fMRI prep container in Windows:
(The data here is in ${HOME}/data - you may need to adjust yours)







docker run -ti --rm `
-v $HOME/data/BrainHackOHBM2020:/data:ro `
-v $HOME/data/derivatives:/out `
-v $HOME/data/work:/work `
poldracklab/fmriprep:latest /data /out/fmriprep-latest `
participant -w /work

If you are on Linux: check fMRIPrep website https://fmriprep.readthedocs.io/en/stable/docker.html !

Run Singularity on an HPC

If your cluster supports running Singularity…
First we need to set up singularity.
Most HPCs will use lmod to manage software

module load singularity/3.5.0

Then pull the same image from DockerHub.

Singularity will handle the conversion from docker and you don't need root access to do this:




mkdir ${HOME}/images
singularity build ${HOME}/images/fmriprep-latest.simg \
docker://poldracklab/fmriprep:latest

The command to run singularity is very similar to Docker.
Check the fMRIPrep website for more details.





singularity run --cleanenv ${HOME}/images/fmriprep.simg \
    path/to/data/dir path/to/output/dir \
    participant \
    --participant-label label

2) Build a Docker file

Here is a Docker file to build an itksnap container



































FROM ubuntu:16.04

LABEL maintainer="Thom Shaw"
LABEL org.label-schema.maintainer="Thom Shaw"

ENV PATH="/opt/itksnap/bin/:${PATH}"
ENV LD_LIBRARY_PATH=/opt/itksnap/lib/:${LD_LIBRARY_PATH}

RUN apt-get update -y \
&& apt-get install -y \
wget \
libglu1 \
libcurl4-openssl-dev \
libsm6 \
libxt6 \
libfreetype6 \
libxrender1 \
libfontconfig1 \
libglib2.0-0 \
libqt4-dev \
libgtk2.0-dev \
\
&& wget -O itksnap.tar.gz 'https://sourceforge.net/projects/itk-snap/files/itk-snap/Nightly/itksnap-nightly-master-Linux-gcc64-qt4.tar.gz/download' \
\
&& tar -zxf itksnap.tar.gz -C /opt/ \
&& mv /opt/itksnap-*/ /opt/itksnap/ \
&& rm itksnap.tar.gz \
&& useradd -m -s /bin/bash itksnap

USER itksnap

COPY --chown=itksnap:itksnap UserPreferences.xml /home/itksnap/.itksnap.org/ITK-SNAP/

CMD ["itksnap"]

Build the image from Dockerfile

If you have Docker installed, you can try building this docker file now!
Copy this file (or use the one from the github
https://github.com/thomshaw92/TrainTrack_containers_2020/blob/master/2_build_itksnap_docker/Dockerfile)
Save it as 'Dockerfile' in its own folder.

Run:

docker build ./folder

Use you new Docker image locally

This one is a little bit tricky because we are doing a graphical application. So I will show how it is done quickly and if you want to try it yourself ask the TAs or HelpDesk after this talk :)

Open Power Shell

 docker run -it thomshaw92/itksnap bash
 export DISPLAY=(your display number)
 itksnap

3) Using NeuroDocker

Sometimes, building your own docker container is too hard. So programs like NeuroDocker from the nice people at ReproNim can really help!
We will be building this on HPC at UQ

Load the latest singularity

#!/usr/bin/env bash
module load singularity/3.5.0

Install neurodocker

pip install --user neurodocker
pip install git+https://github.com/stebo85/neurodocker@add-itksnap

We are using a pull request as our base install of neurodocker as it contains ITKSNAP

Define version and tool to be built

toolName='itksnap'
toolVersion='3.8.0'
buildDate=`date +%Y%m%d`
imageName='itksnap'

Use NeuroDocker to generate singularity recipe

generate singularity \
--base ubuntu:16.04 \
--pkg-manager apt \
--${toolName} version=${toolVersion} \
--env DEPLOY_PATH=/opt/${toolName}-${toolVersion}/bin/ \
--entrypoint /opt/${toolName}-${toolVersion}/bin/itksnap \
--user=neuro \
> Singularity.itksnap

Build singularity container locally (WSL2)

sudo singularity build ${imageName}_${buildDate}.sif ./Singularity

OR if you don't have root access…

Build Singularity remotely

Either go to
https://github.com/singularityhub/singularityhub.github.io/wiki
https://cloud.sylabs.io/builder

Or build online via the terminal

 singularity remote login

You will need to generate an API token, the instructions will appear on screen.

Then use

 singularity build --remote ${imageName}_${buildDate}.sif ./Singularity.itksnap

Run your new Singularity image

(in mobaxterm to use X11 server)

singularity exec itksnap_20200614.sif itksnap

Use software inside a container without changing your scripts

https://github.com/thomshaw92/TrainTrack_containers_2020/blob/master/3_build_neurodocker/show_transparent_Singularity.md

Some useful websites:

For more details, please see:
https://docs.massive.org.au/M3/slurm/slurm-overview.html

Shameless plug for our research

Longitudinal Automatic Segmentation of Hippocampus subfields using multi-contrast MRI

https://github.com/thomshaw92/LASHiS

FAQ and Feedback

Need any help with a particular part? Leave a comment!

tags: Docker Singularity Containers Data analysis OHBM BrainHack
Select a repo