# Guide for running Qsiprep and Freesurfer pipelines from scratch (ComputeCanada-ready) ## Requirements - T1w images - DWI images - Qsiprep & Freesurfer through docker/singularity (recommended, super easy to deal with) - CuBids - BIDS-validator ## Folder tree - Connectome_stuff - BIDS_dir - Derivatives - Qsiprep - <Includes_reports> - Freesurfer - Qsirecon - Singularity_images - work_dir - Custom_atlas - <Niftis_for_the_Parcellation> - atlas_config.json - custom_workflow.json - FSlicense.txt - qsiprep_recon.sh - # Step 1: BIDSifying Qsiprep is a BIDS-app, which follows certain standards in organising the directory structure and naming scheme. Even though Qsiprep triggers bids-validation internally using PyBids, it can be handy to validate it manually. In the case of the content being not BIDS-compliant, CuBids can be used to curate and BIDSify. The details for both of are provided below. # Step 2: Preprocessing Assuming the content is BIDS-ready, it is time for Qsiprep, which builds the pipeline from the available data (and meta-data) to preprocess the images for distortion correction, registration and so on. A typical preprocessing pipeline can be found at https://qsiprep.readthedocs.io/en/latest/. First, installation through docker/singularity image is highly recommended; keep in mind that, starting from Qsiprep 0.15, Freesurfer has been removed from the container. To do so, download the singularity image using `singularity build qsiprep-<version>.sif docker://pennbbl/qsiprep:<version>`. It will download the necessary content and store it in `/home/<user>/.singularity/cache/`. If the HOME is nearing full, HOME directory can be set to somewhere else through `export SINGULARITY_CACHEDIR=<new path>` before pulling the image. At the end, `qsiprep-<version>.sif` image will be created and the content of the sif image can be accessed through `singularity shell qsiprep-<version>.sif`. Second, initiating the Qsiprep via `singularity run qsiprep-<version>.sif BIDS_dir/ Derivatives/qsiprep/ participant -w work_dir`. More details are available at https://qsiprep.readthedocs.io/en/latest/usage.html. At the end preprocessing, Qsiprep generates a report detailing out what steps have been applied and reports errors if any. There are instances where everything would be run without errors, but reports won't be generated. If that is the case, you might want to try specifying the work directory as absolute path (it worked for me on ComputeCanada). # Step 3: Surface Reconstruction It can be done via fMRIPrep, as well as via Freesurfer. fMRIPrep is available in containers, while Freesurfer isn't (AFAIK). Freesurfer 5.3.0 is readily available on ComputeCanada machines, but`parallel` support was not yet implemented (released in 2013). To use the latest versions, either you can contact ComputeCanada folks to ask them to install them in their system directly, which makes it accessible to everyone (I believe), which also saves space from your quota if had to be installed locally. But to install locally in the `/home/<user>/` folder, first download the freesurfer tar archives through `wget https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/7.4.1/freesurfer-linux-centos8_x86_64-7.4.1.tar.gz` the other variants are here https://surfer.nmr.mgh.harvard.edu/fswiki/rel7downloads. If needed, wget can be installed through `pip install wget --no-index`. EasyBuild is what ComputeCanada use to install all their softwares, we can use the same. Installation is done through "easybuild recipe", the recipe for Freesurfer can be found at https://github.com/ComputeCanada/easybuild-easyconfigs/tree/computecanada-main/easybuild/easyconfigs/f/FreeSurfer. ComputeCanada's documentation is pretty good - https://docs.alliancecan.ca/wiki/FreeSurfer. As mentioned there at CC, it is necessary to complete the free registration to obtain the license: https://surfer.nmr.mgh.harvard.edu/registration.html. Once done, download the license file and bind it to Qsiprep # Step 4: Qsiprep recon Workflows for the reconstruction are elaborately described in here: https://qsiprep.readthedocs.io/en/latest/reconstruction.html#custom-reconstruction. This is specified via `--recon_spec` argument -- the json file if it is a customized workflow or just the workflow title directly. Some require Freesurfer derivatives, in which case it is mandatory to supply also the license file. License can be obtained via https://surfer.nmr.mgh.harvard.edu/registration.html. Once done, download the license file and bind it to Qsiprep. Several ways to do that, I simply pointed to the license directly via the CLI. `Singularity run qsiprep.sif BIDS_dir/ derivatives/recon_OP/ --recon_input derivatives/qsiprep/ --recon_spec atlas/qsirecon_atlases/customized_workflow.json/ --fs-license-file FSlicense.txt --freesurfer-input derivatives/freesurfer/` # Life-friendly Slurm flags - Want to be notified throug email id about all the happenings of your job ? `--mail-user= emailid@email.com` `--mail-type=ALL` - Do you keep staring at the terminal, but want to quit, but while still want to peek at the verbose later ? Move to bash scripts, `#SBATCH --output=S-subtrain-%j.out` and `#SBATCH --error=S-subtrain-%j.err` - Big deadline coming up, you are super dependent on the machines and don't want to take a chance with Slurm allocation ? Make several job requests with delayed start such that there are always machines for even the next days `--begin=Xh+now` - You generate data from one job, which is used by a subsequent job and you don't want to keep track and manually start the subsequent job ? `--dependency=afterok:jobid` Tips: Likely that Singularity container won't mount `projects/` (not `project` btw!) and `scratch/` directories. Thus, the input BIDS is expected at `HOME` as well as the output and working directory contents are written in `Home`. `Home` is supposed to be for minimal usage, capped at 50GB. So to deal everything in bigger disks, Singularity offers ways such as binding the directory to the container or mount the respective disks. I personally used the latter through `Singularity run -B /projects/def-<PIname>/<username>/:/mnt/ qsiprep.sif /mnt/BIDS_dir /mnt/OP_dir/ .....`. This will mount your `projects/` disk, and you can write to it directly as everything runs inside the container.