# running ciftify steps on UK biobank derivatives

## trick for mounting the ukbiobank data to a kimel machine

(note - is mounted to home so only works for one machine)
-but good for being able to open images and html files inside directories - and loading connectome-workbench

mkdir ~/ukbiobank_imaging
sshfs <username>@scclogin.camhres.ca:/external/rprshnas01/external_data/uk_biobank/imaging ~/ukbiobank_imaging

## locations for fMRI data on the SCC

CAMH/Kimel has it's own access to the UK biobank - which is mounted to a shared folder on teh SCC

**For your actual work - you will need to use the Park lab's UK biobank credetials to download the data directly to Niagara**

UKbiobank participants outputs (full)

The outputs:
`/external/rprshnas01/external_data/uk_biobank/imaging/brain/nifti/fmri_task_based/nifti/data/<subid>/fMRI/tfMRI.feat`

We will test using the fMRI from the 20 participants dataset.

`/external/rprshnas01/external_data/uk_biobank/imaging/brain/nifti/test_sample_n20`

## Note re running ciftify container

the WARNING message -

`[ciftify.utils] WARNING: setNativeLocks failed: Resource temporarily unavailable`

Shows up a lot - this is only a warning - it usually is working just fine.

## Erin's Notes from looking in the fMRI folders

looks like everything is in native space.
- instead of raw sbref - best registration target is the example_func.nii.gz file
- filtered_func_data is the full timeseries
-zstats are the zstat files for contrasts

**Need to also check on wether volume based smoothing was applied..**

## fMRI mapping plan - if not too smoothed

- idea - ciftify for reals the filtered_func_data - that
s the preprocessed - use the sbref option

- then add a step that manually transformed the zstats to mni -
- then ciftify a nifti?

/well/win/projects/ukbiobank/fbp/bb_FSL/bin/susan prefiltered_func_data_thresh 4700.42871075 2.12314225053 3 1 1 mean_func 4700.42871075 prefiltered_func_data_smooth

susan
-input prefiltered_func_data_thresh
- bt (brightness threshold) 4700.42871075 -
- dt (spatial size - sigma) - 2.12314225053
- dim - 3 (i.e. three dimensional)
- use_median - 1 - i.e. yes

paper says the tfMRI was smoothed with 5mm FWHM

# A Note re: smoothing

Therefore 3-dimensional 5mm FWMH smoothing was definately applied in teh FSL task pipelines - **THIS IS DEFINATELY NOT IDEAL** - because it means that some smoothing (i.e. mixing of data) NOT along the surface is in the data.

*But for the time being we will move on to the next step instead of recalculating the contrasts.*

## the code for running one example subject on the CAMH SCC

## load the singularity module
module load tools/Singularity/3.8.5

singularity_container=/external/rprshnas01/tigrlab/archive/code/containers/FMRIPREP_CIFTIFY/tigrlab_fmriprep_ciftify_v1.3.2-2.3.3-2019-08-16-c0fcb37f1b56.simg


singularity exec ${singularity_container} ciftify_recon_all --help

EID="1104625"
SUBID=${EID}_20263_2_0
fs_dir=/external/rprshnas01/external_data/uk_biobank/imaging/brain/nifti/t1_surface/data/
# fs_dir=/external/rprshnas01/external_data/uk_biobank/imaging/brain/nifti/test_sample_n20/t1_surface/ # while is has less people the org is different
feat_dir=/external/rprshnas01/external_data/uk_biobank/imaging/brain/nifti/test_sample_n20/tfmri_nifti/tfmri_nifti/${EID}_20249_2_0/fMRI/tfMRI.feat
ciftify_out=/external/rprshnas01/tigrlab/scratch/edickie/ukbiobank_ciftify_test/
freesurfer_license=/external/rprshnas01/tigrlab/quarantine/freesurfer/6.0.0/build/license.txt

mkdir -p ${ciftify_out}

## this step converts the freesurfer (structural) outputs to gifti surfaces 
singularity exec \
  -B ${ciftify_out}:/ciftify_dir \
  -B ${fs_dir}:/fs_dir \
  -B ${freesurfer_license}:/fs_license.txt \
${singularity_container} ciftify_recon_all \
  --ciftify-work-dir /ciftify_dir \
  --fs-subjects-dir /fs_dir \
  --fs-license /fs_license.txt \
  ${SUBID}

## this creates QC images for the surfaces
singularity exec \
  -B ${ciftify_out}:/ciftify_dir \
${singularity_container} cifti_vis_recon_all subject \
  --ciftify-work-dir /ciftify_dir \
  ${SUBID}

## this step will project the cope maps to the surface (.dtseries.nii) files
for cope in `seq 1 5`; do
singularity exec \
  -B ${ciftify_out}:/ciftify_dir \
  -B ${fs_dir}:/fs_dir \
  -B ${feat_dir}:/feat_dir \
  -B ${freesurfer_license}:/fs_license.txt \
${singularity_container} ciftify_subject_fmri \
  --ciftify-work-dir /ciftify_dir \
  --FLIRT-to-T1w \
  --func-ref /feat_dir/example_func.nii.gz \
  /feat_dir/stats/cope${cope}.nii.gz \
  ${SUBID} \
  cope${cope}

## this step will make QC images of the projected maps
singularity exec \
  -B ${ciftify_out}:/ciftify_dir \
${singularity_container} cifti_vis_fmri subject \
  --ciftify-work-dir /ciftify_dir \
  cope${cope} ${SUBID}  

done
  


## this last bit can be run after all run once after running all subjects it creates index pages so that they are easier to work with 
singularity exec \
  -B ${ciftify_out}:/ciftify_dir \
${singularity_container} cifti_vis_recon_all index \
  --ciftify-work-dir /ciftify_dir

singularity exec \
  -B ${ciftify_out}:/ciftify_dir \
${singularity_container} cifti_vis_fmri index \
  --ciftify-work-dir /ciftify_dir  
  

To do:

- [ ] adapt this to run from zip files downloaded directly to Niagara (using Ruyi access rights)
- [ ] adapt this to run in parallel on Niagara compute cluster
- [ ] scale the whole process to run (in chucks?) on UK biobank - ten of thousands - we will hit scratch storage quotas

Note - the ciftify container can be found on scinet at ``/project/a/arisvoin/edickie/containers/fmriprep_ciftity-v1.3.2-2.3.3.simg``
or
``/scinet/course/ss2019/3/5_neuroimaging/containers/fmriprep_ciftity-v1.3.2-2.3.3.simg``