# fMRIprep LTS

## Oct 13th, 2022

Attendance: YC, TG, PB
* Major update by YC. [Link to google slides](https://docs.google.com/presentation/d/10e35C_tx_q9EyOSe_kaAerTkNvfsqUrWzA80OgpHuGw/edit#slide=id.g1f88252dc4_0_162)
* Ran a bunch of MCA replications of structural processing, on 7 T1-weighted images.
* Derived a test at each voxel -> p value. Also tried applying different FWE and FDR correction for multiple testing.
* Main experiment is "global null", leave-one-MCA-replication-out experiment.
* Varying the alpha threshold, the proportion of (false) positive detections follows the nominal value closely (YAY)
* With FWE correction, most replicates lead to no detection at all, as expected. PB: need to compute the empirical FWE to compare with the nominal value.
* FDR is more variable. PB: try to run same analyses with a bit of smoothing (fwhm = 3?).
* Couple of feasibility experiments have been planned.
* One is to compare the IEEE (baseline) to MCA distribution -> does not detect variations.
* Another: compare the outcome of one image to another image's distribution. Should lead to lots of detections. WIP
* Another: use a corrupted template for registration. Should also lead to lots of detections. WIP.
* Aim is a paper relatively soon. Looks like all the components are in place. The logic is completely generic and could be applied to any image processing pipeline, or really any software pipeline!
* Once the test is established, PB & TG will look into hiring an engineer (internship?) to complete the software library started by LT.

## May 4th 2022
Attendance: YC, LT, TG, YC
* LT is leaving for France :( Effective May 6th
* missing features for the app
* local run
* logging of tests (central stats?)
* have a clean set of datalad image repos in niprep
* in general move everything to nipreps
* Follow-up LT departure
* Maybe hire a Master student to extend the validation?
* Maybe transfer the LTS maintenance to new data platform at criugm - if it gets created
* Tests still don't work
* MCA more libraries (lapack et blas)
* smooth images prior to stats (nilearn's smooth_img)
* enable random seeds in the pipeline
* Idea: use MCA to do data augmentation in fMRI machine learning models

## Apr 13th 2022
Attendance: YC, LT

* Statistic results (https://nbviewer.org/github/yohanchatelain/fmriprep-reproducibility/blob/master/notebooks/stat_test_normal.ipynb)
* 64 MCA iterations of ds000256/sub-CTS201
* Parametric (gaussian)
* Still lot of false alarms (~10%)
* Non-parametric (min-max)
* ~3% false alarms => generate much more samples (20 cpu.year allows 1500 iterations)

## Mar 30th 2022
Attendance: LT, PB

* presentation of the new datalad version control on all test data
* discussion on pros and cons of using datalad run containers to keep track of test execution.
* preliminary consensus: use datalad only to keep track of reference data. Use pytest to run tests and the same mechanism as mriqc to aggregate results across users.
* for reference: API MRIQC https://mriqc.nimh.nih.gov/

## Mar 16th 2022
Attendance: YC, TG, PB, LT, CM

* Storage
* results of the preprocessing generated by YC is saved on compute canada tapes.
* challenge posting the outputs of fuzzy on OSF
* need to remove the workdir.
* LT prepared a datalad repo with bids convention for the archival on OSF.
* We will save only the T1w derivatives for now: one mean and one standard deviation image for each subject.
* How to push a DataLad dataset to OSF
1.

    OSF_TOKEN=<your_token> datalad create-sibling-osf --title 'BigBrain histogram' \
  --mode exportonly \
  -s osf-export \
  --description "This carefully acquired data will bring science forward" \
  --public

2.

    OSF_TOKEN=<your_token> git-annex export HEAD --to osf-export-storage

3. Make sure to make the dataset public through the OSF portal
* Compute canada outage
* should be ok transferring on cedar.
* Stats
* In practice, RFT "doesn't work": it is too liberal but we don't really understand why. Instead, the plan is to use a (more experimental) cross-validation approach: estimate the mean and standard deviation of the % of rejected voxels and adjust the p-value threshold from that.

## Feb 2nd 2022
Attendance: YC, TG, PB, LT

* Tolerance interval.
* Voxels aren't independent
* Correct for family-wise error (Bonferroni). But with this we'll pay the price of non-Gaussianity and low # samples. So don't go there yet.
* Don't go with non-parametric test if # of non-Gaussian voxels remains modest. We don't want to impose a computational debt for the package maintainers.
* Other possibility: make averages by ROIs. Using the ROIs produced by Freesurfer. For instance: look at average abs diff between voxels in region. ROIs could be considered independent.
* Do we want to use 95% as absolute interval? Where do we put the cut-off?
* -> check test by ROI?
* Other sanity checks:
* Generate more fuzzy samples (30)
* Use cases
* Software update
* Deployment on different hardware
* Simulate a template corruption (ask Basile for corrupted template or how to replicate corruption)
* Functional data: use Hao-Ting's nilearn code to process the images and design the test on this result (don't introduce noise in this analysis)

## Jan 10th 2022

Attendance: YC, LT

* statistics:
* normality testing
* mask for non-gaussian voxels
* non-parametric test for 5 samples ?
* fuzzy status:
* now work on fMRI data! Issue was that fsl-bet caught stderr output from verificarlo as errors.
* reference:
* update on make-reference (raw fmriprep outputs, bids anat mean/std)
* what alse to include in reference (func corr mean/std, masks for gaussian distributed voxels)

## Dec 21st 2021

Notes by TG and YC on next steps following our HBM abstract submission:

# Error maps
* Get error maps for the functional pipeline. Status: we have 5 fuzzy samples for 1 subject.
* Curiosity: check if error map resembles local SNR
* Trace the pipeline with Pytracer to understand where the error comes from

# Test definition
* Check if fuzzy samples are Gaussian (they're probably not)
* If they are not, find a way to define confidence intervals from non-Gaussian samples
* Sanity check: check that new fuzzy samples pass the test

# Experiments with test
* Does the test pass if random seed isn't fixed?
* Does the test pass if multithreading is enabled?
* Does the test pass if dependencies are updated?

# Other
* Clean GH repo and explain how users can test their pipeline

## Dec 15th 2021
Attendance: YC, TG, CM, LT

* Fuzzy outputs:
* Anatomical outputs instabilities mostly on the borders
* Error maps are available for the anatomical pipeline. 8 subjets.
* Tests will be implemented for 3 derivatives (native space, 2 versions of MNI template) x 8 subjets
* Mask subcortical gray matter regions of the MNI template
* https://templateflow.s3.amazonaws.com/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-06_atlas-HCP_dseg.nii.gz
* Overall we keep half of the precision
* For HBM
* TBD push outputs to osf and create datalad repo
* OHBM abstract:
* https://docs.google.com/document/d/1qE2W3qBhf_MZKto_Ywlg93Sesn5aMNAze45fU8J44V0/edit
* TBD fuzzy with numpy update (if available before deadline)
* TBD dataset descriptions
* Statistical reports:
* Relative differences (for anatomical) and pearson correlation (for fMRI)
* We want instead absolute comparison with fuzzy bounds for each voxel
* Current work can be used for fuzzy func ref generation

## Nov 24th 2021
Attendance: YC, LT
* YC:
* run fuzzy-container for anatomic only
* with IEEE => 100% reproducible with sequential mode (what was expected)
* with MCA => big difference related to masking (max std ~10^9 voxel-wise)
* next step to run for functionnal
* explanation on reproducibility error multithreading VS multiprocess:
* multiprocess is on independent task parralelism where multithreading happens at more fine-grained level (@markiewicz and @mathias you confirm ?)
* LT:
* big errors must come from registration: voxel inside mask VS outside mask (from max std)
* will create anat only slurm file for YC to experiment within our tool
* statistical test still on devloppment

## Nov 10th 2021
Attendance: PB, YC, LT
* LT
* has an app working for fuzzy (as fuzzy does not work yet, it's using multiple runs of IEEE). Quantifies correlation of time series across all pairs of replications, averaged in a brain mask, as well as html reports showing the results voxel by voxel.
* PB
* acquiring a better test dataset 100% feasible. We need a way to release them publicly. WIP (high priority because the same issue applies to cneuromod).
* re brainhack global, still unclear, will revisit on Nov 24th.
* YC
* test on anat only. Ran IEEE inside fuzzy, with 8 participants. Big variability in run length - one subject takes 5 days (!!).
* next step to include MCA.
* Then check anat+func.

## Oct 25th 2021
Attendance: PB, YC, LT
* LT: update on the fmriprep fMRI repro test
* fuzzy is broken atm. Yohan is going to look into it.
* simple reproducibility metrics -> PB and LT to draft.
* structural
* dice between reference / new brain mask
* max absolute difference relative to reference intensity in the reference mask
* functional
* dice between reference / new brain mask
* minimum correlation of activity at a voxel between test and retest in the reference mask
* timeline: have a functional app to test inter-os repro in the coming weeks. For fuzzy, unclear, because it's unclear what the problem is.
* move the app from simexp to nipreps?
* YC: plans for identifying reproducibility bottlenecks.
* To be discussed at a later point. We need the app and assess the magnitude of the problem first.
* PB: plans for a brainhack project on fmriprep_reproducibility.
* TBD. Depends if we have a working app soon. Will decide on our meetings scheduled Nov 24th.

## June 29 2021 (2d round)
Attendance: Loïc Tetrel, Chris Markiewicz

* CM: fmriprep team can also work on (github actions) tests
* LT: CI tests may not be ideal for a datalad repo environment?
* CM: It should accomodate the load (fmriprep running 10G containers)
* LT: fuzzy issues
* CM: `_fix_surfs7` - collecting precomputed outputs
* from https://fmriprep.org/en/1.1.4/_modules/fmriprep/interfaces/surf.html
* CM: you would better try re-running `bet` inside the fuzzy environment

## June 29 2021 (1st round)

Attendance: Loïc Tetrel, Yohan Chatelain, Tristan Glatard

* LT: reorganized tests to reduce scan time in subjects
* LT: parametrized tests
* LT: created a Makefile
* LT/TG: we could publish test results to (another) git repo
* TG: will investigate why fuzzy bet crashes

RuntimeError: Command:
bet /WORK/fmriprep_work/fmriprep_wf/single_subject_CTS201_wf/func_preproc_task_restbaseline_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/n4_correct/ref_bold_corrected.nii.gz /WORK/fmriprep_work/fmriprep_wf/single_subject_CTS201_wf
/func_preproc_task_restbaseline_wf/initial_boldref_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_bold_corrected_brain.nii.gz -f 0.20 -m

*

## June 15 2021
Attendance: Loïc Tetrel, Chris Markiewicz, Tristan Glatard, Mathias Goncalves, Yohan Chatelain

* LT: compared reproducibility with single-threading vs multi-threading vs multi-processing - controlling otherwise for random seeding. Multi-processing is completely reproducible (YAY), but multi-threading is not. MG: ANTS is probably to blame. See https://github.com/ANTsX/ANTs/wiki/antsRegistration-reproducibility-issues
* LT: still trouble running the fuzzy fmriprep. Next in line.
* PB: still need to make the code modular for the evaluation, and change the evaluation metrics.
* LT: shows first implementation of the test using pytest. CM suggests looking at https://docs.pytest.org/en/6.2.x/parametrize.html

## June 8 2021
Attendance: Loïc Tetrel, Chris Markiewicz, Yohan Chatelain, Tristan Glatard, Mathias Goncalves

* TG: will make a PR on shellcheck nitpicks in the bash scripts. Nothing major.
* LT: improved submission scripts
* LT: using options `--random-seed 1234 --fs-no-reconall --anat-only --skull-strip-fixed-seed --nthreads 1 --omp-nthreads 1`, results are exactly reproducible for both the anatomical and functional pipelines
* CM: `--skull-strip-fixed-seed` passes `--use-random-seed 1` to ANTs Atropos... cannot specify the seed directly
* LT: still running fuzzy (5 repetitions)
* How to show unchanged images?
* CM: GIF is lossy; niworkflows uses SVG ([link](https://github.com/nipreps/niworkflows/blob/2600e4fe18012a852a9169bfac5804d3ad789eba/niworkflows/interfaces/reportlets/base.py#L89-L110))

* TODO: compare multithreading implementation of fmriprep with fuzzy
* `omp-nthreads` is likely to affect the results but `nthreads` should not

* Tests: we will implement both a global and a voxelwise one. LT to implement a first version.

## June 1 2021
Attendance: Pierre Bellec, Yohan Chatelain, Loïc Tetrel, Mathias Goncalves, Tristan Glatard
Regrets: Chris Markiewicz

* testing reproducibility of the anat pipeline.
* LT: differences across runs relates to seeding the skull stripping step.
* LT: experiment will be done, now that the latest singularity images are ready
* TG: Singularity images were updated to use fmrirep 20.2.1. See https://github.com/SIMEXP/fmriprep-lts/pull/3
* TG: is each repetition really independent?
* LT: yes it should
* MG: you could check in the logs.
* LT: I have, and it restarts from scratch.
* Update on the container script
* BP suggests to try to control versions in these build files:
* https://github.com/nipreps/fmriprep/blob/7e17deaf05a27c577a8cfb0099b93c9883cc63ce/Dockerfile#L9-L18
* https://github.com/nipreps/fmriprep/blob/7e17deaf05a27c577a8cfb0099b93c9883cc63ce/Dockerfile#L73-L80
* PB: we will build a reference set of results from the existing container, then patch the container dockerfile, rebuild and confirm that the results don't change.
* possible solution ND freeze https://neuro.debian.net/pkgs/neurodebian-freeze.html Maybe overkill.
* after some debate, candidate solution would look like:
* add more control on package versioning.
* aim to rebuild periodically with newer packages (including ubuntu).
* systematically test for changes.
* what images and metrics?
* TG: mean and standard deviation
* PB for 4D: voxelwise correlation between time series, or percentage-of-baseline difference.
* LT: how do we test departure from reference distributions.
* we'll implement both a voxel-wise and volume-wise approach.
* in terms of images:
* preprocessed T1 and mask, as well as preprocessed fmri.
TODO:
* TG to have a look at the bash script
* https://github.com/SIMEXP/fmriprep-lts/blob/4e983f7cb5914bf336c54942ccc1663cb3b484e1/code/run.bash#L88-L109
* discuss (and vote?) the solution for maintaining the lts container
* contact BP for acquiring new test data

NEXTTIME:
* report
* how to make code cleaner and modular
* what outputs for the report
* report or reportlets?

## May 25th 2021
Attendance: Pierre Bellec, Chris Markiewicz, Tristan Glatard, Mathias Goncalves, Loic Tetrel
* TG did create PR for building a singularity container for lts with and without fuzzy:
* Building the container: https://github.com/SIMEXP/fmriprep-lts/tree/master/code/containers
* Container images: https://github.com/SIMEXP/fmriprep-lts/tree/master/envs
* https://github.com/SIMEXP/fmriprep-lts/pull/1 containers are stored on OSF.
* Usage: https://github.com/SIMEXP/fmriprep-lts
* -> should pull 20.2.1 instead of 20.2.0
* LT is giving an overview of https://github.com/SIMEXP/fmriprep-lts
* PB Q: where should all these repo live?
* Where to put fmriprep-preprocessed data
* PB: OSF
* CJM: Another option: gin.g-node.org
* CJM: BIDS-style outputs: [`--output-layout bids`](https://fmriprep.org/en/stable/usage.html#Other%20options) (see [nipreps/fmriprep#2303](https://github.com/nipreps/fmriprep/pull/2303), [neurostars #18492](https://neurostars.org/t/sharing-nested-bids-raw-derivatives-in-a-datalad-yoda-way/18492))
* PB: need to decide how to name/organize the different runs
* MG: would be useful to have more slurm options exposed.
* CJM: reproman would be the way to implement a generic tool
* https://github.com/ReproNim/reproman
* TG: maybe decouple the data processing from the test, to better deal with crashes.
* LT: is it ok to develop our code base for linux only? or is there some form of windows and osx support for fmriprep? CJM: bash is a reasonable requirements.
* TG: we should use a testing framework to implement the tests, for instance:
* pytest if Python
* [bats](https://bats-core.readthedocs.io/en/latest/index.html#) if bash
* fmriprep's testing framework (link?)
* Next steps
* investigate discrepancies on the anatomical report
* add CJM to compute canada -> PB to send directions
* CJM to investigate the environment variables of the different jobs.
* implementation of the report
* TG and Ali to review the bash scripts
* also compare fmriprep command line arguments used in Ali and TL's experiments
* Next week
* discuss which images to look at in the tests
* discuss which metrics to quantify variations in the test
* possibly Basile Pinsard update on the container build

## test data with NMIND
**Attendance**: Greg Kiar, Pierre Bellec, Audrey Houghton, Xinhui, Yohan Chatelain, Tristan Glatard, Loic Tetrel, Mathias, Steve Giavasis, Sydney Covitz

* GK: check this issue: https://github.com/nmind/hackathon2021/issues/4
* GK also check this issue: https://github.com/nmind/hackathon2021/discussions/16
* AH: selecting some subjects for ABCD and NKI Rockland. A next step is to run fmriprep on these data, and check changes on differences. Those are quality subjects.
* SC: Q: are there tests done with T1+FLAIR. A: Mathias - no.
* GK: how would you test the LTS for low quality data. PB: our aim is to test if an install is proper, not that the pipeline is robust. So a single subject should be enough.
* PB: for test set: partial field of view (ICBM aging F1000). Hyper-intensity (oulu adhd200). atrophied brains (adni). multi-scanner (simon).
* Mathias: pretty sure the FLAIR is only used with the surface reconstruction so shouldn't effect the fmriprep pipeline much
* would be useful to have infants for a project like https://github.com/nipreps/nibabies
* LT: would you need public data for evaluating data? GK: not necessary. You can have a centralized evaluation of the tests, or a partial evaluation just with the public data.
* PB: what about having a very small dataset to run quick tests? GK: ultimately this type of test is quite different from processing real data. PB: it would be useful only to catch crashes and changes in outputs. GK: not on scope at the moment.

## May 11th, 2021: update and test data

### varia (container image hosting and generation)
**Attendance**: Chris Markiewicz, Mathias goncalves, Ali Salari, Loic Tetrel, Pierre Bellec, Tristan Glatard (late)
* LT: where are we at with the fuzzy container repo? AS: no progress yet. Next week. CM: singularity hub is closing down. Consensus: stand-alone datalad repo, put the singularity images on OSF (which has a datalad remote).
* CM: may be worth saving the apt and pip cache and archive that for the build of the LTS. nd_freeze? https://neuro.debian.net/pkgs/neurodebian-freeze.html Talking to Yarik?
* CM: Looking through our dockerfile it does look like we separated the Python from non-python packages pretty well, so Greg's suggestion seems like a reasonable fallback.

### test data selection
* LT: rationale for selecting data
* children, young adults, older adults
* multiparametric (T2w check, FLAIR check, could not find SBREF... BP -> https://openneuro.org/datasets/ds001399/versions/1.0.1 or CM mentions ds000031, ds000244, ds001178, ds001399, ds001417, ds001734, ds001740, ds001771, ds001818, ds001978, ds002147, ds002278,ds002316)
* fieldmap (all different)
* MG: multi-echo
* BP: every vendors?
* CM: lesion masks.
* CM: create one for one subject, with absolutely everything.
* PB: three use cases: (1) test all aspects of the pipeline on one subject; (2) test the pipeline on all kinds of different subjects. (3) same as 1, but tiny for continuous integration. CM: may need to have dedicated test infrastructure.
* TODO: put together some docs on the test data.

## May 5th, 2021: preliminary results and planning

**Attendance**: Tristan Glatard, Yohan Chatelain, Greg Kiar, Ali Salari, Loic Tetrel, Pierre Bellec

**Notes**:
* [neurostars thread on cross-run differences](https://neurostars.org/t/differences-between-fmriprep-runs-on-same-data-what-causes-them/18543/3)
* LT: investigated reprodudicibility
* selected some data on open neuro
* implemented a pure functional workflow with a fix anatomical workflow.
* also implemented pure anatomical
* running without MCA, but is possible with MCA
* TG:
* Ali focused on the anatomical pipeline
* re-using options in the neurostars post: `fmriprep --random-seed 1234 --fs-no-reconall --anat-only --skull-strip-fixed-seed --omp-nthreads 1`
* perfectly reproducible results with fixed seed and no MCA perturbations
* it was run on local cluster
* Used one session of the SIMON dataset
* Evaluated stability via MCA on single subject (low sig. digs: ~3)
* low significance appears to be due to subtle (but impactful) registration differences
* still some small impact of MCA
* see results here: https://github.com/glatard/fuzzy-fmriprep/blob/main/sigdigits.ipynb
* LT:
* present his project https://github.com/SIMEXP/fmriprep-lts
* TG:
* we should have a pass/fail test
* users need a single run
* GK:
* it is going to be hard to rebuild the same container
* this test could be integrated for developers in CI
* PB: this is a great idea, but would require different data for different use cases
* TG: CI could just grab outputs. Let's focus on the tests first, leave CI for later.
* GK: initiative of Damian Fair, Ted S and Mike M to avoid duplications. Will create a test set. Greg will be our champion so far. NMIND (this Neuroimaging Method Is Not Duplicated)
* TG: should it be part of standard fmriprep tests? PB: no, too heavy.
* GK:
* fmriprep already uses Sentry to integrate some basic testing/logging: https://sentry.io/welcome/
* We could bake in the test for comparing results to the stored MCA-derived mean/variance estimates
* Would be a great way to integrate our tests to the fmriprep existing environment
* PB: how can we find parts of the pipeline which are more variable?
* GK: we could replace inputs of each step by pre-generated ones.
* PB: this would be awesome, but requires lots of data.
* GK: could be done for developers.
* TG: other tracing methods could be used.
* TG: maybe we should focus on the simple test comparing with final outputs.
* PB
* good test sets (GK)
* decide target and performance measure for a regression test
* next decide on structures for the repos
* next decide on structures for the reports and tests
* other issue is to track cleanly our containers
* GK suggests using staged container build
* TG and team to build a datalad repo with fuzzy fmriprep image.
* https://handbook.datalad.org/en/latest/basics/101-127-yoda.html
---

## April 21th 2021: discussion on the origin of variations in fmriprep

### Attendance

Tristan Glatard, Yohan Chatelain, Chris Markiewicz, Greg Kiar, Ali Salari, Mathias Goncalves, Loic Tetrel, Pierre Bellec

### Agenda
* round-table discussion.

### Minutes
LT: we should focus on the tool to quantify instabilities.
TG: if there are random variations even when fixing the seed, then adding numerical noise is going to get drowned in those variations.
CM: Good first step would be to do a full run, and then re-run each step one by one with fixed seed to narrow where variations occur.
PB: maybe first step would be to re-run it with fixed seed, and also try to control for the anatomical variability.
PB+TG: metrics number of digits? correlation? differences and variance?
GK: dataset? HNU1, look at different CoRR sites.

PB: provided we identify sources of instabilities related to seeds, what do we do about it?
CM: some variations may be reasonable.
GK: would need to assess if the tool sometimes diverges, or if it's "expected" variability.
CM: could try to optimize the registration.
TG: boostrap averaging?
TG: compare solutions.
PB: consider parcel-based stability metrics as well?
CM: smooth data? What smoothing level is required to get to something acceptable?
PB: who would like to be directly involved in this work:
* TG and YC

### TODO
[x] create a mattermost channel for the project
[ ] understand better reproducibility with fixed seed (with / without fixing T1 processing)
[ ] decide on data & metric