# Training Schedule + Notes

## House Keeping

E-mail: [e-mail]

Important Links:
- MRI tutorials: https://kcniconfluence.camh.ca/display/ED/Neuroimaging
- Naming Convention https://docs.google.com/document/d/1YF1PaA6ICoQnw711OJl3LhLk7PSAzsQb/edit
- Agenda for Meetings: https://docs.google.com/document/d/1KlTu_TopyAlDCtTsplHNZfSQFo3ouQopdH1VF0BfOgk/edit#
- QA Dashboard: http://srv-dashboard.camhres.ca/

- [ ] REDCap account access - Connect with Emily/Michael from WashU
- [ ] NDA account access: https://nda.nih.gov/. Aristotle may need to sign off access grant
- January 15
- July 15
- Coordinating with the WashU site, Pittsburgh
- Every upload the NDA changes their requirements for how the upload should be structured. You'll need to update scripts to follow scripts.
- Uploads be coordinated at least 1.5 month advance
- [ ] XNAT account access:
- [ ] [XNAT Training](https://kcniconfluence.camh.ca/display/NPP/XNAT+Introduction)
- Alex Barton is the XNAT admin, can fast-track requests for accounts and error fixes
- [ ] MODSOCCS
- [ ] OPTIMUM

## Resources for Neuroimaging Training
- [Carpentries Collection](https://kcniconfluence.camh.ca/display/ED/Neuroimaging)

## OPTIMUM Study
[OPTIMUM Study Overview](https://docs.google.com/document/d/1KlTu_TopyAlDCtTsplHNZfSQFo3ouQopdH1VF0BfOgk/edit?usp=sharing)

- [ ] General Info
- Longitudinal study
- Baseline recordings, 6 and 24 month follow-up

- [x] Kimel lab data organization and flow
- Scanner acquisition
- XNAT upload
- Archive ingestion
- Data naming conventions:
- Datman
- KCNI
- Save files to Scratch, Archive and Projects
- Scratch: Temporary, output data, snapshot every month
- Projects: Documents and scripts to generate data
- Mostly git repos
- Data is in scratch, code is in projects
- Archive: Production version of code
- Github everything
- [ ] MRI Data
- Common folders in each study folder
- bin, runs in background
- docs, self-interest/learning
- Note: UP coil elements, specific parameter for UP patients (settings were not kept the same)
- Dicom
- folder refers to a scan viit
- Dicom file has all scanner information in the header (compare to gold standard to ensure parameters remain the same)
- Nii files
- Preferred format, contains everything in one file
- Scans done in MRI (in this order)
- T1 - anatomical scan
- T2 - same but inverted colors to highlight damage
- T1 and T2 - NORM: preferred scans
- fMRI DAP and DPA
- corrects for distortion
- RST - resting state
- FACES - Task, not 100% necessary
- But, needs to be indicated in Redcap that the task was omitted
- FLAIRS - white matter lesions
- Diffusion MRI or DTI
- To view data, use fsl
- fsleyes or fslview in linux
- .nii.gz - niifty file
- .json has much more header info
- DICOM - has all header information
- QA done twice a week preferrably, Tuesday and Thursday

- [x] XNAT
- FACES Task
- manage files
- resources and behav has the task files
- Check to see these
- one ends in .edot2 and the other is .txt
- Can check on the dashboard
- Need to follow naming convention when uploading to XNAT
- That is on the uploading end
- If messed up, everything breaks
- If mislabeled, missing scan doesnt mean its not there, just mislabeled on XNAT
- on dashboard
- Search up the patient ID, can see if it isn't there or labeled wrong
- We have to fix the data if mislabeled
- How to fix label:
- Go to subject, click on edit
- Then can rename the subject
- DOUBLE CHECK THIS STEP TO MAKE SURE
- Fix it and email them to ensure that they do not make mistake again

- Dashboard
- check dashboard
- if on xnat, thats good
- if not, email
- Then check if faces is there
-
- [ ] Dashboard QA
- Pittsburgh caveat
- [ ] Update IgnoreHeaders
- Issue handling

- [ ] Data Processing Pipelines
- pipelines/ organization
- [in_progress/ organization](https://hackmd.io/ufM3XaUCTxGDWZFLqzdexA)
- OPTIMUM baseline data current status
- [ ] Set up README.md to reproduce pipeline outputs
- Running pipelines w/TIGR-PURR, MRIQC sample

- [ ] OPTIMUM Obligations
- Biweekly meetings for Internal CAMH meetings and PI Meetings
- Recruitment tracking report
- [ ] https://github.com/TIGRLab/optimum-tracking
- Bi-annual DSMB report - process
- https://github.com/TIGRLab/optimum-dsmb
- [ ] Send Yar the previous Open/Closed report
- [ ] SPECIAL NOTES: Send DSMB closed report to Phil Miller before everyone else. Give weeks before deadline
- [ ] warren.d.taylor@vanderbilT: exclude from DSMB closed report
Activate to edit the task name

- Bi-annual NDA upload - process
- [ ] Forward template e-mail
- [ ] OPTIMUM Blood database

- Other on-going OPTIMUM activities:
- REDCap clinical data cleaning, being led by Nicole Schroer. Minimal involvement from our side, but good to stay in the loop
- Baseline data papers:
- Involves baseline data for OPTIMUM which is prepared and published into the archive

## DSMB Resolution Guide
- date_cutoff: 2023-01-01

Run `dbt test --profiles-dir . --store-failures`
- `--store-failures` creates a SQLite database called `main_dbt_test__audit.db` that stores a record of every failure in the `test` command. This is one of the databases you'll interrogate when resolving issues in the DSMB pipeline.

When you run dbt test, it'll output a series of messages, **1 for each class of error**. Here's an example

16:28:44  
16:28:44  Failure in test test_step_1_randomization_appears_atleast_in_arm_8 (tests/test_step_1_randomization_appears_atleast_in_arm_8.sql)
16:28:44    Got 2 results, configured to fail if != 0
16:28:44  
16:28:44    compiled Code at target/compiled/dsmb/tests/test_step_1_randomization_appears_atleast_in_arm_8.sql
16:28:44  
16:28:44    See test failures:
  ---------------------------------------------------------------------------------------
  select * from main_dbt_test__audit."test_step_1_randomization_appears_atleast_in_arm_8"
  ---------------------------------------------------------------------------------------

This block tells you:
1. Which test failed: `test_step_1_randomization_appears_atleast_in_arm_8`
2. How many failures occurred: 2
3. Which query was used to run the test: `target/compiled/dsmb/tests/test_step_1_randomization_appears_atleast_in_arm_8.sql`
4. How to look at the test result: `select * from main_dbt_test__audit."test_step_1_randomization_appears_atleast_in_arm_8"`

Typically the *name of the test* can tell you the nature of the issue. In this case, someone who was randomized to step 1 does not appear in Arm 8 (the REDCap arm containing all step 1 randomizations). Instead they may have appeared in Arm 7. The first step is to take a look at the test result. This can be done in 1 of 2 ways:

1. `sqlite3 ../data/dsmb.db.sql < target/compiled/dsmb/tests/test_step_1_randomization_appears_atleast_in_arm_8.sql`
2. Use the `main_dbt_test__audit.db` database: `sqlite3 ../data/main_dbt_test__audit.db select * from test_step_1_randomization_appears_atleast_in_arm_8`
- Notice we removed the `main_dbt_test__audit.db.` bit from the query, the prefix is a DBT implementation detail that needs to be removed for SQLite to work properly

The result of this is:

LA20011
LA20037

These are the two `record_id`s that supposedly are in Arm 7 but not Arm 8 and happen to be randomized to Step 1. There are many avenues of investigation, including (but not limited to):
1. Opening up the REDCap database:
2. Confirming the randomization status in the **Eligibility Database** which contains Step 1 randomizations
3. Look at the **Participant Tracking Form** for these records which contains additional commentary on the participants randomization

2. Querying the `dsmb.db` database, check the `dbt` documentation for information about each table.

You can use this to confirm that the test is correctly picking up the issue. In the case of LA20011 and LA20037 these both have explanations in the **participant tracking form** suggesting tha they were originally considered for step 1 (hence randomized) but ended up switching to step 2. The choice to place the participant in Arm 7 or 8 is a bit arbitrary at this point, so we can safely ignore these.

### Known Errors during DBT Test

There are **9** known errors:

1. `not_null_eligibility_step_1_randomizations_source_record_id`
- There is 1 known case in which a person has no record_id but is randomized. This has no impact on the DSMB outcome and can be ignored.
3. `test_adverse_event_if_not_date_end_outcome_must_be_ongoing`:
- **These are real errors that need addressing**, RAs should be contacted to amend the database
4. `test_adverse_event_tracking_is_entered_in_only_one_arm`:
- This is a warning that OPTIMUM's data entry guideline is being violated. The code will handle these events automatically but these should be resolved eventually by an RA. All results are from Toronto, Ksenya may need to be reminded to fix these.
5. `test_adverse_onset_is_valid_within_randomization`:
- These are **real errors that need addressing**. CU was on lockdown last DSMB, but you should be able to contact Patrick Brown to fix this issue
6. `test_arm_7_participants_only_randomized_in_step_2`:
- This is explained above and is similar to (4)
7. `test_if_arm_6_enrolled_must_be_consented`:
- This is a **real error**. Also a CU participant, should be resolvable now if you contact Patrick Brown
8. `test_if_randomized_then_treatment_group_available`:
- This is a **real error** and needs to be addressed. All participants randomized to step 1 or 2 need to have a treatment group. Contact the RA
9. `test_medications_is_entered_only_in_one_arm`:
- This is similar to `test_adverse_event_tracking_is_entered_in_only_one_arm`:
- The code will handle this case automatically, but this should be eventually resolved to ensure that database follows entry guidelines
10. `test_step_1_randomization_appears_atleast_in_arm_8`:
- This is explained in the above **DSMB Resolution Guide**. This does not affect the DSMB numbers
11. `test_adverse_event_actionj_is_other_but_missing_other_description`:
- This is not problematic for the DSMB report but it should brought up with an RA and data should be entered if possible.

## Documentation for DBT
- `dbt docs generate --profiles-dir .`: Generates Documentation
- `dbt docs serve --profiles-dir .` : Access Documentation, description of tables and columns
- Raw tables are direct from redcap
- These are redcap columns
- Models: Way we transform the data
- dsmb reports come from the projects tab
Look at the help for the python ingest script
- optimum-dsmb/ingest/ingest_redcap_exports.py
- ingest_redcap_exports.py --help
- OR nano ingest_redcap_exports.py
- This will generate the report ID in case you want to make changes to the table on REDCap