# EM software meeting 2021-07-02

Last meeting minutes: https://confluence.diamond.ac.uk/display/SCI/EM+Meeting+2021-05-21

## Agenda:

### Review action points from the last meeting:
* [ ] Anton Lawrence/Satheesh/Martin: compile a concise list of requirements/issues with database regarding write access via SQLAlchemy
* Neil suggests a review at the end of the run, opening up the EM tables for writing, then another review afterwards.
* [x] James Hall: add fields to allow multiple datacollections in one visit (raw2, ... folders)
* [x] EM data analysis team: improve pipeline resilience
* In contact with Relion developers to see if we can address this at both ends, with more resilience on our side, and atomic writing on the Relion side.
* [ ] EM data analysis team: utilise all database fields that we can
* This is blocked by us not yet making use of the Zocalo Buffer table, which we now have. Should have this sorted by next meeting
* [x] EM data analysis team/Dan Clare/Yuriy: consider which features are desired in SynchWeb, in collaboration with the new starter and eBIC staff. One output of this meeting will be a 6 month plan for the new starter.
* We have a proper meeting series for this now. Positive development, things are moving.
* [ ] Dan Clare: show Martin Walsh the Relion pipeline and SW interface.
* Showed James and Andy the pipeline, but not yet to MW

### Review Agreed Time-table from initial meeting on 2021-04-09:
#### Main Relion stuff:

Future:
#### Q3
- logic to tie data together
- [x] new SW developer start
- fully automate analysis
- Data collection registration and control processing via SW from next run
- https://jira.diamond.ac.uk/browse/SC-3114
- No answer for 2+ weeks, escalate via Nigel next week, if still no traction escalate via SM
- DC will get into contact with Martin
- Analysis is triggered automatically once the data collection is in ISPyB (via user input)
- Keep the options on Synchweb (eg. do 2D/3D classification) unless they can be automatically determined (eg. file extension)
- We need to know when to stop
- current logic: stop when no new files in 2 days and all files have processed
- after 7 days the process will die due to cluster policy
- sometimes we iterate on the last batch rather than ending
- we could use number of holes as given by EPU, may have to consider users adding more holes during data collection
- all relevant data in ISPyB
- driven by visualisation needs, ie. once we want something shown in Synchweb then ensure it goes into ISPyB
- Synchweb features:
- Particle picking view
- 2D classes
- Icebreaker summary report
- Icebreaker running update

#### Q4
- Discuss Q4 targets in September meeting
- continual improvements
- support different user needs as they arise

### Updates on current progress:
1) SynchWeb
* new starter has started and is making progress
2) ISPyB
* new particle picking column added, buffer table set up
3) Relion python/Zocalo
* it is running and so far there haven't been too many support requests
* there is still Josh's classic pipeline around, which MG believes is no longer used. This will be removed in the next shutdown
4) EM
* Want to process EER files, need an option in Synchweb to add .eer extension.
* Setting this extension will use the relion-internal motion correction
* Also needs to add a new integer field to control the EER fractionating
* "How many frames do you want in your movie?"
* then do maths for users
* This brings in a rounding complication, if the number of frames is not an integer multiple of this field then frames will be thrown away, which introduces an error to the ~~dose~~ electron fluence calculation.

### AOB:
* none

## Action points
* See unresolved items at the top of the page
* Next meeting in 6 weeks