# EM software meeting 2021-05-21

Last meeting minutes: https://confluence.diamond.ac.uk/display/SCI/EM+Meeting+2021-04-09

## Agenda:

### Review action points from the last meeting:
1) get the pipeline running
* Pipeline is deployed and available. Being tested by beamline staff
2) make it resilient
* Working on this. DC - Could do with support for additional raw folders.
3) Synchweb updates to be deployed in 2 weeks
* This took longer than 2 weeks but is now deployed. Work on 'raw' folders is ongoing
4) Update database changes to our own fork
* Database updates have been deployed.
5) Recruitment outcome
* SynchWeb developer starting Monday 24th May

### SQLAlchemy discussion
* streamlines the development process - changes are easier
* enables much faster deployment of changes
* downside = less constraints on writing - easier to create more mess
* NS - leave SQLAlchemy as a read option until people build up experience with using it and look at writing in the future
* MW - possibility of EM specific DB?
* AL - KL is _the_ person who knows enough about MariaDB and the DLS setup. Not keen to increase the dependence on him. This isn't ideal and we should be looking to reduce this bottleneck. Requested support for Karl.
* MG - it would be helpful to map out how we are going to work towards using SQLAlchemy as a main interface, with dates.

### Review Agreed Time-table from initial meeting on 2021-04-09:
#### Main Relion stuff:
- Data collection registration and control processing via SW from next run
- Zocalo - WIP, data in ISPYB in a few weeks

Future:
#### Q2
- daily DC simulations
- bare minimum SynchWeb info to user
- limited SW dev - SW dev capacity is low. Neil concurred

#### Q3
- logic to tie data together
- new SW developer start
- fully automate analysis
- all relevant data in ISPyB

#### Q4
- continual improvements
- support different user needs as they arise

### Updates on current progress:
1) SynchWeb - Neil
2) ISPyB - Neil
3) Relion python/Zocalo/ - Markus
4) EM - Dan

### AOB:
* Write permission to the Database, can we streamline/change this process?

## Action points
* Anton Lawrence/Satheesh/Martin: compile a concise list of requirements/issues with database regarding write access via SQLAlchemy
* James Hall: add fields to allow multiple datacollections in one visit (raw2,... folders)
* EM data analysis team: improve pipeline resilience
* EM data analysis team: utilise all database fields that we can
* EM data analysis team/Dan Clare/Yuriy: consider which features are desired in SynchWeb, in collaboration with the new starter and eBIC staff. One output of this meeting will be a 6 month plan for the new starter.
* ? Dan Clare: show Martin Walsh the Relion pipeline and SW interface.

In future invite Anna Rackley
Anna Horstmann contract being pushed by Martin