import logging formatter = logging.Formatter('%(asctime)s %(funcName)s:%(lineno)d: %(message)s') syslog = logging.StreamHandler() syslog.setFormatter(formatter) logging.getLogger().addHandler(syslog) logging.getLogger().setLevel(logging.INFO) logger = logging.getLogger(__name__) logger.info("foo") from unittest import mock

Server Runs on uvicorn + fastapi, and provides (in order of priority) information to the client In the first instance: microscope name, names of ongoing visits, visit duration Total amount of data acquired/transferred in visit Data processing information an interface to PyPI

Pinned notes https://confluence.diamond.ac.uk/display/SSCC/RabbitMQ Tasks BW: Looking into RabbitMQ server configuration, DH offers to assist JH: FITSM/ITSM RG: Added RMQ version of queue monitor (dlstbx/#58) SciComp: HTTPS to admin proxy (SC-2991) SciComp: Host aliases for RabbitMQ servers (SC-2969) SciComp: Set up systemd script for RabbitMQ (SC-2967)

2021-06-18 disussing the future Synchweb interface for single-particle data analysis results Anna's presentation on some ideas of how the Synchweb interface could look. (OneDrive link: https://dlsltd-my.sharepoint.com/:p:/g/personal/anna_horstmann_diamond_ac_uk/ESW0ssH-CK1Okmd2qU3fvJ4B5akSFvHpYRvxtulk_ZiGfA?e=gAQYXR) Google spreadsheet of ISPyB database discussion (dated 2020-06 so maybe don't rely on this too much) Example github for displaying 2D and 3D results from Relion

Last meeting minutes: https://confluence.diamond.ac.uk/display/SCI/EM+Meeting+2021-05-21 Agenda: Review action points from the last meeting: [ ] Anton Lawrence/Satheesh/Martin: compile a concise list of requirements/issues with database regarding write access via SQLAlchemy Neil suggests a review at the end of the run, opening up the EM tables for writing, then another review afterwards. [x] James Hall: add fields to allow multiple datacollections in one visit (raw2, ... folders)

Background Installation Cygwin The detector machine can not access the internet directly, so you have to jump through a few hoops to install cygwin. You first need to find a different Windows machine, on this one download cygwin from https://www.cygwin.com/ Run the installer, pick "Download Without Installing", any mirror, and add packages git rsync python3

Ticketed EM work: https://jira.diamond.ac.uk/browse/SCI-9795 (buffer table) Use the zocalo.ZocaloBuffer table. Karl to fix up references to the production ISPyB tables, and to add relevant permissions to the ispyb_ro, ispyb_scripts, and ispyb_api users (possibly others?) https://jira.diamond.ac.uk/browse/SCI-9803 (no results shown before completion) Sounds suspiciously simple to fix NS to add a comment to the ticket

Last meeting minutes: https://confluence.diamond.ac.uk/display/SCI/EM+Meeting+2021-04-09 Agenda: Review action points from the last meeting: get the pipeline running Pipeline is deployed and available. Being tested by beamline staff make it resilient

Status 2021-04-16: AH: Need to connect up ISPyB service via recipe MG: Add cryolo envionment module load EM/cryolo/1.7.6 to the recipe and indirector script. DH: looking at simulated data collections Status 2021-04-15: DH has been making releases AH: Need to connect up ISPyB service via recipe

TEMCenter computers controls lenses, can take TIFF images, but need to copy it to a different machine to then transfer to DLS We do not use this for acquisition Source of the metadata, knows experiment timings One other detector of the following: Gatan PC (Acquisition PC) 10.182.0.2 save files with bm3 or bm4 extension

Yuriy's email Hi Colin, Here is my AP follow up from our previous Relion meeting: I have tested idea of using FSC at 0.5 for selecting best model from InitialModel classes. It seem to work. Relion resolution criteria worked fine to exclude really bad model, which was initially selected (see below), but could not distinguish between two better models, classes 1 and 2 – it returned the same resolution. However, FSC at 0.5 criteria was better for model 2. The steps are below: Select every class into individual group. I had to do it from GUI for every class, since I could not find how to do this with a single step. Split every subset into 2 randomised subsets

This is a "live" document to keep track of current status and plans for launching the RELION processing pipeline from SynchWeb. 2021 Software commissioning visit Proposal cm28212 Visit start Visit end CM28212-1 09:00 Fri 1 Jan 2021

The new EM table schema has been deployed to production ISPyB. ISPyB SQLAlchemy Credentials Karl to set this up by Monday 8th of March. Will be a new credentials file in the usual place. File permissions: gda2.dls_dasc rw-rw---- That user will have write access via stored procedures and SELECT access for SQLAlchemy across the database. ISPyB API package release

present work done to date New role advertised for the Synchweb developer, 21st of February shortlisting. Likely decision by end of March. 12 applicants so far. Database schema - we have a pretty good idea what the version 1 of our changeset is. We plan to implement that instead of iterating the design first. We can then review in 4-5 months time. James so far was 100% busy on shipping, which will continue for a few weeks. He should be able to pick up Relion work in the February/March timeframe. Synchweb pages do not cover DB schema V1 features plan the next steps Version 1 of the schema should be deployed in the next week or two.

IT infrastructure 15 eBIC GPU nodes are being installed (possibly named gpu04-xx). There are some open networking configuration questions. SciComp aiming to have them available by the end of shutdown. If it slips beyond the end of shutdown then we can still make them available as soon as possible. A cluster usage policy will be drawn up by James Thorne and Luke. gpfs03 storage will be extended to a total of ~15 PB during the shutdown. This should have no effect on eBIC operations. Full details of the Scientific Computing plans for Shutdown 4 2020 are in the shutdown calendar. New links

Attended by Markus Gerstel, Anna Horstmann, James Hall and Colin Palmer. Most of the discussion was around what we think the EM data acquisition and processing software architecture should look like in the perfect (ha!) long term future. Markus explained how things are organised in MX. A visit can have multiple data collections. Related data collections can be linked together in data collection groups. A data collection can contain multiple processing jobs. A data collection must have a unique integer ID. In other parts of Diamond, the IDs are created by the ISPyB database servers. Because synchronising multiple servers is a Hard Problem, the IDs for successive data collections are typically not sequential but are guaranteed to be monotonically increasing. We're not sure where the best place is to put the creation of data collection IDs in the EM context. One option is to use some kind of web service: the rsync data transfer script could send a request (using curl, wget or powershell Invoke-WebRequest) to ask for an ID, then put this into the path used for the file transfer. It could also send information to the web service about the files that have been transferred. We would need to work out how to fall back to allow data transfer to continue if the web server were unavailable. The alternative is to trigger events on the receiving end of the rsync connection. The rsync script itself remains mysterious. Markus has not yet seen a copy! It is installed locally on the microscope computers, but we're not sure if there is any kind of master copy or version control. From what we do know, the eBIC local contacts typically run the standard script to copy files from the microscope to a raw/ subdirectory in the visit directory. If the acquisition changes significantly (perhaps for a new sample? This would probably correspond to a new data collection, in ISPyB terminology), they stop the original transfer script and run a different version that copies files to a raw2/ subdirectory.