DIALS core meeting 2022-12-08

Previous Actions

• std::string support for messagepack. This is used in XFEL module, should try to fix. dials/dials#1858.
  • Is using pickle until this is fixed, so probably good to do
  • AB Added to codecamp list. Might get to this!
  • Looking at this week. Removing seems acceptable and probably the easiest solution
• AB: Complete coordinate updates (Fix beam center and intensity readout of pixel coordinates)
• https://github.com/cctbx/dxtbx/issues/559
  • GW: Checks to see what affects.
  • AB: Make PR for above
• GW: https://github.com/cctbx/dxtbx/issues/562 - create a one-page summary on projecting resolution rings onto a matplotlib image
• ND Investigate getting psana tests running on the DIALS xfel-regression testing
• ND: Unpin setuptools in DIALS builds
• ND: Look at https://github.com/dials/dials/pull/2263

Agenda

C++14 & libtbx

• stop blocking c++14 in libtbx? work in progress -> win

Revert Walrus

• did not solve problem -> dependency car crash (lots of; starts with y2k :headdesk:)

So many things don't work with Python 3.7. Ran out of numbers. Made more numbers -> new psana. 4.0.48. Minor point release critical to work. Cannot read xtc without. Data on Tuesday. Build was ready to go. Critical but needs boost 1.78. Problem WAS NOT DXTBX. ..47 uses boost 1.74. Did not release Python 3.9 version, only 3.7. Removed Walrus, f-strings -> win. But way worse. So reverting Walrus. Using branch. Have ability to set the c++ version in libtbx builds with C14. cctbx-base builds on C14. Not yet on main but coming real soon.

Still have a dependency snarl to unpick. Dependency block between psana / boost / … means you cannot install other cctbx dependencies. Cannot find a matching glibc.

Postscript: exafel people want c++17. auto_ptr stuff now problem -> will be looked at in January. SCons vs. cmake. "too hard" Nick D has a mechanism to transcribe to CMakefiles.

CMake would be better if we were there.

Dropping Data Blocks

https://github.com/cctbx/dxtbx/pull/504

• GW PR
• After PR/PRs, ASB to contact e.g. Mike Wall about this, too.
• GW thinks reasonable to merge when these concerns addressed
• Waiting on AB now, will get sorted as soon as can
• Progress!
  • DataBlock in Lunus is one of the outstanding issues
    • AB Identified scripts, Mike Wall agreed that they should be deprecated
    • This is Done! Was entirely in deprecated scripts that weren't being used any more
  • XFEL repos are still WIP
    • ls49
    • cxid9114 (this should be fixed)
• Planned time to work on this stuff in October
• Just after branch for 3.12 release of DIALS merge this
• Did this! However… this broke a lot. We reverted and made again, but a bit more complete.
• Currently actively being worked on -> https://github.com/cctbx/dxtbx/pull/570
• Moving test data to dials-data from dials-regression but ultimately should be done

Stills to sequences

https://github.com/dials/dials/pull/2273

• dials.import convert_stills_to_sequences=True broken for h5 data - several github issues relating to this (see PR above for links)
• With current dxtbx code, PR does not work for formats that include lazy formats (e.g. SACLA test data), due to an assertion in FormatMultiImage.py: https://github.com/dials/dxtbx/blob/7494dc0332f94c0d4593865f82c7c08a88ae65ea/src/dxtbx/format/FormatMultiImage.py#L215 What is the rationale behind this assertion (developer warning/sanity check?), can it be removed (seems to work fine if removed)?
• Everyone go and reaffirm their understandings and find out what the real reasons/solutions could be

JBE: got impatient and merged it in. Added change to switch "lazy" on or off. Does not affect any ongoing workflows, no issues, no fallout -> all good. Contemplating how to better structure and construct image sets to avoid heat death. Aaron may, or may not, work on this in 2023. Image sets -> sequences. "Time for image sets to go away everything should be a sequence - ASB 2022-12-08@1624UTC" - however we may not be able to fully populate the sequence interface e.g. len() or dxtbx.load(). Some element of future laziness may be needed e.g. not reading whole XTC stream to get wavelengths for every frame. Also treating serial data as a scan useful for mmCIF reasons.

NXMX Python

https://github.com/dials/nxmx

• Important to be able to work on/modify directly in a bootstrap environment, make sure that this can be added as a likewise dependency
• Nexpy?

Ben started this then vanished for 9 months: do we want to do anthing with it? Action RJG.

[ ] - Richard take forward nxmx python module.

AOB?

Richard - image view fast-api REST based tool to make bitmaps. Can also do spot finding so could retire spot finder server-client app and replace with RESTFUL thing. Server client still used but probably by people who would be happy to use a new thing do to the same job. Add as another package. Add deprecation flags to server client once flask version available?

Add to dials namespace. More eyes would be useful. What else would people want?

Next meeting

Thursday, December 22nd, 4pm (GMT), 8am (PST) TBD

Thursday, January 12th, 4pm (GMT), 8am (PST)