DIALS core meeting 2022-04-14

--- name: DIALS core meeting 2022-04-14 tags: core meeting --- # DIALS core meeting 2022-04-14 ## Previous Actions - [x] (*) DXTBX/pycbf [[prev](https://dials.github.io/kb/core/20210826#cbflib-conda-forge-packagepycbf)] * [x] ND: conda-forge pycbf: Make new release to use dials-data directly for tests * [x] Nick: Add lots more attribution - [ ] (*) Nick: dxtbx and read-only base - [ ] Nick: Make sure that HKLviewer ([prev](https://dials.github.io/kb/core/20211209#new-item-hklviewer)) works at Diamond on `module load` - [x] Attempted, installing pyside2 is easy, but fails after that ([log](https://gist.github.com/ndevenish/fce20861a335385a5a65ed27c982e141)) - [x] **Action** ND: ask RO about this - Sent enquiries, under discussion - [ ] Aaron: Reduce zenodo dataset for contiguous nexus - [ ] Put on Dan Paley backburner also - time is short - [x] On the Todo list now! - [ ] ND to review: TOFBeam - A time-of-flight beam in dxtbx [`cctbx/dxtbx#492`](https://github.com/cctbx/dxtbx/pull/492) ([prev](https://dials.github.io/kb/core/20220303#proposal-for-a-time-of-flight-beam-in-dxtbx)) - [ ] Continue to discuss mac dispatcher issues (e.g. no GUI launch for image_viewer etc) in issues ([`cctbx/cctbx_project#739`](https://github.com/cctbx/cctbx_project/issues/739)) - [ ] `std::string` support for messagepack. This is used in XFEL module, should try to fix. [`dials/dials#1858`](https://github.com/dials/dials/issues/1858). - Actually try to fix this - [x] This is not going to happen before April at the earliest - [x] GW: Notice and merge pathlib changes [`dials/dials#2038`](https://github.com/dials/dials/pull/2038) ## Agenda ### PyCBF - Version 0.9.6.5 now out - Added more attribution in [`dials/pycbf@b8d4f6`](https://github.com/dials/pycbf/commit/b8d4f6ed47588a1145430f6fb53dbea9aec51e5d) and [`conda-forge/pycbf-feedstock#10`](https://github.com/conda-forge/pycbf-feedstock/pull/10) - Now uses dials-data for testing data - Added windows build: [`dials/pycbf#21`](https://github.com/dials/pycbf/pull/21) - somewhat scattered list of changes applied to CBFlib, at pycbf-build-time in [`002_windows.patch`](https://github.com/dials/pycbf/blob/56e99bafe9fded6d4a738febb943bf102d58b414/patches/002_windows.patch) - Wheels live for multiple manylinux flavors, MacOS x86/arm, windows 64bit/32bit, for Pythons 3.6/3.7/3.8/3.9 live on pypi and, conda-forge mac/mac-arm/linux/windows 3.6/7/8/9/10. - Can we close this item out for now? - Yes ### Contiguous Nexus - [ ] "Contiguous Nexus" [`cctbx/dxtbx#356`](https://github.com/cctbx/dxtbx/pull/356) - [ ] Non-draft state is pending checks against issues @dwpaley raised, and checking for non-contiguous cases [[prev](https://dials.github.io/kb/core/20210715#contiguous-nexus-pr)] - AB wants this to be applied against new nexus writing - RG: Probably useful to mock up an example in the nxmxtests - **Action: Aaron** Reduce down zenodo dataset used in the issue ([`cctbx/dxtbx#93`](https://github.com/cctbx/dxtbx/issues/93)) so that we can include it in dials-data. - Afterward: Coordinate with RG to make sure the ideas in this are integrated into new nexus reading - WIP - Dan has reduced to Six images - 300mb, wondering if this is enough. Discussion about what is required for the files and how small to absorb into the dials-data repository. ### DXTBX read-only base - Issue [`cctbx/dxtbx#444`](https://github.com/cctbx/dxtbx/issues/444) - Read only base by default would be the ideal - Previously discussed downstream use cases and if DIALS needs to be included - PHENIX - depending on the readonly dxtbx stuff - until that is fixed DIALS will be pinned at 3.2 or similar. They also have a hard stop at Python 3.7, due to a slowly changing dependency. - Defer for two meetings - pinning on DIALS 3.2/virtualenv script working around for now - Originally merged in, but turned out to be more involved - **Top Priority** - Reopened under [`cctbx/dxtbx#511`](https://github.com/cctbx/dxtbx/pull/511) - main change is regenerating dispatcher immediately during configuration, this mostly seems to work - AB: going to run in next few days ### DIALS `src/` layout - https://github.com/dials/dials/pull/2077 - Moves to the same `src/` based layout as dxtbx, xia2. This enables us to write a standard `setup.py` to actually install DIALS and it's entry points, like a normal python package. It also provides isolation against accidental namespace imports, namespace pollution. - Once merged, will go through other open PRs (including draft) to update them the the new layout, though this isn't expected to be much of a problem. - Uses the same mechanism as above dxtbx PR, e.g. regenerates the DIALS dispatchers once the environment is updated to properly recognise the package layout - _Relies_ on [`dials/dials@cafaf4`](https://github.com/dials/dials/commit/cafaf4df9454b1a63cac2940322784d0ca020fc7) to force dials to configure before xfel, so that xfel picks up the correct locations - Planning to merge this in over the next couple of weeks ### Circular XFEL dependency - [`dials/dials#2076`](https://github.com/dials/dials/issues/2076) - xfel has crept up from optional to a hard dependency - `combine_experiments`, `cosym` and a few others use as unqualified imports - Not sure how difficult it would be to resolve this and keep all functionality - `stills_process`/`/indexing/nave_parameters.py` look reasonably dependent - might have to accept that these won't work outside a libtbx bootstrap. ### Dropping Data Blocks https://github.com/cctbx/dxtbx/pull/504 - [x] GW PR - [ ] After PR/PRs, ASB to contact e.g. Mike Wall about this, too. - GW thinks reasonable to merge when these concerns addressed - Waiting on AB now, will get sorted as soon as can ### Still process - Going to be a PR for "Diffbragg stage 1 refinement" - Uses pixel models of image to refine crystal using polychromotic spectra as per Mendez 2020 paper. - Implements an override in stills_process ASB wants to put diffbrag stage 1 into dials.stills_process as a refinement engine. Depends on simtbx. This is very much dependent on GPU. Can use OMP etc. - Derek Mendez going to send changes to Aaron - more of a long term thing, drop until PR comes up ### Changes from XFEL sessions - Dan: Image viewer: now support ellipses instead of just rings - And can write out detector hierarchy file - Mentioned an issue with detector Z, DGW remembered a related-sounding issue https://github.com/cctbx/dxtbx/issues/472 - PRs incoming. - Known orientations parts of stills_process needed to be rewritten to support this ### Ransac indexing - Processing data using stills_process and crystfel, were getting more lattices than DIALS. - Crystfel uses alternative indexing programs (e.g. ND remembers compiling them specifically for users at DLS) - "RANdom Subsampling And Consensus" - Index image 50 times, randomly subsetting the data - Cluster solutions - In mid-feb implemented similar approach - Reduce data in small portions until failing/getting same answer - Still unresolved how to choose threshold for ending this - Much higher indexing rates! 25% more lattices, _and_ of higher quality ### Next meeting Thursday, April 28th, 4pm UK (BST), 8am PDT.