---
name: DIALS core meeting 2021-05-20
tags: core meeting
---

# DIALS core meeting 2021-05-20

[](https://hackmd.io/BGwjOs12RJ6yTlskFXof8Q)

## Previous Actions

(Note from ND: Marking items with further discussion on the agenda below with `*`)
* [ ] MG: organise a [typing intro lecture](https://dials.github.io/kb/core/2020917#typing-mypy), likely to happen in June - _in progress of having an allocated speaking slot_
* [ ] [dx2 proposal](https://dials.github.io/kb/core/20210128#dx2-proposal)
* [ ] MG to put in a PR to create a proposal in the proposal-space
* [ ] MG+ND: Come up with a proposal to [move away from all code being in header files](https://dials.github.io/kb/core/20201001#overall-architecture-discussion) and consolidate into a single library
* [ ] [Move to `hdf5plugin`](https://dials.github.io/kb/core/20210225#change-hdf5-plugin-library).
- [ ] GW/MG: Make pull request for [moving](https://dials.github.io/kb/core/20210506#hdf5plugin-performance-evaluation) to `hdf5plugin` package
- [ ] **Conclusion**: Wait for the 3.5 release before moving over
- [ ] \*[cbflib conda-forge package](https://dials.github.io/kb/core/20210422#cbflib-conda-forge-package)
- [x] ND: Try using old cbflib 0.9.6 (allowing regenerated pycbf) to see if it would be suitable for DIALS/dxtbx to release just the old version
* [x] ND to set up a [`cbflib_adaptbx` pull request](https://dials.github.io/kb/core/20201125#cbflib_adaptbx-dependency) for MAR detector fixes. Verify with `labelit_regression`
- [x] ND: [Remove](https://dials.github.io/kb/core/20210506#updating-pre-commits) `black` autoupdate from DLS "Clutter" job
- [x] \*XFEL-regression tests running on DIALS azure ([prev](https://dials.github.io/kb/core/20210506#adding-xfel-regression-to-dials-azure-tests-dials1671))
- [x] MG: Merge [dxtbx#351](https://github.com/cctbx/dxtbx/pull/351) to use prebuilt cctbx packages
- [x] MG: Evaluate XFEL-regression pull request [dials/dials#1671](https://github.com/dials/dials/pull/1671)
- [x] \*MG: Put in [proposal ("DC3")](https://github.com/dials/kb/pull/5) to follow [NEP-29](https://numpy.org/neps/nep-0029-deprecation_policy.html) directly for Python deprecations

## Agenda

### Image viewer with two-theta offsets

- [`dials/dials#1715`](https://github.com/dials/dials/issues/1715) "dials.image_viewer fundamentally broken if 2θ != 0"
- Means image viewer broken for I19-1 data
- Fallout from [`dials/dials#1182`](https://github.com/dials/dials/pull/1182) ("Single panel special cases") is somewhat larger than we had understood ([saving](https://github.com/dials/dials/issues/1385), [angle bikeshedding](https://github.com/dials/dials/discussions/1358))
- Rather urgent to fix for next run (starts June 15th)
- Proposed fix: [`dials/dials#1716`](https://github.com/dials/dials/pull/1716) "add option to display in the best fit frame"
- Discuss
- Problems working out how to make this default/change defaults
- Consensus seems to be to merge
- Problems all gone away.
- Outcome: **Merge for 3.5**

### [cbflib conda-forge package](https://dials.github.io/kb/core/20201125#cbflib-conda-forge-package)
* From [past discussions](https://dials.github.io/kb/core/20210422#cbflib-conda-forge-package) it is clear that getting a newer release of cbflib is currently an insurmountable problem
* Making changes somewhat problematic as test suite failed for a long time
* Debian support for raw package very important. Must not break anything underneath their [18 patches](https://sources.debian.org/patches/cbflib/0.9.6+dfsg1-2/), even for new versions
* `pycbf` tests never been run on python 3, longstanding problems and pain-points e.g. `SWIG_PYTHON_STRICT_BYTE_CHAR`
* HJB estimates 3 months of graduate student effort to fix
* (In ND personal view) doesn't seem keen to have us take over maintenance burden
* Need to investigate alternative methods
* Have split off just the pycbf parts of CBFlib into [`dials/pycbf`](https://github.com/dials/pycbf), based on the 0.9.6 sources
* Currently doesn't touch CBFlib sources, generates off of 0.9.6 tag (windows compatibility will need patches)
* `pip install pycbf` works today - gets `0.9.6.2`
* 0.9.6.0 gets plain `pycbf`, regenerated from literate sources via nuweb, `CBFlib.html`, SWIG 4.0, with `SWIG_PYTHON_STRICT_BYTE_CHAR`
* conda-forge staged recipe [`ndevenish/stages-recipe`](https://github.com/ndevenish/staged-recipes/tree/pycbf) working but unsubmitted
* 0.9.6.2 has:
* Uses `dials-data` ([branch](https://github.com/dials/data/tree/pycbf) right now) for regression data
* Expose `HAS_SWIG_PYTHON_STRICT_BYTE_CHAR` as a python variable to help aid transition away
* Add missing binding to `cbf_read_buffered_file` - which we [manually bind](https://github.com/cctbx/dxtbx/blob/a10281ac70e7be2dc76fb6ee335f9b890b904f22/format/boost_python/cbf_read_buffer.cpp) in dxtbx
* CBFlib functions that accept basic `char *`strings will accept _both_ `bytes` and `str` objects
* Basic `libimg` [bindings](https://github.com/ndevenish/pycbf/pull/1) to allow python-only `FormatMarIP` support
* **Doesn't currently support**: Windows, HDF5, custom libtiff fork, cbf-REGEX, use as linking/`#include` target. Extra dependencies are hard, building manylinux wheels.
* Usage in dxtbx depends on removing binary CBFlib dependencies:
* Binary dependency on cbflib in `FormatMarIP` via `iotbx.detectors.marIP` (pointed out [here](https://github.com/cctbx/dxtbx/issues/264#issuecomment-732379705))
* Somewhat simple to resolve - new `libimg` bindings built for this, have WIP branch [`ndevenish/dxtbx@marrpy`](https://github.com/ndevenish/dxtbx/tree/marrpy) to make pure python
* Binary dependency on cbflib for `FormatCBFFull` DetectorBase interfaces
* Less simple to resolve without probing and rewriting large parts of CBFAdaptor
* This is initialised by everything that is based on `FormatFullCBF` - but only used by one file that only matches `FormatFullCBF` in the regression tests - `SPring8_ADSC_SN916/Xtal17-2phi_3_015.cbf`
* This image is root cause of only two tests that fail (when detectorbase regression tests are skipped)
* WIP branch at [`cctbx/dxtbx#368`](https://github.com/cctbx/dxtbx/pull/368) to do most of this
* What is Detectorbase otherwise supporting still? Could we only initialise it if requested directly?
* What is left to consider?
* What's the usage status of `DESY_BW7B/mar345_01_001.mar2300` - is this okay to use in regression tests?
* need an 'OK' to publish the file, should probably be added to dials-data/image-examples. (The file is also used in dxtbx regression tests)
* Have attempted to run `labelit` and `labelit_regression` as requested
* **labelit** tests fail on 3.9 - seemingly because they have hardcoded pickle sizes. But passes on 3.6.
* **labelit_regression** test suite fails to run with `AssertionError: Duplicate tests found`
* Also, `$ grep -r dxtbx labelit` suggests that it does not use dxtbx?
* In any case, since tests don't seem to have worked for a while cannot verify that they are still working
- We spoke about differences between sdist/bdist, explained approach taken
- **Action**: Aaron to look into labelit-regression tests
- Going to think about, speak to HJB

### [dxtbx#288](https://github.com/cctbx/dxtbx/pull/288) "Construct Experiments directly" post-merge outstanding issues
- [dxtbx#336](https://github.com/cctbx/dxtbx/issues/336) - DataBlockFactory broken for single images
- XFAIL test for this now put in: [dxtbx#337](https://github.com/cctbx/dxtbx/pull/337)
- Discussion: DataBlock has been deprecated for a long time in DIALS, if someone is using DataBlock they probably don't want to/can't use latest DIALS Anyway
- Outcome: No further action
- Leave ticket for a while and see if any problems caused, revisit later in the year

### Python 3.6 Support
- Releases have been built with 3.8 for several versions
- We had a previous agreement in place in [dials#1327](https://github.com/dials/dials/issues/1327) to follow conda-forge. At the time, we understood that conda-forge was following [NEP-29](https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table), which they have now diverged from.
- "DC3" now proposed in [dials/kb#5](https://github.com/dials/kb/pull/5)
- Do we want to consider dropping support for 3.6
- **Merge DC3 draft** -- done: https://dials.github.io/kb/proposals/dc3
- **Aaron: To ask Billy what the phenix plans are for 3.series**

### Stop cctbx testing at Diamond?
We currently run Jenkins builds for cctbx on Linux and MacOS. At some point we started seeing transient network issues, a fix for this has now been merged ([cctbx#614](https://github.com/cctbx/cctbx_project/pull/614)). Build times are between 5 and 35 minutes, and recently we ran on average 3 times a day on Linux and once a day on MacOS.
Does running these tests ourselves still add anything? cctbx now has good coverage on Azure, and we rarely (if ever?) commit directly to `master`, and I suspect most of us only pull from the `stable` branch.

Outcome: **Remove**

### bootstrap: make `--mamba` the default
So far we haven't seen any issues using micromamba on Azure. Should we make this the default?

Outcome: **Make it so.**

### XFEL regression tests on Azure
This has been merged, [`dials/dials#1707`](https://github.com/dials/dials/pull/1707).

- What's required now?

Only superficial changes remain: currently when a pull request is built, you get the green ticks from the commit build. After this, you do not have any indication that the pull request build is still running until, about 10 minutes later, those results come in. MG will pick this up at some point.
* fix incoming in [dials#1718](https://github.com/dials/dials/pull/1718)

### AOB: Dials regression
- Aaron worked out how to split dials_regression
- Three folders for non-regression-data stuff
- dials2-scaling
- doc
- duelling_profiles
- Everything else data and python scripts
- Nick: Thought main problem was provenance e.g. are we allowed to publish all the data sources
- If this was solvable we could add them to [dials-data](https://pypi.org/project/dials-data/) - even add the ability to pull from LFS
- Aaron: Intends to go through and work this out
- Markus: There is already a dials-data [`image_examples`](https://github.com/dials/data/blob/master/dials_data/definitions/image_examples.yml) definition, which includes those files from dials-regression where we know the provenance and have permission to publish them. Would rather we add to that than go back to the model where one needs to check out test dependencies as a separate step. The exact list of files in dials-regression that we still use is also known and was made explicit in [dxtbx#346](https://github.com/cctbx/dxtbx/pull/346).

## Deferred to next meeting

### Next meeting
Thursday, June 3rd, 4pm UK (BST), 8am PDT.