Brainglobe dev meetings

# Brainglobe dev meetings [EOSS5 project board](https://github.com/orgs/brainglobe/projects/2) [Open PRs](https://github.com/search?q=org%3Abrainglobe+++&type=pullrequests&state=open) [Issues (excluding new atlases, by newest opened)](https://github.com/search?q=org%3Abrainglobe+++-label%3Anew-atlas&type=issues&state=open&s=created&o=desc) [Issues (excluding new atlases, by recently updated)](https://github.com/search?q=org%3Abrainglobe+++-label%3Anew-atlas&type=issues&state=open&s=updated&o=desc) ## 5 Feb - AF - Grant writing, tibia atlas - Reviewing PRs from SP - SP - Finishing off the grant money (40 hrs left) - Improvements to template-builder - Fixed a few bugs - Added user feedback - SH - Working with Harry on the BrainGlobe side of her thesis - Transforming segmentations from histology space to atlas space - HC - Updating PyNUtil to work with `brainglobe-registartion` output - Add an abstraction layer that allows easier integration with other tools - VP - SWC rat atlas released - Updating `brainglobe-atlasapi` documentation - Annotations between the Waxholm and SWC rat template aren't perfectly aligned - May wait until 25um template is ready to optimise - 25 um template is still running (week+ at this point) - Estimate - 18 days remaining - For 10um template, we could use the deformation fields from 25um to transform and average - IT - Converting atlases to new version - `brainglobe-registration` touchups - Converging with the Allen on the spec ### SSI BrainGlobe application - Small (£150k) grant application is open - 1 year of developer time - OSSS - BrainGlobe docs - More data types (2D registration) and more atlases to open the ecosystem to more users - Integrating ccf-translator into the rest of the ecosystem ### GSoC Review - Link to NIU contributing guide on BrainGlobe - This can be linked to remind contributors of etiquette - Prioritise people not PRs ### PR and Issue review - AF follow up on https://github.com/brainglobe/brainrender-napari/issues/218 - IT follow follow up on https://github.com/brainglobe/brainglobe.github.io/issues/434 - IT follow up on https://github.com/neuroinformatics-unit/fancylog/pull/70 ### Present - Alessandro - Viktor - Saarah - Adam - Harry - Stella - Igor # 22 Jan - GSoC projects - has project: - brainrender-napari: maintenance of QT GUI side, add small features - brainglobe-heatmap: go from mesh to annotation - brainglobe-meta: simplified high-level BrainGlobe API, standardise terminology/arg+kwargs/..., reference API Docs, conda forge, tox -> uv - cellfinder: (No ML) several channels and 2.5 D - can have projects: - other packages - out of GSOC - morphapi - brainrender - brainreg - SSI BrainGlobe EoI - registration and API: expands to 2d data and more species - Student work with irv+helga money - adding atlases - data api catalogue ### Present Alessandro Adam Igor Niko Saarah ### Notes - VP - Working on the SWC Rat template paper - Need to be careful with up/downsampling mapping from SWC to Waxholm - 25 um template is still running - IT # 8 Jan ### Present Igor Adam Niko Saarah ### Notes - Stella has shared [summary of ARC's work](https://brainglobe.zulipchat.com/#narrow/channel/552728-atlas-creation/topic/ARC's.20work.20on.20template-builder/with/566946116) - Igor on BG Atlas API v2: - there is a `legacy` flag to get the old version. It's useful for people who already have old atlases downloaded AND have no internet connection. If they have a connection, the new v2 atlases will be fetched. - Where to store hemispheres? Probably as a compressed 3D volume, to live with annotations - Rest of week Igor to focus on writing a BBSRC grant - Adam has finalised the application for harmonising IBL and BrainGlobe tools ### Issues - https://github.com/brainglobe/brainglobe-atlasapi/issues/722 AT will create the tag and tag a few high priority atlases - https://github.com/brainglobe/brainglobe-atlasapi/issues/720 will wait until v2 `atlasapi` ### PR - https://github.com/brainglobe/brainreg/pull/266 IT follow up - https://github.com/brainglobe/cellfinder/pull/572 AT follow up # 11 Dec ### Present Alessandro Niko Kimberly Stella Saarah Igor Viktor - Atlas creation scripts that live in `brainglobe-template-builder` will be marked as legacy and moved to `atlas-forge` - Need to double check documentation for `brainglobe-template-builder` to make sure it's up to date with the most recent changes - ARC is a bit under-budget: - Stella and Kimberly can use some time to write methods for the zebra finch paper - SP - https://github.com/brainglobe/brainglobe-template-builder/pull/120 - KM - Working on zebra finch template, curating images, preprocessing - Updated naming schemes (x, y, z to 0, 1, 2), (source, raw to stanardise, preprocess) - https://github.com/brainglobe/brainglobe-template-builder/pull/127 - https://github.com/brainglobe/brainglobe-template-builder/pull/131 - Last day on BG is tomorrow (Dec 12) - SH - Mapped Dorr atlas to Perens atlas - That's now available via `brainglobe-ccf-translator` - AF - Reviewing KM PRs - Making figures for Molerat paper - NS - Mostly discussion in person and on GitHub - IT - `brainglobe-atlasapi` V2 - Some atlases uploaded to `s3://brainglobe/atlas/` bucket - Current V2 atlas class is in https://github.com/brainglobe/brainglobe-atlasapi/tree/spec-wrapup - VP - 25um SWC rat template - 50um SWC atlas will be republished in Waxholm space Issue/PR review: - https://github.com/brainglobe/brainglobe-heatmap/issues/103 AF will follow up - Transition from using meshes to annotations image for 2D - Could be a GSoC hopeful PR - https://github.com/brainglobe/brainglobe-template-builder/issues/116 - Either combine standardise into preprocessing - Elevate standardise to it's own module - Have standardise.py preprocessing.py and postprocessing.py as files in the root package directory # 27 Nov - What should a BrainGlobe output look like - Coordinate with Locari for atlas aligned data storage - Deprecate `xml` in favour of `yaml` for cell storage - Throughput in writing is the main concern - AF - One merged PR for Matt - Stella Prins (SP) - Docs PR for axes ordering in `napari` and BrainGlobe https://github.com/brainglobe/brainglobe.github.io/pull/403 - Pair programming with Kimberly (testing infrastructure) - Kimberly Meechan (KM) - Generic downsampling PR https://github.com/brainglobe/brainglobe-template-builder/pull/100 - Generalise `read_with_dask` (generalise for 3D tiff) https://github.com/brainglobe/brainglobe-utils/pull/139 - Tests for source to raw once unblocked - AT - Packaged molerat atlas https://github.com/brainglobe/brainglobe-atlasapi/pull/679 - Octopus atlas (no midline and squishy) - STPT rat template in Waxholm and SWC space collaboration with EBRAINS. Gubra is out (restrictive license). - Collaborate with Oslo on BrainGlobe output formats - Harry Carey (HC) - `brainglobe-ccf-translator` - Update orientation for transforms to align with standardised BrainGlobe releases - Increasing testing coverage - Update the atlas names and tidy file storage locations - Will meet with a collaborator to discuss integrating pan-primate deformation fields - PR and Issue review: - https://github.com/brainglobe/brainglobe-registration/pull/129 IT follow up - https://github.com/brainglobe/brainglobe.github.io/pull/405, one person without a web presence, will merge without a link, using the atlas image for the card for contributors without a GitHub account - https://github.com/brainglobe/brainglobe-template-builder/pull/46 VP to follow up on - https://github.com/brainglobe/brainglobe-workflows/issues/160 wait for follow up - Further discussions: - Planning of NIU OSSS - Kimberly will help with Large Array Data - Harry will help with BrainGlobe etc... # 30 Oct - AF - Application to make a bone atlas - Crab template is moving forward - `brainglobe-template-builder` reviewing PRs for Kimberly and Stella - Small PRs for docs - IT - Prepping for Montreal - Dorr atlas added to BrainGlobe - Few `brainrender` bug fixes - NS - Working with Kimberly and Stella for `brainglobe-template-builder` - AT - AWS hosting sponsorship for S3 hosting - Small docs PRs - VP - Waxholm annotations for new rat template - 3 new animals for an imporved template - PRs - Merge the current version rat template script for versioning (VP/AF) - AF review https://github.com/brainglobe/brainglobe-segmentation/pull/223 :white_check_mark: - AF follow up on https://github.com/brainglobe/brainglobe-atlasapi/pull/631 :white_check_mark: - IT take over `brainglobe-registration` PRs from GSoC - AF follow up on https://github.com/brainglobe/brainglobe-utils/pull/131 - Issues - Double check the desciption of https://github.com/brainglobe/brainglobe-atlasapi/issues/666 to make sure it has enough context to be tagged as a good first issue :white_check_mark: - https://github.com/brainglobe/brainrender-napari/issues/218 AF and Nick Thomas-Low (superfrog) continue conversation on [Zulip](https://brainglobe.zulipchat.com/#narrow/channel/408841-development/topic/Canonical.20annotation.20colors/with/546624495) :white_check_mark: # 2 Oct - ARC TI planning coming up, will try to recruit as many people as possible before the end of November - Hopefully at least Kimberly - Crab atlas - There are good images, need to be processed - Crab template will follow - Atlas building scripts/preprocessing moving to [atlas-forge](https://github.com/brainglobe/atlas-forge) - https://github.com/brainglobe/brainglobe-atlasapi/pull/440 closed due to inactivity - IT/SH follow up on [brainglobe-registration/124](https://github.com/brainglobe/brainglobe-registration/pull/124) - Atlas ontology - All entries in the ontology must have at least one pixel in the annotation stack - Entries in the ontology may have a mesh, meshes are only for visualisation purposes - Need to follow up on [cellfinder/443](https://github.com/brainglobe/cellfinder/issues/443) - Test fix - Pin `napari` - Remove xfails - Symmetrical BG atlases can be defined if the plane can be defined by a simple index (axis aligned) - Does not have to be in the middle of the template - `napari` badges are back # 18 Sep - Sphinx gallery in the process of being added to brainglobe.info - [brainglobe.github.io/pull/381](https://github.com/brainglobe/brainglobe.github.io/pull/381) - New atlases - [DeMBA](https://brainglobe.info/blog/DeMBA.html) - [Enhanced and Unified Mouse Brain Atlas v2](https://brainglobe.info/blog/kim-isotropic-mouse-brain-atlas-added.html) - `brainglobe-atlasapi` - `BrainGlobeAtlasAllen` can load from Allen S3 bucket - [brainglobe-atlasapi/dev-allen-organization](https://github.com/brainglobe/brainglobe-atlasapi/tree/dev-allen-organization) - Sphinx gallery - Where do the API dev docs go? - GUI and API tutorials should exists in the same place - Need to over ride the default main page generated by Sphinx gallery - IT follow up with [image.sc issue](https://forum.image.sc/t/probe-tracking-results-reversed-in-brainrender-after-exporting-from-napari/116572) - IT follow up on [brainglobe-registration/pull/124](https://github.com/brainglobe/brainglobe-registration/pull/124) - AT may look at [brainglobe-napari-io/pull/102](https://github.com/brainglobe/brainglobe-napari-io/pull/102) - IT will look at [brainglobe-workflows/pull/151](https://github.com/brainglobe/brainglobe-workflows/pull/151) # 4 Sep `brainglobe-template-builder` - Install works WSL, but not Windows natively (which may be okay) `brainglobe-atlasapi` - Document recieved from Allen outlining new atlas spec, will be used to align `brainglobe-atlasapi` V2 [Demba dev mouse](https://github.com/brainglobe/brainglobe-atlasapi/pull/492) - Atlases can either be downloaded one by one, or all at once. All at once is 500+GB. - Will write script for downloading one by one, validate, upload to GIN, and delete local copy - Need to replace `allensdk` with BG hosted ontology, instead of relying on `allensdk` `brainreg` - `napari` plugin currently broken see https://github.com/brainglobe/brainreg/issues/254 - Follow up in a week or two PRs - https://github.com/brainglobe/brainglobe-atlasapi/pull/633 IT follow up Issues - https://github.com/brainglobe/brainglobe-atlasapi/issues/629 IT follow up - https://github.com/brainglobe/brainreg/issues/252 go back to no t using `napari` to load 2D stacks for processing # 21 Aug Quaterly reflection and planning # 7 Aug Alessandro - Small PR reviews - High priority on OSW Viktor - 4 animals to perfuse for rat atlas to be imaged in different anatomical orientations Igor - [Docs PR review](https://github.com/brainglobe/brainglobe.github.io/pull/329) - [`brainglobe-atlaspi` docs review](https://github.com/brainglobe/brainglobe-atlasapi/pull/584) - Working on OSW teaching materials Saarah - `napari` widget PR almost done, blog post in progress Adam - Working on grants for BrainGlobe # 1 May planning next 3 months - version 1.5 - there is a dev 1.5 branch - more improvements - registry file: how to - force-fetch option (--local?) - input needed: - metadata - coordinate space - could this be an extensible YAML schema - how does BG know what atlas is in the same space? - zarr header can store affine matrices - but on higher level: what is the concept of compatibility between spaces - Template building - Continue with RAtlas (annotations, new data etc...) - Fly wing progressing (imaging nearly done) - Molerat waiting on annotations from anatomists - Zebrafinch waiting on imaging - Fiddler waiting on imaging - Bone template for grant - Blackcap reviews should be done this quarter - Goals for next quarter: - Refactor brainglobe-template-building - Update figures for Blackcap paper - Make progress on SWC Mouse/Rat - Prioritise ray/finch/Crab - Write an annotation guide - Generic guide - Dos and do nots - ITK Snap based - Input ready for packaging script - brainrender-napari - Didn't move forward much due to GSOC - Host our own data - `brainglobe-dataapi` - Postpone until `brainglobe-atlasapi` V2 - brainglobe-stitch - ready to advertise after incorporating Tschern's comments - dependency on bigstitcher with changing API might cause us problems in the future - brainglobe-registration - GSoC student may work on auto-estimate of subvolume or 2D position (if we get her) - might get involved in registration grant with KCL (point-clouds) - more work needed to make it work with downstream tools ## 3 Apr - ## 6 Mar - No need to split Rat brains manually, can mirror all brains to get symmetry in the atlas - GSoC - Can be strict - Enforce good practice from the start - `brainglobe-atlasapi` test coverage - Unused function found in the `brainglobe-atlasapi` - Can remove if not used anywhere in BrainGlobe globally - Add BrainGlobe [citation counts](https://github.com/brainglobe/brainglobe.github.io/pull/295) to the NIU dashboard - Follow up with [DemBA](https://github.com/brainglobe/brainglobe-atlasapi/pull/492) and [Allen](https://github.com/brainglobe/brainglobe-atlasapi/pull/500) CCFV3 atlases - Follow up with https://github.com/brainglobe/brainglobe-utils/pull/116 - Close https://github.com/brainglobe/brainglobe-atlasapi/pull/440 for now - Check https://github.com/brainglobe/brainrender/issues/410 locally in a notebook environment - IT to try reproducing https://forum.image.sc/t/brainglobe-output-files-do-not-comport-with-cellfinder-visualizations-in-napari/109269 with our test data, follow up afterwards ## 20 Feb - Better advertise existing tools - regular blog posts? - advertise all new versions ## 6 Feb - Suggestions/thoughts on GSOC projects - explore resnet alternatives for cellfinder - rewrite packaging scripts for atlas API v2 - make `elastix` go faster - 2d cellfinder - arbitrary number of channels - merge brainrender-napari and brainrender - - what to do with `morphapi` and feature-equivalence between `brainrender-napari` and `brainrender` - AF to look into feasibility - More detailed planning for ARC work - https://app.codecov.io/gh/brainglobe - suggested order of priority of getting coverage 90%+ - brainglobe-atlasapi :fire: - brainglobe-segmentation - brainglobe-heatmap - brainreg (?) - morphapi (?) - ccf-translator (?) - Discuss how symmetry is handled in brainglobe-atlasapi - Should we enforce it down to the pixel level? - mention in how-to-package docs + template script. - Jung Woo would like another BG project - choose whatever issue - `brainglobe-registration` option to mask registration target by annotation>0 - AF to open issue - Plan for API v2 - meet in late March to look at Igor's prototype https://github.com/brainglobe/brainglobe-heatmap/pull/71 make draft and ping Carlo https://github.com/brainglobe/cellfinder/issues/479 Igor to open issue and close this one https://github.com/brainglobe/brainrender/issues/404 Adam to politely decline ## 23 jan plan for atlas generation: docs for annotating well (including strategies) and for preparing data for template creation, keep template workflow internal for now. - test data preparation for rat, with Viktor? - doc symmetric v asymmetric, - revisit morphio? remove allensdk longterm: simplify: big part: brainrender napari/metapackage just bugfixes for render + heatmap rat atlas - liaise with oslo - release raw data on Brain Image Library - SWC data is uniquely good: enough for M/F and developmetal stages atlases - people won't mind sharing autofluorescence channel - rats are important animal models, but they don't vomit igor and ale to time-limited pair-program on https://github.com/brainglobe/brainglobe-heatmap/issues/72 ## 9 Jan Carlo: - LSFM atlases - MRI atlases: - annotation + ontology problem - regions don't exist at all developmental stages. - number of atlases problem: issue - annotation resolution problem Kaixiang: template generation of developing fly-wing, starts next week. ## 12 Dec - parellelisation with qbatch is magic! - Kaixiang (MSc student) starting his project in January, we will try to make a flywing atlas. - Blackcap atlas 1.1.: how should we coordinate advertisement of blackcap atlas (blog, various lists of available atlases) - let's meet or at least coordinate async the remaining tasks and their order. - brainglobe-registration - go with elastix suggestion first - saving the brainreg way for now - working on area of regions using flattened array count: L+R+total - rat atlas: - NS has done QC and diagnostics - symmetrisation by flipping each brain - small fixes like increasing padding need doing - PRs and issues ## 28 Nov - make microscopy rat brain template for waxholm - new cloud platform to host atlas data - cloudflare costs are likely v small - ask for special exception from ebrains or dandi - adam to find an cost-limiting way to privately fund - CI failures - release API - brainrender has unreproducible bug, somewhere in vedo and vtk: new version of vtk released 3 days ago? - try pin to older vtk - jasmine's problem - atlas grant - niko to link to best rat - af to link to molerat test template - Jung woo and henry - nwb - probe interface ## 14 November AF+IT: slackbot AF: bone atlas grant NS: blackcap figures AT: worked on cellfinder for blackcap, but not finished; focussing on QQR blog about blackcap on BG website? yes, but template building process as separate maintained high-level docs. ## 31 October ## 18 October - elena driosti new zebrafish atlas - Igor to put in touch - rat atlas: niko will report when he has finished counting rat images we can use from SWC - Alessandro: try to merge some major stale PRs - Viktor: continuing atlas packaging once PR review, will help with rat atlas - Igor: napari readers cannot yield, and cannot return within a context manager - Kimberly to come for lunch to discuss BigDataViewer and BigStitcher - Rat atlas: - some data on winstor and some on ceph? - Niko to downsample and explore data, Viktor can help - some samples may miss cerebellum ## 3 October - should we use the project board more and look at it every two weeks? - for BG atlas-api maybe a multirepo board would be useful - currently just issues per repo works - project board for wrapping up CZI grant - good for next round of EOSS? - close to finish - improve testing if easy - register 2d data for 3 species if possible - example data - visualise other things in br-napari (morphapi at least... maybe heatmap) - would napari-vedo bridge help - ## 19th September - funding plans - Atlas generation facility: RSE and imaging technician time. - frog grant: Adam helping, may lead to bigger application - BBSRC transformative tech: BoneGlobe - mid-january: BBSRC standard. Opens presumably in late October. - atlas gen - make 50um blackcap the example? - cellfinder PR - merge - make issue for exposing - write docs as separate PR - data sharing - Globus, Cloudflare, Dropbox, Gdrive, dryad, figshare, datalad, all of these? - distributed solution might be nice (people can make their own), but QC and syncing may cause trouble. - no great solution - maybe a good first task for new data steward? - Adam to talk to Datalad people at INCF - bird figures - flip order of B and C - D can be close-ups instead - add icon for direction to 4x4 supplementary figure ## 5th September - standardising packaging scripts: - `config.py` - template ## 25th July BrainTrawler - initial/intermediate step: a python API that creates a URL - some bureaucratic and financial limitations - longterm: BrainGlobe DataAPI - maybe Jamie Dean lab. - initially merge data-fetching from brainrender + morphapi Atlas packaging - Black cap (AF): ready, waiting for better annotations - Mouse barrel cortex (@abisi): more complicated than expected - Axolotl (@saimaabdus19): pair-programming Tuesday, should be done next week - Prarie vole (Sacha H-A?): a csv-manipulation job, seems to be on it - Human (@imeMFK01?): sent initial info to get involved - see what happens - Cuttlefish (Jung-Woo?): discuss next Friday Atlas building: * tadpole * top of the list, but data not great * crabs * molerat - keen ## 19th July (extraordinary dev meeting with Matt McCormick) - interface with tools the best way possible - mostly registration, and maybe template generation - elastix - BrainGlobe moving away from niftireg - want its flexibility and ease-of-use - want `brainglobe-registration` to be a light-weight as possible - what are plans on elastix side? - be good to keep in touch periodically - 5.2 just started (C++ release, Python coming up) - hackathon end of next week: transforms as first class object (particularly OME-zarr) - possibility to apply transform on-demand, on various datatypes - speak about transforms in BrainGlobe next Friday? BrainGlobe has dodged this decision so far, wants to adopt commmunity standard? - Igor to attend - ANTS people to attend too - other opportunities to collaborate - large scale out-of-core registration - useful for template building - nifti files don't scale - OME-zarr transformation foundation of this - abstraction allows decoupling of backend registration tool from interface - Matt working on version of ANTs that doesn't write to disk - accessibility of template building? - BrainGlobe would like to do this - Matt agrees there's a need - Matt working on template for mouse brains imaged with custom light-sheet microscope, + registration to allen CCF3. ## 11 July 2024 - [PR triage](https://github.com/search?q=org%3Abrainglobe+++&type=pullrequests&state=open) - AF to update brainrender-napari docs - qt widgets leaking :scream_cat: - GIN PRs - brainglobe-space map_points_to is weird - see https://github.com/brainglobe/bg-atlasgen/issues/61, which we needed to fix the WHS rat - but reverting that change needed to get at least P14 admba atlas (possibly all ADMBA) mesh and annotations to match - we need to understand better how the order of flipping and scaling points matters. - meet with Sofia next wednesday - documentation and planning of atlas packaging refactoring (AF, AT, VP - others welcome) - e.g. axolotl atlas has two sets of labels for the same reference - how do we keep it simple (so you don't need a CS degree to package an atlas) - links a bit to API V2: - how to deal with several references - how to deal with several resolutions - how to deal with anisotropic atlases (naming and in general) - don't think too far ahead - don't wait for this to happen if there is interest in a certain atlas - issues to open: - consistently use pooch - label atlases with commit in metadata - standardise everything to ASR - helper functions for streamlining variable data into expected data types - already have - brainglobe-space - load_any (but could include niftis?) - definitely need - reformat ITK-snap labels txt file, and other standards (Amira, ) - standardise order of generic actions (orientation, mesh generation, etc, ...) - write some docs in a draft PR to the documentation, that describes the standard script to package an atlas def download ## 27 June 2024 * 50% of 5000-line PR reviewed * napari work going forward * paper just published still uses "cellfinder" (now brainmapper) * otherwise very happy customers, including BlueBrain people * atlas * pigeons! good quality data, lots of people, routine light-sheet * cetacean imaging: 1 atlas made? registered and started naive averaging * soma splitting code for swc files made by Igor for Steve Lenzi, where does it go? * morphapi - just calls morphio. * brainrender-utils * how often is this used? * swc doesn't specify single soma * solution: * split in morphapi and pass to morphio ## 13 June 2024 - discuss all open PRs - use of zulip as user forum? nudge not push - planning Will and Kimberly's work - introducing Saima - brainreg installation issues :cry: : https://github.com/brainglobe/brainreg/issues/206 and 205 - AT: Matt McCormick: going to work on itk-elastix. Might additionally wrap ants. Also good potential collaborator. Also working on template creating tools (for specific experiments). - NS: packaging issues. With napari. and with brainglobe-utils. Possible solution: Move more things to brainglobe utils extras. - NS: will start first attempt at template building for blackcaps when ceph is back (was scheduled to be offline yesterday) ## 30th May * please confirm you're happy to be an author * Viktor and Adam nodded * atlas generation * kim dev atlases slow but underway * we should refactor loads once all atlases are fixed * e.g. use pooch everywhere * avoid mesh generation using lots of RAM * kim dev and kim mouse write to same folder?? * meta issue for refactoring (Alessandro) * blackcaps * Niko aims to create a population average (symmetric) * preprocessing steps almost ready * intensity correction * splits along mid-sagittal plane * seems fast for 50um * will spend a few more days next week * ANTS: installation via conda is OK, but not all commands work (missing c libs). Used system-level installation on desktop. Installation now easier thanks to bash script for local compilation. * should we get IT to compile it on the HPC and make it available as a module? Yes. Niko to start process. * Alessandro has started working on the packaging script * use median filter for annotation? * at least fix "islands" * add instructions for annotators to avoid anisotropic images, and to check their annotations from 3 orthogonal planes. altases - make zero versions as beta * would be nice to support local atlases. * weekly run on pypi * CI fixed for cellfinder, rc out. * macos-latest workflows CI broken ## 16th May \ ## 2nd May - (FYI) Will/Kimberly back from 12th/17th June to 5th July using up rest of ARC budget - Atlas generation, black cap ms - Alessandro And Niko should have some time soon - PR triage - we should do this every dev meeting - some PR review reassigned - there is a Brainreg mamba workflow already - cellfinder PRs - lots open but we know what's going on ## 18th April (minutes by @adamltyson, which is why they are sparse) - Alessandro away :( - Atlas generation update - Progress being made across all fronts - External PRs (& making sure we have reviewers) - Only one PR (well 3 linked ones) has had no comments yet, @adamltyson will let the author know we will get back to him - Sporadic test segmentation faults on cellfinder CI with numba disabled - Igor is going to look into this - Timing of the blog post for cellfinder keras 3.0 upgrade - To come after merging the keras 3.0 branch, but before release ## 4 April 2024 No items - cancelled. ## 21 March 2024 Simon presented Blackcap work. General catch-up. ## 7 March 2024 - Atlas generation - updates on black cap annotations (Simon W.) - potential publication in journal of comparative Neurology: see an example brain atlas description : https://onlinelibrary.wiley.com/doi/10.1002/cne.24647 - what do we count as contributors? - people that have tested bugfixes on their data? nope. - " a substantial effort to making BrainGlobe as a whole better, not just for the thing they need it right now for" - UCL symposium poster - yes - who wants to present? Igor as test run for SciPy - will submit a title - conda pmcubes contributor? - remove from maintainers? yes, can be added later - new PR without the original developer - Will to change the one file and ask for review from conda - atlas generation and packaging - crab, marsupial and cavefish progressing - Niko, Viktor and Alessandro to make this a focus in the next quarter ## 22 Feb 2024 - updates on some bugs: - https://github.com/brainglobe/brainglobe-workflows/issues/83 - Potentially related to transformations between pixels/atlas/um - niftyreg is passed mm - https://github.com/brainglobe/cellfinder/pull/388 - uint16 bug, seems to be fixed - waiting on feedback from 2 users - Suggestion to document our unit conventions - Alessandro and Igor will work on generic BG poster that can be reused across conferences this year ## 8 Feb 2024 - Pandas 3.0 deprecation warning related to Pyarrow - yes, just add as dep - Niko: should reusable widget be refactored out of `brainglobe-utils` into a separate package (`brainglobe-widgets?`)? - would like to use widgets without utils - not a problem for now - Adam: don't do anything unless blocking e.g movement ## 25 Jan 2024 Results from Niko/Igor/Ale hack - https://hackmd.io/@y2ILkV3LTr2jyn3wAIxPIw/HyFd0WdKa - not this TI, but maybe next - Niko may have time, and Viktor's validation scripts may be ready Brainmapper on new BrainGlobe HPC module times out :scream: AF has a few things cooking and is prioritising in this order - Will's PR requests - napari layergroups - developer docs - bg-atlasapi widget for version management using proxymodel NS update on template builder work - main contribution is top-level user-friendly wrapper around hard-to-use functionality - still early days VP merged two PRs - next step is writing a test - what is the next validation step/refactor? IT - brainglobe-registration: - now uses collapsible widgets - added functionality to scale to resize sample image to atlas resolution - non-orthogonal slicing not implemented in napari: Igor has implemented this himself for now with dask - needs a preview and some blocking a button for threading to work correctly. - some napari info might be useful to propogate back to the registration backend - brainglobe-stitcher - microscope takes tiles with 10% overlap - current stitching tools take days for 180G dataset, but this is improvable - this is due to IO and memory - implementation is a wrapper around the ImageJ BigStitcher logic. Calls command line, no need for ImageJ-Python bridge to exchange data. Works with OME-zarr and hdf5 files, but not tiff (can one write tiffs chunk by chunk) - provide one level of a pyramid as tiff for preview, but keep zarr for downstream analysis - zarr compression backend `zstandard` seems to be doing a very good job at handling the data writing. - Peter G to alpha-test, lot of productionisation potential further down the line. - napari widget design may take some time WG - lots of PRs around renaming `bg-space` (v low-level) to `brainglobe-space` - no access to `bg-space` PyPI - similar task planned for `bg-atlasapi` and `bg-atlasgen` - working on citation in the meantime - citation.cff for each repo - tool to ask for citation e.g. `brainglobe.cite('brainreg','bibtex')` - current tools work for software, but not (traditional) articles. May need to write at least part of this on our own. Need mega-issue for Kimberly - Igor suggested tests for brainglobe-utils, and reducing code duplication - agreement this is good idea - where to look for duplicate code - at least BG-headerwidget and dialog - https://github.com/brainglobe/brainrender-napari/blob/014f5c5908065ddaa5d6b05ecdf90493383cfa2f/tests/conftest.py#L11 - needs to be trustworthy - nice if everything just would depend on utils and space ## 4 Jan 2024 - standardise naming of versions - e.g. v1.2.3 - not "dependency updates" - (AF) `pathlib.Path(__file__).parent....` vs `importlib.resources` - prefer `importlib.resources` - ARC TI report: what has been achieved, lessons learnt, needs for next TI. The remaning funding is ~1/2 a person till Nov 2024. - to discuss over the next two weeks - at the very least, planning to have 0.5FTE of Will next TI - (AF)-`brainrender-napari`: - split table for updating and downloading - separate tableviews for local and remote atlases - to think about in the future: move manager widgets into brainglobe-atlasapi - have `bg-atlasapi[gui]` and `bg-atlasapi[napari]` - presence at COSYNE hackathon: maybe...? - (AF) atomic 3-month projects (potential projects for ARC staff?) - Will G might be interested in not-packaging and maybe visualisation-related? - How should BG functionality be exposed? (Design Python API and expose it in the meta-package) - Wrap up conda packaging for brainglobe? - Improve test coverage? - Remove test data from code repo? - org-level .github, general reduction of duplication (Paddy?) - contributions to core napari in the interest of BG? 1. Get everything on `conda` 2. reduce duplication on org-level (.github) 3. push napari group layers forward - postpone retro - Friday 12th with issue triage - then move to week on/off model (AF to update calendar) ## 7 Dec 2023 * Adrien tends against merging `morphapi` with other API to avoid bloated dependencies for neuromorph * Authorship & email on PyPI Everyone happy with `authors = [{ name = "BrainGlobe Developers", email = "hello@brainglobe.info" }]` as a default? * yes (AF) - publish a list of new and existing contributors each year? * schedule a 3-month retro sometime in January * plus items from Zulip: re-release brainrender, brainglobe course, users can still use napari GUI installer to install brainreg-napari, cellfinder-napari ---- * Attendees: Alessandro, Will, Adam, Sofia, Viktor * Public vs private dev notes * Continue with technical items public, * some agenda items public, so that people can assess if they want to attend a meeting or not * hack.md notes stay private * we continue with current approach. Reassess if we get many external collaborations. * Authorship in PyPI (what goes in pyproject.toml) * BG devs and generic email - everyone is fine with that * AF suggests a yearly thankyou explicitly naming, maybe in the website? * AT: In the About section the core dev and steering committee is outdated. Remove governance model. We could outline there who is involved in what ("Who are we"). We are in favour of being broad in terms of acknowledging contributors, but also low admin / maintenance. AT will write an issue to rethink gov model. * WG: maybe a who-are-we and also a thank-you page (people who were integral but no longer active in it) - nice idea * Retrospective * last one was in June when David Stansby left * AF thinks it could be useful and helpful * keep it with UCL internal for now, not with externals / BG as a community * usually every 3 months, around Jan? * WG: should we align with issue triage (1pm) or separate? * 11th Jan 2.30pm * Collab work with Adrien * discussing how to work together? * prevent him from merging his own PRs, just like with anyone else * Rereleased brainrender * two new issues * Igor and AF were thinking about deprecating the GUI soon-ish (it doesn't have much functionalities) and recommend to use brainrender-napari instead. * AT suggests to do it the standard way, add a deprecation notice to the GUI if people download it. * The little functionalities that the GUI should easily be implemented in the brainrender-napari GUI. Maybe an afternoon's job? Maybe just enough to document it? * AF will add to is TODO list * Brainglobe course * AT and AF gave this first course on 6/12/23 * well-attended * also found a lot of bugs * installation is still tricky across OS (Apple Silicon) and still need to figure out the best strategy (conda + pip?) * but WG did an excellent job on the packaging side :D * many of the issues related to napari? * Installing from napari GUI * cellfinder-napari, brainreg-napari ---> need to be renamed? * we do want to give this option to the users, but under the new names * WG: all the files we need are in the brainreg repo? It could just be adding brainreg to the pyproject. * Things can be hidden in napari (https://github.com/chanzuckerberg/napari-hub/wiki/Customizing-your-plugin's-listing#visibility) * WG can maybe take over this, or have a go. * WG we'll need to make a new release with the new classifier (the ``“Framework :: napari”`` trove classifier ) * napari GUI fetches what is downloadable from PyPI * framework napari classifier needs to be added to brainreg * and add the hidden tag to the napari config to brainreg-napari and brainglobe segment? * WG's updates * we are good to go into next stage: * merge cellfinder into workflows * current cellfinder marked as deprecated, archive main branch * next (after Xmas): pull stuff where current cellfinder is, keep cellfinder and cellfinder-core branches separate * big issue in BG org repository * blogpost has been merged * AT's updates * next TI will spend time working on the issues backlog * AF's updates * we would like to keep WG, AF and AT will have a selection of projects to work in collab with ARC * SM's updates * started with BG this TI * priorities * Viktor * no updates ## 30 Nov 2023 (Atlas generation meeting) * Viktor has added another rat sample ready to feed into a new rat atlas * Igor to add the fixed, slightly differently smoothed kim atlas at all res * lucky to get 4 more (non-fluorescently-tagged) blackcap brains * collaborators are sending annotations in itksnap (including magneto-sensitive regions) tomorrow ## 24 Nov 2023 (Extraordinary issue clean-up) - categories - quickfix - assigned (to be worked on "soon") - unassigned (to be worked on "at some point") - close - ## 23 Nov 2023 (General dev meeting) - `morphapi` - AF to try to figure out Adrien Berchet's current deal - Issue triage on Friday (1.30pm) SWC library & (zoom if needed). - tidy, close, prioritise - Will to attend in person - SfN update (Adam) - people are using it, which is good! - CZI grant application (Adam) - 2x applications in progress... to discuss after meeting. - [website crawling issue](https://github.com/brainglobe/brainglobe.github.io/issues/65#issuecomment-1776936927) (Alessandro) - Igor to investigate further - Sphinx-gallery for documentation screenshots? (Alessandro) - seems to be the way napari and napari-matplotlib do it. - no better thoughts right now - AF will make one of these work. - When can we make a new release of brainglobe-meta? (Alessandro) - Will to publish 0.3.0 with brainglobe-segmentation available, but cellfinder mega-merge not yet - cellfinder-core-napari-merge in the works, but will wait to merge until - Publish zoom link and dev meeting notes? (Alessandro) - new public (not web-public) stream ## 16 Nov 2023 (Atlas generation meeting) - no progress beyond a great PR review by Niko, for Viktor to tackle when he's back - people have other priorities right now :) ## 9 Nov 2023 (General dev meeting) * first public developer meeting * quick intro round if anyone new shows up * retrospective of BG mega-merge: * went well * only unforeseen minor hiccup was needing to manually `twine upload dist` for renamed package `brainglobe-segmentation` * well done Will Graham :tada: * maybe just before SfN wasn't ideal timing :sweat_smile: :shrug: * AF to ask Will about state of new brainreg and bg-segmentation on conda * from yesterday's napari community meeting: https://github.com/napari/napari/issues/6397 * Do we expect our users to [use DIRECT color mode](https://napari.org/stable/api/napari.layers.Labels.html#napari.layers.Labels.color_mode) for annotations labels? What is the max expected number of annotations (ballpark)? 1000 (according to Niko and my eyeballing of Allen atlases) * if we do, it might mean that visualising the labels in napari might be a bit laggy. * make issue in brainrender napari for if and when we want to use direct mode to map Allen colours to annotation. So we remember performance checking. * AF to improve tutorials next, VP to work on validation next, IT to work on brainrender resurrection and brainglobe-registration next. ## 2 Nov 2023 (Atlas gen meeting) - Adam away on for next meeting (16th) - Niko demos some prototype napari widgets for preprocessing brain (before they are fed to the template-builder pipeline) - Adam suggests "Fill holes" functionality for first plugin - specificaly, several rounds of area opening and closing - more to do - debug Linear algebra step - test for performance - Advice on slurm script for atlas validation - getting out-of-memory error - increasing cores may help -> Viktor will try - we agree on a species-specific tilt of the brain, to be defined after averaging the input images used to create the template. - Adam to try and find potential users at SfN with Niko's sketches. ## 26 October 2023 (General dev meeting) * Sofia working with :crab: until Nov 30th when she returns to BG land. * Website repo is now https://github.com/brainglobe/brainglobe.github.io so we can host individual sites (e.g. quarto presentations) easily at brainglobe.info/repo-name * [Potential roadmap](https://hackmd.io/@adamltyson/rJDCf7hbT) * please have a look and make suggestions * brainrender-napari and possibly brainrender into next version * We should create milestones for individual repos to meet before v1 of BrainGlobe. * Will: could make release brainreg and maybe cellfinder before breaking everything for the metapackage. May make releasing metapackage easier! * Decide on metaplugin(?) architecture * How do users interact with BrainGlobe as a whole within napari * Metaplugin as an interface to other plugins? * Good thing to ask users? * Viktor: maybe have all widgets in one plugin * Sofia: 3 widgets for each stage of the pipeline (import/register/widget) * Igor: power-users can just set up napari/(python is the napari power mode) * possible stages: load/register/detect/analyse/visualise -> light-weight meta-widget around widgets. * There are many old and stale issues. Issue triage day? * yes. Friday hackathon * Sofia update: * architecture decided * asv also ready * initial implementation for cellfinder-core main script * initial setup for internal runner is here * future work: * finish setting up internal runner * refactor workflow * comparative analysis on small data * run on bigger data * publish results on GH pages * expand to other tools * set/improve org-wide settings with renovate https://github.com/UCL-MIRSG/.github * Alessandro to open issue on Brainglobe/Brainglobe * Igor - Feature request to discuss - having some way to import 3D meshes from BG into Imaris for making videos * for presentations (is this already possible?) * yes: inside atlas folder, see meshes/ folder and structures.csv * Adam - scivision meeting update * nice people that we are friends with (Alan Lowe from btrack) * maybe not that useful yet, due to lack of infrastructure and standards * keep an eye because may be useful for future users to upload their models for re-use. * Igor - bg-elastix UI demo * Plan to decide how to represent transformations * really cool: good UI (less overwhelming) * advanced settings: * can we have dropdown boxes inside table? * can we group parameters together * think about how interface would look like for 3d subvolumes of * rename to brainglobe-register/registration or similar * Open up these meetings (or at least advertise on zulip)? * weekly reminder to ask for zoom meeting * Alessandro to post discussions on zulip * Alessandro: Brainrender megafix: [all CI tests fail in different way :laughing: ](https://github.com/brainglobe/brainrender/actions/runs/6653756271/job/18080426353?pr=254) but we have [a plan](https://brainglobe.zulipchat.com/#narrow/stream/408841-development/topic/brainrender.20megafix.20plan/near/398656598). * maybe pair-program soon * hopefully removing GUI code is easy as it was an add-on ## 19 October 2023 (Atlas generation) * Niko - prototype of a tool to assist users in seperating hemispheres ## 12 October 2023(General dev meeting) * Zulip :tada: * Igor - New BrainGlobe registration plugin * Architecture * What should be in napari and what in Python API * data processing before and after registration step can go in Python API * napari-close stuff can go in widget code * Wrap transformation functions to remain independent of napari API * Viktor - user generated files in the brainglobe folder * will users ever provide anything more than one or more reference images and annotations image? * yes, mesh generation is not always based on annotations - e.g. Allen mouse and human * Sofia benchmarking update * design will be script with optional input argument * PR for cellfinder workflow * PR in draft for cellfinder benchmarks * end-to-end benchmark and a few individual benchmarks * self-hosted runner and remote runner * less noise in self-hosted and longer running times: absolute times? * relative benchmarks for new PRs between feature and current main * focus on self-hosted runner first * run on dispatch and schedule * with local config file for small and large dataset * Sofia getting credentials for internal runner * * Alessandro to review cellfinder workflow PR before leaving on holiday. * Alessandro to release `brainrender-napari` before leaving on holiday after linking to docs and padding out readme. ## 5 October 2023 (Atlas generation) * Viktor - apologies (back after Optics coures, from 19th October) * Simon - existing datasets * Niko has what he needs for now * Adam - collaborating labs (no reply for some reason) * not a worry for now * Niko - template generation * Figured out how to install ANTs on HPC (with conda) * Started [brainglobe-template-builder repo](https://github.com/brainglobe/brainglobe-template-builder) and populated README with project scope and background * Read about how the template construction algorithm works * Niko has made nice [explanatory schematic](https://excalidraw.com/#json=afDPTYRPLl9CK7s4n6L4P,uiojoNooR10G4ftwvdqXRw) * use downsampled versions for debugging, but high-res for final version (and then create low-res final versions from high-res final version) * preprocessing step to split brains in two hemispheres for extra data and easy exclusion of damage, maybe easy to implement in napari (for now in slicer or similar though) * use manual anatomical landmarks for this to determine the mid-sagittal plane * initial attempt without "masking" (optional mask determines which pixels to keep for registration) * in practice, 10-20 samples is already very good * Progress on choosing annotation tool? * nope * Atlas packaging improvement - where to begin? * sooner rather than later? Help others contribute * rename repo? `brainglobe-atlas-packaging`? * [Dan Birman](https://danbirman.com/) & [Tyler Sloan](https://www.quorumetrix.com/) interested in contributing to improve mesh generation * a possible refactor here would be to take out the mesh generation function so it can easily be replaced by them. * AF to try and find time with Viktor next week * Start with refactoring? modularisation. * Add new atlases and test for ourselves (e.g. Waxholm v4) ## 28 Sep 2023 * AF needs reviewer for 2 small and 1 bigger (not-yet-ready) PR on `brainrender-napari` (Igor?) * how to deal with differing entry points/IO methods when benchmarking workflows (see [Sofia's PR](https://github.com/brainglobe/brainglobe-workflows/pull/15#issue-1900805697))? * AF got lots of good advice on screenshots * RSE slack * David stansby had already asked and done [this for `napari-matplotlib`](https://napari.zulipchat.com/#narrow/stream/212875-general/topic/Screenshots.20during.20doc.20builds) :joy: * may need to fix leaking qt widgets first? * brainrender-napari work slightly delayed by INCF, teaching and work on BG-workflows * Will is likely back in first part of next TI (from 23rd October) - aim for big upgrade mid-November? * agreed that brainglobe > bg and we'll move to this gradually. ## 13 Sep 2023 - what shape should our workflows take? Python script/sphinx-gallery/python notebook? - can we automate screenshot-taking? - AF to ask RSE slack and napari zulip for advice on screenshots. - Can we define and write down our "currently supported workflows" - bulk feature segmentation: silicon/neuropixel probe tracks - whole brain cell detection - whole brain registration - combination of detection and registration - need to be writing these python scripts soon (and possibly build on them to make documentation/teaching materials/jupyter notebooks) - could be useful to benchmark `bg-atlasapi`: already has a (notebook:)[ https://github.com/brainglobe/bg-atlasapi/blob/main/tutorials/Atlas%20API%20usage.ipynb] - maybe a good place to start - refactoring progress bar for bgatlasapi would be good. - Sofia presenting at Janelia next week: advertise BrainGlobe - benchmarking (sofia), registration (igor), visualisation (alessandro) work ongoing/being picked up ## 31 Aug 2023 Napari plugin [viewer workflow](https://github.com/brainglobe/cellfinder-core/actions/runs/6014955232/workflow) - Do we still need it? - It only creates a _preview_ page of the description of our plugin, but we recommend pip-installing most things - [ ] Service is shutting down anyway so we can remove these from the repos they are in. BrainGlobe & cellfinder package restructuring notes: https://hackmd.io/@willGraham01/SkxXXYhTn/edit (might want to move this to a public place rather than my private hackMD). - See (and edit) that HackMD for planned decisions. - Decided to move _everything_ out of cellfinder into workflows, alias the old CLI script in workflows as cellfinder, then move things into new cellfinder Harvest - worktribe mismatch??? - Alessandro to investigate - Alessandro hasn't forgotten to contribute to progress report ## 31 July 2023 Todo list for brainglobe packaging - The list: https://hackmd.io/@willGraham01/BkO5gpMY2/edit - depend on bg_utils - dependency tree: https://hackmd.io/@willGraham01/SJapWCz92 - build pip dependency diagram - then translate to conda - hopefully done in 2 weeks? Priorities for packaging: - include as much as we can - not brainrender (too old pinned dependencies) ## 20 July 2023 - AF: Will and Sofia schedules for next 3 months - more time-chunking :heart_eyes: - Sofia's asv-velocity PR? - try to finish soon (as a team if needed) - Will G on COMP0233 assignment writing for first two weeks, but will also ensure conda stuff progress - always available for review throughout TI - thank you! - AF: brainrender napari GUI feels close - AF: feels like real-life workflows are urgent now, also for me - AF: balance against getting stuck into napari group layer PR - AF: napari hackday (+ presentation) next week. - talk about current work and lessons learnt - WG: working on one single source of information for all BG packages - discrepancies between pip and conda, conda is fetching outdated dependencies and missing some others - created list of dependencies within BrainGlobe and - brainglobe_utils > imlib progressing. Cellfinder bug fixed, but a different bug in cellfinder-core. - once this is fixed, upload to pypi, upload to conda - conda will have optional dependencies included - morphio still not building on conda - bug on CI not replicable - Ale or Sofia can try to reproduce too if needed - WG: - missing from conda brainglobe-meta - cellfinder - morphio - pip done - WG: need a clever bumping workflow - SM: only just back on the project, - reading PR comments, - opened issues for further benchmark tests for main functions of various key steps, - benchmarking on CI - cool blog - good approach for benchmarking on GH actions (without own hardware) - if time, work on cellfinder-core tensorflow - SM, WG: `asv` PR - updates docs for pyproject.toml ## 6th July 2023 - Should we open up these meetings? Maybe informally for now (slack?) - Yes. AT to start advertising on slack. - Should we make the slack more public? Maybe zulip (like napari, image.sc)? - zulip v slack - zulip has unlimited messages - zulip is publicly readable (without an account) - easier for potential new joiners to gauge whether they'd like to join - napari zulip works well - zulip a more open - Technically we're 6 months into the CZI project. Are we on track? - Packaging project is in the spirit of the grant, but not in the grant. - feels like we're kind of on track? - next thing that AF would like to do is end-to-end workflow test - maybe work with Sofia - should we use GH actions or pytest - workflow: BrainGlobe - job: an analysis workflow - step: find cells, count cells, register atlas, - or a simple python script - plan a half-way milestone with an external person? - AT's responsibility to be that person :) - AF unsure about time needed for 2D registration - AT: prototype version for axis-aligned 2D slices seems straightforward - won't need to do ABBA plugin, we'll do elastix instead. - Is it worth looking into porting from TF to PyTorch? - Looks to be the most promising alternative - On conda-forge - Windows support - Part of the linux foundation and the pytorch foundation includes meta, google, AWS, MSFT, NVIDIA etc, so hopefully (!) won't get killed off - Not too hard to convert (?) - Who has the time to do it? - Possibly could be an ARC project? (AT cautious of eating up too much ARC time on non-EOSS work) - AF to ask around in ARC whether someone suitable is interested. - `mypy`: think about how benchmarking and tests folders fit into the package or not. (Current problem is due to package architecture, I _think_) - SWC hiring an RSE to work on image analysis (inc BrainGlobe). Grade 8, 5 years. Spread the word. ## 22nd June 2023 delegation of documentation, particularly now that we have the meta package. - What documentation belongs where: website, repo readme - Website for now. eventually we will include dev/api/cli docs from repos into the main site. - Tool instructions **Packaging** - pypi: everything is good - looking into cellfinder-core versioning bug: blocking brainglobe package on conda - `conda install cellfinder-core=0.4.0` then `pip list` reports `cellfinder-core` is at version `0.0.0` - checkout older release to see if the problem persists - if older release doesn't have bug, likely the switch to `setuptools_scm` that's the culprit. Push a version 0.4.1 with the fix and see if that resolves things - if the final point is correct, will need to be careful across the repos - cellfinder: waiting on multiprogressing-logging to press a button then that can go onto conda & brainglobe can be updated - morphapi: hdf5 issues - can't build the c/c++ library because header files that come with hdf5 pre-req are not in the place they should be - check in with other bluebrain projects **Benchmarking** - asv: benchmarking history, but impossible to have benchmarks of different scales - pyperf is the other alternative - ask arc people for their experiences - start with asv normal usage - look into internal testing (w/ SWC runner) later for realistic use cases - no tests use real-world-representative data: maybe we want our benchmarking tests to be more representative? - large >=40GB files, so can't download as timeouts will occur - needs internal runners - SM added to repo for running things on SWC runner - get started with cellfinder-core: profiling and benchmarking ## 8th June immediate next steps: - Issues for Sofia and Will are drafted :white_check_mark: - `bg_utils` for source code - `bg_benchmarking` for benchmark tests across repos - Developer docs next - maybe expand on BG-specific parts - AT to review - Then bugs or widget? - numba bug/sluggishness? - does it compile correctly? - cuda jit? - reproduce on Troy's machine to have regression tests on paper data about speed (to a reasonable extent) - worst case revert entirely to 0.2.8. - reverting may be non-trivial: - see https://github.com/brainglobe/cellfinder-core/pull/84#issuecomment-1451629764 - maybe this is expected though, because MacOS was not supported at the time? - AF to work on widget this week, and maintenance next week. Then holidays. - bg_atlasapi - metadata could support new tags - metadata should be exposed without needing class instantiation - split atlases off one element (e.g. metadata, reference) at a time Architecture of smaller napari tools - We will eventually have multiple "things" in napari that don't really process data, but are interfaces to existing napari functionality, or do some very simple analysis, things like `napari-brainrender`, `atlas manager`, `brainreg-segment`, "layer interpreters" etc. - While cellfinder, brainreg etc should be their own repo/plugin, should these other things should all be in one place or seperate? - it depends. We'll have `bg_utils` (and others like `bg_benchmarking` and `bg_atlasapi`) for not user-facing tools. `napari` plugins should form part of their backend codebase. AF needed a refresher on layer interpreter idea: - simple functionality like "count points/labels/image in a labels layer" or skimage `regionprops` - exposed through napari plugin(s) - bit like "Analyze" in ImageJ - Examples - [Analyse volume of brain regions](https://github.com/brainglobe/cellfinder/blob/460af929f1d4053aa1998931dd47fe845b6330ec/cellfinder/analyse/analyse.py#L40) - [Assign cells to brain regions](https://github.com/brainglobe/cellfinder/blob/460af929f1d4053aa1998931dd47fe845b6330ec/cellfinder/analyse/analyse.py#L75) - [Count cells per brain region](https://github.com/brainglobe/cellfinder/blob/460af929f1d4053aa1998931dd47fe845b6330ec/cellfinder/analyse/analyse.py#L139) - [Assign segments of a 1D track to a brain region](https://github.com/brainglobe/brainreg-segment/blob/a84d4b2fab53c256590b79d57beb1eac5810ef0f/brainreg_segment/tracks/analysis.py#L58) - [Summarise (size etc) segmented 2/3D region](https://github.com/brainglobe/brainreg-segment/blob/a84d4b2fab53c256590b79d57beb1eac5810ef0f/brainreg_segment/regions/analysis.py#L44) - [Summarise segmented 2/3D region in context of atlas (e.g. 1/3 in region A, 2/3 in region B)](https://github.com/brainglobe/brainreg-segment/blob/a84d4b2fab53c256590b79d57beb1eac5810ef0f/brainreg_segment/regions/analysis.py#L102) ## 25th May 2023 ### Retrospective #### What went well? - Project is enjoyable! - David made good progress on work - `numba` refactor went well, helps maintainability, packaging, encouraging other contributions - Solved memory issue we set out to solve - Had good conversations around next steps - Good to do a block of maintenance work at the beginning - Having one full time person on project has been great to get stuff moving #### What didn't go so well? - Existing tests weren't sufficient for catching bugs we introduced - Will have on-site CI for large data - Progress is slow with old (>~5 years) code due to tech debt - Lack of coordinated release process makes it hard to know what to prioritise - Will will think about this in June, try and abstract away package names and build a brainglobe distribution - Had significant context switching with other projects - Alessandro is now on project 100% of time - Didn't keep strictly to project boards and issues in terms of prioritisation - Sunk time into investigating `dask` without it being a priority - Try not to get distracted - finish one thing before moving on - Had a long period of time with 'asymmetric' workloads between two people, which led to review queues building up - One way to improve is to plan in advance which weeks people will be working on things at the same time #### What have we learned - Thought about building a whole "Brainglobe" package/distribution. - Plan out which packages to focus on, given how much time we have. - Time shared on tests/infastructure/docs at the start is well worth it later down the line. - When working on the project fora long time it's easy to see user issues, but there are actually lots of very happy users! - Lots about `numba` - beyond just using `@jit` - still a package in progress though! - More about neuroscience - Lots about BrainGlobe as a project on a big picture #### What still puzzles us? - How good is good enough? - Have some expectations of how much needs to be done in terms of docs/tests/etc. - Do we want to encourage external contributors? If so, how to do this? - How should work be divided up between SWC and ARC? - How do we maintain software and keep up momentum? - How do we keep funding work when CZI grant runs out? - What is the project documentation going to look like in the long term? - What is the best way to transfer knowledge/experience from more experienced members to other members in the project? - What's the best way to make napari plugins programatically? And how do we choose one to keep a consistent development experience? - How to deal with code changing very fast? - Have good onboarding docs - Make sure projects are tightly scoped to one package - Which BrainGlobe repo has which tool? - Mainly due to confusing naming of existing package for historic reasons Celebrate cellfinder-core release :tada: - David happy to review PRs until the end of June - Should use `napari-matplotlib` in a plugin at some point ### Tasks for new ARC team members - plans/tasks for Sofia - would like to learn more about how Python works under the hood, maybe some benchmarking and profiling - could work on adding benchmark tests/reports (both to internal CI and to "normal" tests) - plans/tasks for Will - Would like to learn more about how python packaging works - https://www.pyopensci.org/python-package-guide/ - https://jacobtomlinson.dev/series/creating-an-open-source-python-project-from-scratch/ - https://conda-forge.org/docs/maintainer/adding_pkgs.html - https://www.youtube.com/watch?v=FpIJ7T1Gpjo&t=1640s (SeptembRSE talk on Python wheels) - integrate napari repos into backend repos? - pip & conda metapackage - see issues 3,4,5,12,13 on https://github.com/brainglobe/BrainGlobe/issues ### New docs - Status of new documentation: - https://brainglobe.github.io/new-website/ - https://github.com/brainglobe/new-website - Want to re-think structure of docs, with main entry point being lots of tutorials - Currently mainly migrating content from GitBoook to sphinx - End goal is to have API reference living in individual packages, but centrally hosted on main website - `napari` already does this, so maybe a good template to follow? https://github.com/napari/docs - Want to separate source from style, so final website should be rendered in one go with one sphinx config ### Other - some user research planned for 2 June with LMCB PhD students - AF working on napari PR 5132 (knowledge gained there also useful for Atlas Manager plugin ## 24 April 2023 * David to finish up `cellfinder-core` work and get a new `cellfinder-core` release out, as final work on the project. * Ale to review with priority from https://github.com/brainglobe/cellfinder-core/milestone/1 * pin tensorflow version for this release * memory issue fixed for David and Nicholas * Adam to link Ale to real-life data for manual testing * David using `test_brain_SK_AA_71_3_subset` * Adam working towards new BG webpage with docs and tutorials * new docs to be added there * Ale working on GroupLayer with Lorenzo and others * Release * David to document steps to release * Adam to announce on twitter (for smaller) and image.sc (for larger) * Adam to write a script that is the start of formalising/standardising the test workflows * Adam to check if `asv` is still being actively maintained * alternatively: https://github.com/benchmark-action/github-action-benchmark * or `pytest-benchmark`? ### Agenda * If we had an initial plan for e.g. each [of the tutorials we want in two years](https://github.com/brainglobe/BrainGlobe/issues/11) we could do some [tutorial-driven development](https://chryswoods.github.io/blog/return_of_the_king/#tutorial-driven-development) that may help keep us focussed? * tutorial to visualise an atlas as a start? * https://github.com/brainglobe/BrainGlobe/issues/16 error-handling - how does napari do it? * Ale to chat with Joe, who has similar problems with his data transfer library * write napari best practices for plugins about this (https://napari.org/stable/plugins/best_practices.html#best-practices) * is standardising test data a priority? * everything should run on a standard UCL laptop (lowest spec is currently 16GB RAM, 512 GB SSD, i5 processor) * have some slow tests (on notebooks) that we can run manually or on internal CI on real-life data * use synthetic or downsampled data on CI for speed * `cellfinder-core` dask update x * having a JIT-compiled numba class makes pickling (which dask needs) so we'll leave dask for now (refactoring class will be lots of work) * David has solved memory issue in different way * switch to numba very valuable * easier * Remaining tasks for `cellfinder-core`0.4.0 * should we hold to fix [`tensorflow` 2.12.0 issue](https://github.com/brainglobe/cellfinder-core/issues/103)? x * staffing strategy for the ARC role going forward * AF to think of projects that require minimal intro and are on the scale of 1-2 months. * or try to find one person for whole project? ## ~~16~~ 30 March 2023 ### Agenda - discuss what BG should look like in 2 years - Linked to the above, what is the "product"? BrainGlobe, the individual tools, or a combination of both. - One idea (AT) - BrainGlobe-napari as an integrated suite of interoperable tools (i.e. one set of docs, changelog etc), and then the individual Python tools (cellfinder-core, brainreg etc) slightly seperate (but dependencies of the plugins). Some examples of high-level functionality that would be nice and generic (some of this is in the grant proposal): - Layer interpreters - tutorials that show neuroscientists how to do a typical analysis/workflow. (e.g. [brainreg-segment](https://docs.brainglobe.info/brainreg-segment/applications/silicon-probe-tracking)) Where should we keep a roadmap? - GitHub discussions in a roadmap repository? #### Quick retro and short term planning - progress on numba -> finish before Easter, then work on dask to solve :crossed_fingers: memory issue? - great that numba competes with cython! - DS to focus on moving from `multiprocessing` > `dask` https://github.com/brainglobe/cellfinder-core/issues/118 - Aim to get new release out ~Summer - Atlas manager plugin -> AF to familiarise and have a plan by next meeting? == brainrender? - BG atlas api v2 - only underlying representation to change, API to remain same - moving the docs to sphinx? - structure relatively high-priority and possibly easy - remain close to NIU docs style - speak to Niko and Laura - David S has a branch for this https://github.com/dstansby/cellfinder-core/tree/api-doc - Adam to copy and proofread - pyData + small config + NIU GH action - GH pages vs readthedocs - advantages of readthedocs - versioned docs - no need to maintain a branch for GH pages - can still have custom (sub)domains: https://docs.readthedocs.io/en/stable/custom-domains.html#custom-domains - `cellfinder-core` issue: https://github.com/brainglobe/cellfinder-core/issues/116 - (AF to use the tools to get more familiar with neuroscience side? Maybe fixing https://github.com/brainglobe/cellfinder-napari/pull/117 in the process) - maybe related to https://github.com/brainglobe/cellfinder/issues/216 #### To mono or not to mono? (probably one monorepo for each tool) - from that question of monorepos will follow - disadvantage of monorepo? - cannot have functionality re-used across several napari plugins? - advantage of monorepo - less complicated dependencies in CI tests? - possible ways to structure - 1 repo for each pipeline step, e.g. cellfinder and cellfinder-napari in one repo, but brainreg and brainreg-napari in a separate one - 1 repo for all pipeline steps (e.g. called "brainglobe") and 1 repo for all napari plugins (e.g. "brainglobe-napari") - disadvantage: cannot install just brainreg if I only want that. - everything in its own repo - disadvantage: complicated CI - Paul: but could pin dependencies in CI? Repo plan: - cellfinder-napari to be merged into cellfinder-core [BrainGlobe#3](https://github.com/brainglobe/BrainGlobe/issues/3) - rename cellfinder-core to cellfinder (archive cellfinder)https://github.com/brainglobe/BrainGlobe/issues/6 - merge brainreg-napari into brainreg (https://github.com/brainglobe/BrainGlobe/issues/4) ~ - Napari layer group PR to be merged at unknown date - how can we design the Atlas napari plugin in the meantime - AF to post on zulip (first PM Genevieve, then post in general.) - Add David and Alessandro to authors in pyproject.toml - yes, also credit other people better: - citation file format - https://allcontributors.org - see e.g. https://github.com/alan-turing-institute/the-turing-way#contributors - credit e.g. testers - can have a bot - Adam to work on this - https://github.com/brainglobe/BrainGlobe/issues/2 #### Infrastructure - Running tests as part of conda feedstock (--> requires putting test code inside distributed package) - don't want to include testdata in package though! - have an issue to remove testdata with synthetically generated data and/or pooch to tackle first - not high-priority for now - only brainreg currently has OS-dependent build because of niftyreg binaries (which could be replaced by niftyreg for conda) - https://github.com/brainglobe/BrainGlobe/issues/5, https://github.com/brainglobe/BrainGlobe/issues/7 - Remove `gitpython` and `check-manifest` from dev dependencies? - `gitpython` useful for debugging CLI tools? Keep in `brainreg` and `cellfinder` (https://github.com/brainglobe/BrainGlobe/issues/8) - `check-manifest` to move to `pre-commit` ## 27 Feb 2023 - DS working through [spreadsheet](https://docs.google.com/spreadsheets/d/1fSqYWU580CXNByur-ZYofvyuc7A85YT1KMJRqqiRqa8/edit?usp=sharing) to standardise infastructure/tooling ### To-do on infastructure: - Change default from master -> main - Sonarcloud: remove comments and activate everywhere - Codecov: disable comments, keep badge - Conda to become default - ARM CI - use cirrus for now - GH actions may support this in the future ### `cellfinder` multiprocessing - Issues with memory usage due to multiprocessing - Potentially one way to fix this is to re-write to take advantage of `dask.distributed`, and then dask can handle the scheduling - `cython` code also has maintainability issues - not a widely used language, and also means there's a compiled extension to be maintained - Should split into a - Short term fix for the memory issue - revert changes? - Long term improvements to memory management and multiprocessing - key is that researchers' hardware is used well - most people use BG on a laptop - don't need perfect numerical correspondence - this was an imperfect port [of a java version](https://brainglobe.slack.com/archives/C04M7KBV2JH/p1677498344059919) - cellfinder-core is used by many people so improvements will have big impact ### Initial plan - for now, follow grant application prioritising short term bugfixes as they appear - initial feature is using and visualising Atlases in napari - are napari layer groups an option or do we need our own Atlas Napari layer? - can we interact with a layer group programmatically? - visualisation: either image or mesh - Atlas: - reference image (scan or average of scans) - annotations image (brain/not brain or more complicated) - pre-computed meshes for whole brain and regions (layer group within layer group) - metadata - preprocessing typically required before registering: - unzip, open niche file format, resize/sample, reorient (this is what bg-atlas api helps with) - feature needed: open atlas on its own - view all metadata and visualise various images and meshes - own new repo to replace brainrender,

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.