NGFF-VIV Presentation by Trevor

--- tags: NGFF --- # NGFF-VIV Presentation by Trevor Attending: Trevor Manzt, Frances, Josh, Will, Simon, Petr, J-m, David, Chris, Emil, Dom, Seb - Deck.gl is a WebGL framework from Uber for customizing rendering - viv broken into: - data loader components ("data not in browser") - getTile() function (also decodes) not called by user/client. - deck.gl layers calling functions to load data - Easiest to use viv from a react project (good support for deck.gl) - deck.gl canvas takes layers - a number of declarative (rendering) properties have existing shader code - no way to update the properties (handled by applications) - avivator - for anything available via HTTP - some advanced rendering - first 4 channels by default - heuristic for the settings - works the same for OME-TIFF and bioformats2raw zarr - more feature rich: - linked pan & zoom - Will: can the channels be different for each? Don't think so. But building blocks are there in the viv library, but the applications needs to manage the state - picture-in-picture - local lens transformations (black-to-white color ramp) - looking to use it as a way to engage with the community about what they would like to see. - vizarr - guessing is less sophisticated - "source" needs to point to a zarr multiscale - wanted to integrate viv into a notebook / remote computation location --> ImJoy - allows you to use a Python backend - ImJoy manages secure data transfer - loaded if in an IFRAME - biolerplate for setting up an ImJoy client - can wrap any zarr-python store for access in viv - jupyter kernel is functioning as a proxy to the image data - No additive blending between images, but up to 6 layers in one image - Discussion - _looked through examples_ - for hacking, suggest cloing imjoy_plugin.py to a forked version of the vizarr (localhost:3000) - recently added a versioned imjoy api (NPM packages) - **Seb**: support for future specs? overlayed masks or multiple images (HCS) **Trevor**: can create separate layers for multiple images, but there are other modalities. Don't yet have anything built into viv for loading masks or other types of layers. But should be a matter of using other deck.gl layers for those modalities. see https://vitessce.io Need to map JSON objects to a deck.gl component. - **Will**: avivator vs. vizarr seems like a lot of duplication. Implement every UI feature twice? **Trevor**: vizarr spawned as a way to quickly provide access in a Jupyter environment. Hopefully could bring all of that into the library, but don't want to proscribe the UI that someone is using. In the long-run, likely want to add ImJoy API to Avivator (demonstrates more of the features of Viv) even though it's much more tied to OME-XML (5D). **Josh:** the future OME model should be a superset of OME-XML and that would be a good target for viv and _all_ of the applications. - **J-m**: how difficult to deal with a plate (grid)? **Trevor:** similar to picture-in-picture but you want a special layer in the Viv library. A grid layer, an array of different loaders that get drawn. A question of what API for that layer. Several questions: same color transformations to all? Same channels? But there's definitely the posibility of having a composite layer. **Will:** you could also load the plate on the fly (new data source in Python) - Addendum (Trevor) - Something I just realized that I didn't this morning but forgot who asked. The "grid" view for a plate could be achieved with viv currently (if each image is not multiscale). Viv's ImageLayer is a deck.gl layer for where the data must be non-pyramidal (e.g. just a zarr array, not list of zarr arrays), but it allows you to pass scale and translate as props as well. - You could create n x m separate ImageLayers and just ensure that each image is translatedon to some arbitrary grid. Deck.gl would take care of rendering - Here is an example of a MultiscaleImageLayer(MxIF image) with an ImageLayer(IMS data, non-pyramidal). We translated and scaled the ImageLayer to register with the layer below. http://vitessce.io/?dataset=vanderbilt&theme=dark - I opened this issue a while ago, but we haven't run into the use case yet to spend the time to add this feature: https://github.com/hms-dbmi/viv/issues/217