Zarr Summit Developer Days - Icebreaker Activity

# Zarr Summit Developer Days - Icebreaker Activity **Welcome to Rome! 🇮🇹** Let's get to know each other before we dive into the technical work. Please take a few minutes to introduce yourself below. --- ## How This Works 1. **Add your introduction** under the section below (5-7 minutes) 2. **Read others' introductions** while people are still writing (3-5 minutes) 3. **We'll discuss** a few highlights together (5-7 minutes) **Total time: ~15-20 minutes** --- ## Introductions --- ### Example: Max Jones - **Where I'm from**: Durham, North Carolina (USA) - **My Zarr story**: I started using Zarr while working for a climate non-profit called CarbonPlan. - **What hats do you wear related to Zarr**: Cloud Engineer at Development Seed supporting NASA's effective usage of Zarr; Zarr-Python core dev; VirtualiZarr core dev - **One thing I wish Zarr could do**: Array concatenation - **What I'm hoping to accomplish this week**: Help people build connections and momentum. Also, formalize a multi-scales extension (ideally a cross-disciplinary one) - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: We have annual summits with tangible outcomes from each one. - **Fun fact about me OR what I'm excited to do in Rome**: Go for some runs around the city --- ### Davis Bennett - **Where I came from**: Würzburg, Germany - **My Zarr story**: I started using Zarr as a graduate student, then comitting to the project in my post-PhD job. - **What hats do you wear related to Zarr**: Zarr Python maintainer - **One thing I wish Zarr could do**: Lazy indexing - **What I'm hoping to accomplish this week**: Move various high-impact specs forward, and build infrastructure to ensure compatibility across Zarr implementations. - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: Quoting @maxrjones: "We have annual summits with tangible outcomes from each one." - **Fun fact about me OR what I'm excited to do in Rome**: I'd like to go bouldering. --- ### Josh Moore - **Where I'm from**: I'm originally from Alabama, but I've lived in Germany for some 25 years. - **My Zarr story**: While working with the Human Cell Atlas team from CZI in 2018, it was pointed out to me as a (better) alternative to a stack of TIFFs. - **What hats do you wear related to Zarr**: Zarr steering council, Zarr Python maintainer (largely v2), project lead of OME-Zarr which is a format built on top of Zarr for (large) microscopy images ("OME" stands for Open Microscopy Environment, which is an OSS project perhaps a bit like Pangeo) - **One thing I wish Zarr could do**: Be (or get) stable for many years to come. - **What I'm hoping to accomplish this week**: Mostly get to know people, since I'm only here until Tuesday morning. - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: leads to a regular series of summits that lead to continual growth of the community, onboarding lots of new and exciting people. - **Fun fact about me OR what I'm excited to do in Rome**: I won't be in Rome long enough for anything too exciting, so a fun fact: I grew up in a small town in Alabama (Fairhope) that was built by utopists following the economic principles of Henry George called "Single-tax colony" which was the (anti-capitalist) basis for Monopoly. --- ### Justus Magin - **Where I'm from**: originally from Munich, but I live in Brest, France since about 4 years - **My Zarr story**: mainly I've been using zarr through xarray, but more recently I dove into the zarr-python code to investigate sparse array storage - **What hats do you wear related to Zarr**: xarray, extending eopf-zarr - **One thing I wish Zarr could do**: encoding different array types (e.g. masked, sparse, units, geometries, etc), in particular automatically through zarr-python; variable-length chunks - **What I'm hoping to accomplish this week**: vaguely, to standardize sparse array storage to something that's usable - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - **Fun fact about me OR what I'm excited to do in Rome**: --- ### Tom Nicholas - **Where I'm from**: The UK (a small village in the Cotswolds). But I have lived in New York City (Harlem) for 4 years. - **My Zarr story**: During my PhD I used and contributed to Xarray for multiple years before really understanding what Zarr was. I then went to work with Ryan Abernathey when he was an Oceanography Professor at Columbia, and was a user of Zarr. Then I worked for a non-profit and got put in charge of making Zarr stores from netCDF files - this led to VirtualiZarr. Now I work for Earthmover. - **What hats do you wear related to Zarr**: - VirtualiZarr developer - Bug-raiser - Icechunk advocate - **One thing I wish Zarr could do**: Variable-length chunks - **What I'm hoping to accomplish this week**: Marry Zarr and STAC - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: If we found a way to better integrate with the Arrow ecosystem. - **Fun fact about me OR what I'm excited to do in Rome**: I've been trying to learn to sketch so I'm going to find some old stuff and make a bad drawing of it. --- ### Eric Perlman - **Where I'm from**: Portland, OR, USA - **My Zarr story**: Working at Janelia, Stephan Saalfeld were exporting a portion of our EM connectomics data in n5 format. One day, over beer, he said "there's someone else doing a very similar thing caleld Zarr" - **What hats do you wear related to Zarr**: User-of-tools; work with various groups for using Zarr as their internal storage in processing pipelines. Worked with JAX (Jackson Labs) to convert all their public data into ome-zarr. - **One thing I wish Zarr could do**: Just Work; I end up using a missmatch of multiple libraries (e.g. zarr-python for group management & tensorstore for high performance writing) - **What I'm hoping to accomplish this week**: Relieve some of the pain points & understand where extra focus would be the most useful - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: If the zarr ecosystem becomes easy to use for a typical user; becoming a transparent way to access large data - **Fun fact about me OR what I'm excited to do in Rome**: Eat! --- ### Hugo Gruson - **Where I'm from**: Currently Heidelberg Germany. Originally, France. - **My Zarr story**: I inherited the maintenance of the Rarr R package to - **What hats do you wear related to Zarr**: - **One thing I wish Zarr could do**: - **What I'm hoping to accomplish this week**: - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - **Fun fact about me OR what I'm excited to do in Rome**: --- ### Lachlan Deakin **Where I'm from**: Canberra, Australia - **My Zarr story**: Needed fast chunked I/O for my work and PhD, saw a draft spec for Zarr V3 and decided to just implement it. We use it in my lab to store large tomographic images. - **What hats do you wear related to Zarr**: zarrs (Rust), zarrs-python, + related libraries/tools - **One thing I wish Zarr could do**: support every extension I want to use across various implementations + viewers - **What I'm hoping to accomplish this week**: standardise various extensions (variable shaped chunking) + remove constraint on sharding for evenly divisible chunks - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - **Fun fact about me OR what I'm excited to do in Rome**: --- ### Nishadh Kalladath - **Where I'm from**: From India, living in Nairobi Kenya for last three years - **My Zarr story**: Started using zarr for Impact Based Forecasting a type of risk assessment and Early warning System for Floods and Droughts. Cloud native use of weather forecast is need of the hour and excited about the win-win-win solution provided by zarr/virtualizarr/kerchunk ecosystem. - **What hats do you wear related to Zarr**: User of the tool, documenting methods to use GFS/GEFS forecast reduces the scan_grib in kerchunk routine by using grib index files reduces 95% scan_grib called as grib-index-kerchunk. - **One thing I wish Zarr could do**: Virtual zarr or refernce based access to large open data on weather/climate forecast which boosts cost reduction and avoid duplciated storage of forecast datasets - **What I'm hoping to accomplish this week**: Apart from supporting group out sessions, I would like to work on pencil and pancake chunking demo on netcdf datasets and ecmwf grib-index-kerchunk - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: Hopefully to user of the tools to Developer - Fun fact about me OR what I'm excited to do in Rome: Randonneur, hoping to do cycling --- --- ### Mattia Almansi - **Where I'm from**: Italy - **My Zarr story**: While doing my PhD in oceanography I've started using various packages in the Pangeo ecosystem, including Zarr - **What hats do you wear related to Zarr**: B-Open is involved with many projects using Zarr, the most important is the Earth Data Hub (EDH) - **One thing I wish Zarr could do**: Make it easier to find the best chunking/sharding strategy - **What I'm hoping to accomplish this week**: Understand better how we can improve perfomance using sharding - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: Zarr becomes the main format used in all our projects - **Fun fact about me OR what I'm excited to do in Rome**: I live in Rome, but the Hub is a very cool place! :) --- ### Sebastian Galkin - **Where I'm from**: Argentina (living in Brazil) - **My Zarr story**: I learned about Zarr when I joined Earthmover and they gave me the first task of implementing renaming of groups/arrays. - **What hats do you wear related to Zarr**: Icechunk mantainer - **One thing I wish Zarr could do**: Let "smart" stores optimize away some of the I/O. - **What I'm hoping to accomplish this week**: Gain more understanding of what new features are or will be in development soon, how they may impact Icechunk, and how could Icechunk experience influence their design. - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: all stores use a new high level `Store` interface. - **Fun fact about me OR what I'm excited to do in Rome**: :pizza: :stadium: :walking: --- ### Tom White - **Where I'm from**: I live in Wales in the UK - **My Zarr story**: I started using Zarr to store genomic data for the sgkit project - **What hats do you wear related to Zarr**: VCF Zarr spec, Cubed developer, Zarr bug reports and fixes - **One thing I wish Zarr could do**: store (missing, ragged) strings well - **What I'm hoping to accomplish this week**: meet other community members and hear what they are working on; get movement on how to get v3 format working well for use cases I care about (e.g. strings, missing data); understand the standardization process - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: Zarr gets serious adoption in multiple scientific fields - **Fun fact about me OR what I'm excited to do in Rome**: I've forgotten all the Italian I learnt when I lived in Florence. I'm looking forward to walking a lot in Rome (plus the coffee and wine) --- ### Alistair Miles (@alimanfoo) - **Where I'm from**: Oxford, UK - **My Zarr story**: I got really frustrated with slow file formats for genomic data, and so went looking for something better! Did early work on the Zarr spec design and zarr-python implementation. - **What hats do you wear related to Zarr**: Former zarr-python developer, now Zarr Steering Council member. - **One thing I wish Zarr could do**: Support a vibrant and happy community of developers exploring new approaches to scientific data storage. - **What I'm hoping to accomplish this week**: Support efforts to standardise extensions and make it easier to develop and share extensions. - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: It unlocked an explosion of new extensions and development directions. - **Fun fact about me OR what I'm excited to do in Rome**: Excited to see really old stuff and eath great food and drink good coffee! --- ### John Kirkham - **Where I'm from**: Seattle, WA, USA - **My Zarr story**: Previously I worked at Janelia Research Campus at Howard Hughes Medical Institute with Josh Dudman on calcium imaging data from mice. This generated a bunch of array data, which we needed to process. Tried first with HDF5, but found that didn't work well for our computational needs. Had recently been using Dask and found out about Zarr in the process. It was a great fit! Then started working with Alistair on Zarr. Fast forward many years and so many changes have happened in the community, but the core focus on making array storage work well especially in distributed and cloud settings remains the core of Zarr and its key selling point. - **What hats do you wear related to Zarr**: I am Zarr Steering Council member (ZSC). Have contributed to Zarr Python and Numcodecs (compression alogrithms Zarr uses) - **One thing I wish Zarr could do**: - V3 solves a number of problems we had before. Am interested in supporting its adoption. Also learning any blockers we can address to support that. - V3 provides support for extending the spec in a number of ways. What ways are people here interested in extending the spec? Are there common needs across domains that would be useful in such extensions? - What are the performance issues we are running into today and how can we address them? - **What I'm hoping to accomplish this week**: Learn what this group cares about and try to support as best I can - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: This was the moment where V3 adoption really took off. People found how to address the problem they came to solve. That many people refer to this summit as an important moment in their careers. - **Fun fact about me OR what I'm excited to do in Rome**: I've been to Rome before and done a fair bit of things here. Still have plenty I haven't done and do enjoy being in this city. Had not visited Castel Sant'Angelo, which I just did yesterday. Also noticed that work on Mucha (Czech artist) is on display, which I've always wanted to see. --- ### Vincent Sarago - **Where I'm from**: France (Bretagne) - **My Zarr story**: People are trying to convince me I should use it. I'm building visualization tool for earth data so for the past 10 years I've been using COG but recently there was a great interest for using Zarr as archival format. - **What hats do you wear related to Zarr**: I'm a user, trying to create visualization tools for Earth Observation data stored in Zarr. - **One thing I wish Zarr could do**: Simple specification for GeoReference and Multiscale - **What I'm hoping to accomplish this week**: just listening - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - **Fun fact about me OR what I'm excited to do in Rome**: I wasn't supposed to be here ### Michelle Roby - **Where I'm from**: Denver, Colorado, USA - **My Zarr story**: I am and have been Zarr adjacent for a few years. My job has be adjacent to a lot of cloud-native efforts at CNG (Cloud-Native Geospatial Forum). At CNG, we do a lot of making the space for great minds to come together and collaborate. - **What hats do you wear related to Zarr**: Making the space! - **One thing I wish Zarr could do**: - **What I'm hoping to accomplish this week**: Learn a lot more about Zarr so that I can support Zarr data users and providers on Source Cooperative. - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - **Fun fact about me OR what I'm excited to do in Rome**: No parlo molto italiano ma je parle francias and hablo español (mejor que italiano). Talk to me in french! Talk to me in spanish! Talk to me in your own made up language, doesn't really matter! ### Mark Kittisopikul - **Where I'm from**: HHMI Janelia Research Campus, Ashburn, Virginia, USA - **My Zarr story**: I first encountered Zarr after working on file formats for microscopy for use by microscope builders. - **What hats do you wear related to Zarr**: I am working on Zarr v3 support in Zarr.jl (Julia package for Zarr) - **One thing I wish Zarr could do**: Optional codecs - **What I'm hoping to accomplish this week**: - Finish working on Zarr.jl Zarr v3 support - Make Zarr v3 shards practical - Discuss an "optional" codec - Matches optional filters in HDF5 - Adds a small header with bitfields to indicate whether nested codecs were successfully applied - Allows implementations to set an upper bound for the size of a compressed chunk (header + raw size) - Make codecs Turing complete ?? - Could you write a program using codecs? - Discuss nesting of shards (recursion) - Consider an external shard index - Currently shard indices may live at the beginning or end of a shard and are accessed by reading a byte range - Allow for an external index (e.g \[chunk-key\].zindex) - Formalize integration of foreign file formats as shards - Address Zstandard and extending Zarr with codecs - Create a test suite for Zarr v3 implementations - Available via HTTP and easily clonable locally - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - We tackle hard questions - We create practical resources - **Fun fact about me OR what I'm excited to do in Rome**: - I once got in trouble for inferring my middle school's computer adminitrator account password was "1230" - I also attend HDF User Group meetings - I used to live in Wiesbaden, Germany when I was five years old ### [Even Rouault] - **Where I'm from**: Maisons Laffitte, near Paris in France - **My Zarr story**: I came to Zarr by developping support for Zarr V2 in GDAL (C++) and then Zarr V3 - **What hats do you wear related to Zarr**: - **One thing I wish Zarr could do**: - **What I'm hoping to accomplish this week**: - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - **Fun fact about me OR what I'm excited to do in Rome**: ### Joe Hamman - **Where I'm from**: Seattle, WA, USA - **My Zarr story**: Through the Pangeo project. I helped create the first Xarray<>Zarr bridge and that got me started on my journey. I have since worked on data visualization using Zarr in the browser (CarbonPlan-maps), ETL tools for creating Zarr archives (Pangeo Forge), web APIs for serving Zarr over HTTP (Xpublish), the Zarr-Python 3 refactor, and Icechunk (Earthmover). - **What hats do you wear related to Zarr**: Zarr-Python dev, CTO at Earthmover where we use Zarr heavily - **One thing I wish Zarr could do**: have a really clean extension story - **What I'm hoping to accomplish this week**: Build collaborations across projects - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: where we activated the developer community and began to solve problems for the user community - **Fun fact about me OR what I'm excited to do in Rome**: I - In Rome: I plan to eat as much as I possibly can 🍕 - After Rome, I'm headed into the countryside with my mountain bike 🚵 ### Julia Signell - **Where I'm from**: Rhode Island, USA - **My Zarr story**: via xarray - **What hats do you wear related to Zarr**: I work at the intersection between STAC and Zarr. - **What I'm hoping to accomplish this week**: - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - **Fun fact about me OR what I'm excited to do in Rome**: Drink some good wine! ### Alan Liddell - **Where I'm from**: Columbus, OH, USA - **My Zarr story**: I started by trying to help stream directly from cameras to Zarrs - **What hats do you wear related to Zarr**: I develop a Zarr writer (acquire-zarr) - **One thing I wish Zarr could do**: - **What I'm hoping to accomplish this week**: - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - **Fun fact about me OR what I'm excited to do in Rome**: Walk around a bunch, see some historical sites, and eat some good food ### [Add your introduction here - copy the template] **Format**: Copy/paste the template below and fill it in with your details. **Template:** ``` ### [Your Name] - **Where I'm from**: - **My Zarr story**: (How did you first get involved with Zarr?) - **What hats do you wear related to Zarr**: - **One thing I wish Zarr could do**: - **What I'm hoping to accomplish this week**: - **Two years from now, what would make you say 'that Rome summit was a turning point'?**: - **Fun fact about me OR what I'm excited to do in Rome**: ``` --- ### [Next person - copy the template] --- ### [Keep adding below...]