---
# System prepended metadata

title: '2022-11-02: scverse scaling discussion'
tags: [' meeting', ' anndata', ' performance', scverse, ' scaling']

---

---
tags: scverse, meeting, performance, scaling, anndata
---

# 2022-11-02: scverse scaling discussion

*Attendees:* Adam, Rahul, Danila, Isaac

## Agenda

* Short presentations on current work
    * Isaac (10 min)
    * Danila (10 min)
* Discussion on:
    * Needs from scvitools
    * Data access patterns + usecases
    * Prioritization

## Notes

* Question from Danila
    * IO for jax array
    * AnnData does not know how to handle Jax arrays
        * Isaac: this should be easy
* Jax + pytorch
* Adam
    * Fast random access to X, layers
    * Concurrency
    * Issues with memmap
* 
* Danila – PostData
    * Working with multiple anndata objects
    * `PostData` object
        * Collection of tables with named axes
        * Contraints on axes
        * Hashed data + unique names for tables
    * Query based on sets of axes/ tags
    * `map` map a function over input
    * `bind`
    * Written as:
        * directory of parquet + json
    * `Shadows`
        * Calling scanpy on a shadow, cached loading on access
        * Cannot subset
    * Parquet storage
        * `uns` as json
            * Seperate complex stuff, then merge
        * Can use SQL files on `parquet`
* Usage from Adam
    * Random access asap
    * Multithreaded access from pytorch
    * Benchmark case?
