[](https://hackmd.io/VRtGf5WjQR6C9grokvpknQ)

Meeting Minutes January 9, 2024 - Kaitai-Awkward
===
**Andrea:**
- Working on tests and wanted to put everything needed for the tests inside the same folder. This works fine with one file! But with two files there's a conflict.
- Is this something that is cause by awkward-kaitai-build?
- Yes, fixed!

**Amy:**
- CDMS v1 data description
- Continue working on our paper

Meeting Minutes December 5, 2023 - Kaitai-Awkward
===
**Amy:** What's Amy supposed to be doing
- CDMS v1 data description
- Continue working on our paper
- Write motivation part of Manasvi's paper, https://www.overleaf.com/4278529621jmmfsyfxqftd#796540
- PyHEP repo - https://github.com/ManasviGoyal/PyHEP-2023-Awkward-Target-for-Kaitai-Struct

https://indico.cern.ch/event/1252095/timetable/#10-awkward-target-for-kaitai-s

> Data formats for scientific data often differ across experiments due the hardware design and availability constraints. To interact with these data formats, researchers have to develop, document and maintain specific analysis software which are often tightly coupled with a particular data format. This proliferation of custom data formats has been a prominent challenge for the Nuclear and High Energy Physics (NHEP) community. Within the Large Hadron Collider (LHC) experiments, this problem has largely been mitigated with the widespread adoption of ROOT.
>
> However, not all experiments in the NHEP community use ROOT for their data formats. Experiments such as Cryogenic Dark Matter Search (CDMS) continue to use custom data formats to meet specific research needs. Therefore, simplifying the process of converting a unique data format to analysis code still holds immense value for the broader NHEP community. We propose adding Awkward Arrays, a Scikit-HEP library for storing nested, variable data into Numpy-like arrays, as a target language for Kaitai Struct for this purpose.
>
> Kaitai Struct is a declarative language that uses a YAML-like description of a binary data structure to generate code, in any supported language, for reading a raw data file. Researchers can simply describe their data format in the Kaitai Struct YAML (.ksy) language just once. Then this KSY format can be converted into a compiled Python module (C++ files wrapped up in pybind11 and Scikit-Build) which takes the raw data and converts it into Awkward Arrays.
>
> This talk will focus on introducing the recent developments in the Awkward Target for Kaitai Struct Language. It will demonstrate the use of given KSY to generate Awkward C++ code using header-only LayoutBuilder and Kaitai Struct Compiler.

Meeting Minutes November 28, 2023 - Kaitai-Awkward
===
**Andrea:** What's the best issue to start with? Manasvi recommends https://github.com/ManasviGoyal/kaitai_struct_awkward_runtime/issues/14.
- the Scala code, https://github.com/ManasviGoyal/kaitai_struct_compiler/blob/1a3cd3b5b5f85c612e5eaa888aed3528b17ed853/shared/src/main/scala/io/kaitai/struct/languages/AwkwardCompiler.scala
- the file that generates the C++ code, of which the AwkwardCompiler was initially a copy: https://github.com/kaitai-io/kaitai_struct_compiler/blob/master/shared/src/main/scala/io/kaitai/struct/languages/CppCompiler.scala
- layout builder tests https://github.com/scikit-hep/awkward/blob/main/header-only/tests/test_1494-layout-builder.cpp
- the layout builder header file https://github.com/scikit-hep/awkward/blob/main/header-only/layout-builder/awkward/LayoutBuilder.h
- https://github.com/scikit-hep/awkward/pull/2408
- https://github.com/scikit-hep/awkward/pull/2553
- previous implementation of IndexedBuilder https://github.com/scikit-hep/awkward/blob/71b334a82570b15eb07f87f6992552d93accabfa/header-only/layout-builder/awkward/LayoutBuilder.h

For a PR into Awkward: LayoutBuilder.h needs a generic `IndexedBuilder` to complement the generic `IndexedOptionBuilder` that it already has. When that exists, it can be used to convert Kaitai "enum" types into Awkward categorical by setting parameters. The only difference between `IndexedBuilder` and `IndexedOptionBuilder` is that it's possible to append a `null` value to an `IndexedOptionBuilder`.

Limitation on data structures that Awkward Array can represent. (In Uproot, you get the `CannotBeAwkward` exception if you encounter ROOT data like this.)

* The data type is a finite tree (no cycles).
* For example,
* you can have "list of records in which field x is an integer and field y is a list of floats,"
* and you can have "record type A in which field x is record type B in which field y is an integer"
* but you can't have "record type A in which field x is record type B in which field y is a list of record types A." The type is cyclic. Values of such a type may be finite, but nevertheless, they're not allowed in Awkward Array.
* If the data type is a DAG, it's equivalent to a tree with common subtrees duplicated.
* For example,
* in "record type A in which (x is a record type B) and (y is a list of records of type B)," you could have just been talking about two record types, B and B', in which the fields of B and B' happen to all be the same.
* Since the data type is a finite tree of depth $n$, values are also finite trees of at most depth $n$.
* This is good for data like "events that contain tracks that contain hits" and bad for data like "generic trees in which each tree node has a list of children, which also have the tree node type."
* The above only limits the kinds of graphs you can build with an Awkward Array. The types of data that can go into that graph are:
* primitive types: booleans, numbers, dates, and time differences
* lists of some other type, with fixed length or variable length
* missing values (`None` in Python); also known as option-type of some other type, or "nullable"
* records in which a fixed set of named fields have specified types (i.e. a C `struct`), and tuples, which have numbered fields instead of named fields
* heterogeneous unions of the above
* other useful data, like strings and catageorical enums, are built out of the above with special `parameter` values to say that, for example, a string is not just a list of uint8
* special case for "unknown type" (singleton)
* Awkward Arrays are implemented by a tree of nested nodes, each of which provides one feature:
* [NumpyArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.NumpyArray.html): primitive types, as general as a `np.ndarray`
* [ListOffsetArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.ListOffsetArray.html): list of X, in which the list boundaries are represented by an `offsets` array (cumulative sum of the array of list lengths)
* [ListArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.ListArray.html): like the above, but `starts` and `stops` are two separate arrays for more flexibility
* [RegularArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.RegularArray.html): like the above, but the list lengths are all equal/fixed, and so it has no need of any `offsets`, `starts`, or `stops`
* [RecordArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.RecordArray.html): implements records and tuples with a list of field names (`None` if tuple) and a list of `contents` with the same length
* [BitMaskedArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.BitMaskedArray.html): makes its `content` nullable with a bit array
* [ByteMaskedArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.ByteMaskedArray.html): same but with a byte masked array
* [UnmaskedArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.UnmaskedArray.html): formally option-type, but no values are actually missing, so no need for a mask
* [UnionArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.UnionArray.html): represents heterogeneous unions ("sum types"), which is the most complicated case
* [EmptyArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.EmptyArray.html): an array of length zero with "unknown" type
* [IndexedArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.IndexedArray.html): an array whose type is the same as its `content`'s type, but rearranged, duplicated, or filtered by a delayed [advanced slice](https://numpy.org/doc/stable/user/basics.indexing.html#integer-array-indexing)
* [IndexedOptionArray](https://awkward-array.org/doc/main/reference/generated/ak.contents.IndexedOptionArray.html): like the above, except that negative indexes are legal; they represent missing values
* Every [Content](https://awkward-array.org/doc/main/reference/generated/ak.contents.Content.html#) (superclass) node type, with its children, map to one of the data types described in the block above
* For finer type-granularity, [Forms](https://awkward-array.org/doc/main/reference/generated/ak.forms.Form.html) are types that are one-to-one with a [Content](https://awkward-array.org/doc/main/reference/generated/ak.contents.Content.html#) node type
* There's more or less one LayoutBuilder type for each [Content](https://awkward-array.org/doc/main/reference/generated/ak.contents.Content.html#) node type, with some minor exceptions that derive from LayoutBuilders only being used in compiled code (C++ and Numba) and all the other data structures being used for Python
* only LayoutBuilder (templated, compile-time) and ArrayBuilder (dynamically discovers data type at runtime) can change arrays, and they do so by appending only at the end
* everything else considers arrays to be immutable

Meeting Minutes November 21, 2023 - Kaitai-Awkward
===
**Manasvi:** Issue with CDMS data v8 seems to have been not unpacking the data from .gz format

Tried to merge upstream and it's failing - need to talk to Andrea about this and see if he can work on it?

For the enums, right now we have a numpy builder. We'd like to add an index builder instead.

Need to set up proper automated tests, right now Manasvi is checking cases by hand.

**Amy:** still working on CDMS data v1. Will send out a new poll for a new meeting time.

Meeting Minutes November 14, 2023 - Kaitai-Awkward
===

**Manasvi:** old thing left: IndexBuilder type in LayoutBuilder.

One error when sending data on the Python side, doesn't find a node?

Meeting Minutes November 7, 2023 - Kaitai-Awkward
===

**Manasvi:**

Needs `LayoutBuilder` nodes in Scala to have `parent` links to implement `UnionBuilder`.

There's a whole class of Kaitai descriptions that can be read into C++ but not into Awkward: those with cycles in their types. This is because Awkward Arrays do not allow arbitrarily deep nesting.

https://en.wikipedia.org/wiki/Topological_sorting

This came up because Manasvi noticed a dependency on order of processing

Publishing? [Computing and Software for Big Science](https://link.springer.com/journal/41781) and [ACAT 2024](https://indico.cern.ch/event/1330797/abstracts/#submit-abstract).

Meeting Minutes October 31, 2023 - Kaitai-Awkward
===

Happy Halloween!

**Amy:** I'm struggling with scdms_v1 :(, something is different about the MIDAS header. Thanks to Manasvi and Ianna for helping with Manasvi's rec letter. Will check in with Andrea on what kind of tests he's comfortable writing - overall tests of the repository, data validation, both?

**Manasvi:** Do we need the instances? Amy: yes, the instances are the human-readable values. Hasn't contacted the Kaitai authors but Ianna encourages her to reach out on gitter starting now. https://app.gitter.im/#/room/#kaitai_struct_Lobby:gitter.im, Amy's username is @pibion, Jim's is @jpivarski.

Our CMake file for the awkward runtime is cloned, so the question is - is that okay? We needed to create only one shared object, whereas their CMake file creates multiple. So we won't inherit changes to their CMake file. Do the kaitai developers want us to update their CMake file, or is a separate CMake file okay?

Meeting Minutes October 24, 2023 - Kaitai-Awkward
===

**Manasvi:** Very nearly done; some questions about where to put some code and documentation. Haven't contacted Kaitai authors yet. ~~Will need a union-type for CDMS.~~ Nope, just three option-type fields, which is much easier.

**Amy:**

**Andrea:** A couple of things so Andrea can help: once documentation exists, Andrea can write tests. We can setup Github actions on https://github.com/ManasviGoyal/kaitai_awkward_runtime to grab development version of awkward array. To get this started, open issue on github and tag `@zonca`.

**Yana:**

Meeting Minutes October 17, 2023 - Kaitai-Awkward
===

Looks like Manasvi can wrap up in a week and a half.

There's a PR on kaitai_struct_compiler and a hand-over of kaitai_struct_awkward_runtime to do. Preparing this will act as a reminder of all of the things that need to get cleaned up. Adding tests, for one thing. Removing parts that were generated by cookiecutter but aren't used. (Any dead code.)

Also the hand-off to Andrea, who will be the long-term maintiner.

>>> import awkward_kaitai
>>> cdmsv1 = awkward_kaitai.Reader("/full/path/awkward_kaitai_cdmsv1.so")
>>> cdmsv8 = awkward_kaitai.Reader("/full/path/awkward_kaitai_cdmsv8.so")
>>> array1 = cdmsv1.load("file_v1.raw")
>>> array8 = cdmsv8.load("file_v8.raw")

Meeting Minutes October 10, 2023 - Kaitai-Awkward
===

**Amy:** working on getting Andreas back on the project; done. He'll be describing some data formats, including ctapipe EventIO! He'll be using SCDMS to make a web API to retrieve individual event data. Haven't rewritten CDMSv1, but it's #1 priority because Andreas will be using it; will get it done next week.

**Ianna:** nope.

**Manasvi:**

* Discussed with Angus about shared and static libraries. For now, need shared; will need to change cpp_runtime to fix that.
* Angus: "The module name will be the repo name". Will have to do something about that. (Presumably, need to write the pyproject.toml.)

Separate Python packages for each KSY:

>>> import awkward_kaitai_cdmsv1
>>> import awkward_kaitai_cdmsv8
>>> array1 = awkward_kaitai_cdmsv1.load("file_v1.raw")
>>> array8 = awkward_kaitai_cdmsv8.load("file_v8.raw")
>>> awkward_kaitai_cdmsv1.load("file_v8.raw")
IOError: this does not appear to be a CDMS v1 file

One Python package with a plug-in system:

>>> import awkward_kaitai
>>> awkward_kaitai.path = ["/full/path/to/some/"]
>>> cdmsv1 = awkward_kaitai.register("awkward_kaitai_cdmsv1.so")
>>> cdmsv8 = awkward_kaitai.register("awkward_kaitai_cdmsv8.so")
>>> array1 = cdmsv1.load("file_v1.raw")
>>> array8 = cdmsv8.load("file_v8.raw")

wrapped up in

>>> import cdms.awkward
>>> array1 = cdms.awkward.load("file_v1.raw")   # dispatches to the right one based
>>> array8 = cdms.awkward.load("file_v8.raw")   # on magic bytes or file extension

With one Python package and a plug-in system, the Python package could be written in pure Python (no platform-dependence or Python version-dependence) and the plug-ins could be written without any pybind11 (no Python version-dependence). These C functions would be called by [ctypes](https://docs.python.org/3/library/ctypes.html):

extern "C" {
    struct Result {
        LayoutBuilder* builder;
        const char* error_message;
    };

    Result fill(const char* filename);
    const char* form(const LayoutBuilder* builder);
    int64_t length(const LayoutBuilder* builder);
    int64_t num_buffers(const LayoutBuilder* builder);
    const char* buffer_name(const LayoutBuilder* builder, int64_t index);
    int64_t buffer_size(const LayoutBuilder* builder, int64_t index);
    void copy_into(const LayoutBuilder* builder, int64_t index, void* buffer);
    void deallocate(const LayoutBuilder* builder);
}

Python drives the process:

1. Python calls `fill` and holds onto the (opaque) `LayoutBuilder*`.
2. Calls `form` to get the Form.
3. Calls `length` to get the length.
4. Calls `num_buffers` to get the number of buffers to allocate.
5. For each buffer, Python calls `buffer_name` and `buffer_size` to fill a Python dict named `containers`.
6. Python allocates all the buffers with NumPy or whatever.
7. For each buffer, Python calls `copy_into` to ask the C code to fill it.
8. Python can now `deallocate` the `LayoutBuilder*`. (Should always do this in a `try`-`finally` block.)
9. Python calls `ak.from_buffers(form, length, containers)` to get the Awkward Array.

Examples to use as reference:

* https://github.com/scikit-hep/awkward/blob/main/awkward-cpp/src/awkward_cpp/libawkward.py
* https://github.com/scikit-hep/awkward/blob/a6e426ed8943233b97ee17ca95e51c6d42cb6a54/dev/generate-kernel-signatures.py#L262-L268

class Result(ctypes.Structure):
    _fields_ = [
        ("builder", ctypes.c_void_p),
        ("str", ctypes.c_char_p),
    ]

fill = lib.fill
fill.argtypes = [ctypes.c_char_p]
fill.restypes = Result

Meeting Minutes October 3, 2023 - Kaitai-Awkward
===

**Manasvi:** working on modifying the end-to-end connection. Issues:

* On the Scala side, something in the PIXIE file, file header and file footer have common names for fields. What naming convention should we take so that they don't clash?
* Jim: to make the C++ code as debuggable as possible, you'd want to distinguish between "header/num_trace_blks" and "footer/num_trace_blks" by encoding the full path in the C++ name.
* You can't use `/` in a C++ name, and if you use something like `_`, then there could be unintended conflicts.
* You can use an [encoding like this](https://github.com/scikit-hep/uproot5/blob/0096168b7a4f2d7d66ee73e67310364a9f6d5320/src/uproot/model.py#L171-L229)
* or a simple rule: (a) if a name has a single underscore in it, make it two underscores (b) use a single underscore as the path separator and (c) put prefixes/suffixes on every word. For example, `header/num_trace_blks` becomes `AheaderZ_Anum__trace__blksZ` (with `A` and `Z` as the prefix and suffix).
* Medium alternative: have a counter for each name ensure the uniqueness: `header/num_trace_blks` becomes `num_trace_blks_0` and `footer/num_trace_blks` becomes `num_trace_blks_1`. Which is which would require you to look back at the original KSY—that's the downside.
* Bad alternative: make a new hash for each one: `ABBACDBAD`. That would make everything unique, but not easy to debug.
* The end-to-end workflow is already looking good! Jim only had comments about names.
* How to get the path to Awkward header-only code? Jim: maybe https://github.com/scikit-hep/awkward/pull/2280? No, Manasvi is already using that. Could ask Henry Schreiner and/or Shahzad Muzzafar.
* An error:

ImportError: /home/manasvigoyal/miniconda3/lib/python3.11/site-packages/awkward_kaitai.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN6kaitai7kstreamC1EPSi

Demanges to: `kaitai::kstream::kstream(std::basic_istream<char, std::char_traits<char> >*)` (probably not an issue with that one symbol; it's the runtime code that's not linked into the final Python module).

You'll want to statically link the Kaitai code into the Python extension module, so that the Python extension module is portable.

For the PyHEP talk: install Jupyter in the same Conda environment with all of the compilers (`cxx-compiler`, `cmake`, `make`, ...).

**Amy:**
Amy Roberts, [e-mail]

PONDD NSF Award Number [OAC-2104003](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2104003).

> Support for this work was provided by NSF cooperative agreements OAC-1836650 and PHY-2323298 (IRIS-HEP) and grants OAC-2104003 (PONDD) and OAC-2103945 (Awkward Array).

PyHEP: https://indico.cern.ch/event/1252095/timetable/#20231009.detailed

Meeting Minutes September 5, 2023 - Kaitai-Awkward
===

**Amy:** not much for updates, still working on the grant side of things. The semester hasn't settled down yet. In the next couple of weeks, will start on C++ tests.

**Manasvi:** adding strings in everything on the Scala side. Will need option-types. Started working on the CMake and modifying pybind11 code.

Last week at CERN, laptop stopped working. New laptop, installing things, extracted work from old laptop and getting back to that point.

Awkward-Kaitai test repository: moving things over to the new one so that the old repo can be removed. From https://github.com/ManasviGoyal/awkward_kaitai_tests to https://github.com/ManasviGoyal/kaitai_awkward_runtime.

Target for PyHEP: end-to-end demonstration using the pixie4e.ksy file.

**Manasvi needs Amy to provide a KSY without switch statements: scdms.ksy and midas.ksy.**

Meeting Minutes August 22, 2023 - Kaitai-Awkward
===

Manasvi generated a `kaitai_awkward_runtime` from the cookiecutter: https://github.com/ManasviGoyal/kaitai_awkward_runtime

Trouble running kaitai-struct-compiler with a KSY file that Amy sent (couldn't find it?).

Status on generating C++ for all variants of KSY: made temporary structures, need to generate C++ code in all the right places.

Meeting Minutes August 15, 2023 - Kaitai-Awkward
===

Manasvi: you can get all the information contained in the ksy file and _then_ create the code. Have all the types and all the field names in a mutable map - a string mapped to a list of strings. Kaitai provides a list of attribute specifications (name, type, etc.). -Awesome- Can iterate over the structure to generate the code.

Scope: right now we're concerned with records and lists. Are currently excluding option types and union types.

Cookiecutter sets up
* pybind11 C++ file
* CMakeLists: cross-platform C++ compilation
* pyproject.toml: installing as a Python package
* tests directory for pytest
* pre-commit for code linting
* GitHub Actions to run pytest and pre-commit on GitHub


cookiecutter gh:scientific-python/cookie  # choose scikit-build-core
# answer all the questions
git init
git add everything
git commit
# create the repo in GitHub and add it as a remote
git remote add ...

pip install .   # to compile and install in your environment
pip uninstall NAME  # to uninstall it from your environment

pytest tests  # manually run the tests, only works if installed
pre-commit run -a  # find out if GitHub is going to accept your commit

python
>>> import NAME

(The `NAME` comes from the KSY file.)

What pieces are needed to write the test (creates non-empty JSON) in python?
- use cookiecutter with scikit-build! for C++-pybind11 project creation
- need a fork of Kaitai-Struct compiler on a working branch
- need it to generate correct output for at least one .ksy file with no print statements
- a runtime (starts as a forked version of the C++ runtime), which needs to include pybind11 to make a python extension.
- We want data pulled into python to use the garbage collector without issue (python needs to "own" the data)
- Goal is to (in python) open a raw file and pull in the data AND own the data

- @Jim can PyBind11 accept file-*like* objects, or just something that goes to a posix file?
- One option could be to have the input to python be a string and then hand it to C++ to open and close the file. This is fragile (e.g. can't use http reference to file).
- We can instead hand C++ a more general file-like object. It needs to have a read and seek method.
- Not sure what the Kaitai C++ runtime needs. If it needs a C-style handle to a file then there's no way to hand a more general file-like object -> need to find out if Kaitai needs a way to read bytes, or if it also needs a seek
- Can create (with much pain) a FTNO (file handle) to a virtual file, see e.g. https://pypi.org/project/thingking/, https://man7.org/linux/man-pages/man3/fopencookie.3.html, https://gist.github.com/matthewturk/4285fd21d3ca6bb64472ce87d4fedb15. Matt: this is really a pain to get working across platforms/OS versions
- Motivation for file-like objects: remote files, [fsspec](https://filesystem-spec.readthedocs.io/en/latest/).

Meeting Minutes August 8, 2023 - Kaitai-Awkward
===

Amy and Jim discussed ideas for tests. Jim suggests: the code makes non-empty JSON so that as fields change the test doesn't break! RapidJSON is a nice C++ library. But we'd like Manasvi's input on whether this is necessary first :0

Question for Manasvi: is it helpful to have this test now, or can it wait to be a python module?

Most of Manasvi's work has been modifying scala code that she'll need to merge back in.

Kaitai also requires runtime libraries and we'll need to set that repository up.
- Jim: May want to use nanobind rather than pybind11 (but on second thought no, just to keep things simple)

Some links:
- [Modern CMake](https://cliutils.gitlab.io/modern-cmake/) (Henry's book)
- [Scikit-Build](https://cliutils.gitlab.io/modern-cmake/) (Henry's project)
- [scientific-python/cookie](https://github.com/scientific-python/cookie) to make repos using `cookiecutter gh:scientific-python/cookie`
- [PyBind11](https://pybind11.readthedocs.io/en/stable/) for binding Python with C++
- ~~[nanobind](https://github.com/wjakob/nanobind), a more minimal/faster version of PyBind11~~
- [RapidJSON](https://rapidjson.org/) is a good JSON library for C++, and Awkward binds it in CMake with [these lines](https://github.com/scikit-hep/awkward/blob/f1f93fcba330dc0fbcbd0a34f7f52d7006846744/awkward-cpp/CMakeLists.txt#L55-L56). Only for testing in C++. If we get to the point of a running Python module, then it will be easier to test JSON output in Python.
- Similar to what we did at the [tac-hep-coding-jam](https://github.com/ckoraka/tac-hep-coding-jam), which started with a cookiecutter template and the students added a C++ extension in Python. (Link to [Indico](https://indico.cern.ch/event/1293313/timetable/).) (Link to https://tac-hep.org/)

Meeting Minutes August 1, 2023 - Kaitai-Awkward
===

Jim asked Manasvi what has happened in the past 3 weeks.

* Has generated template instantiation.
* Working on generating runtime code.

Recommendation:

* Instead of writing the whole template instantiation in one place, do it in parts in each specific class as data members (private) which can be accessed by accessor functions. Initialy modify the C++ code to see where to place the members.
* Define the builder such that the we can refer to the type for the kind of builder an id has at each level of the nested structure.
* Do it in a separate fork instead of modifying the current code.

Meeting Minutes July 25, 2023 - Kaitai-Awkward
===

**Questions about Kaitai**
- Why is there a `raw_file_header` in the code for pixie4e.ksy and why is it a string?? We think this is related to the `size` specifier. The documentation states that specifying a `size` in a user-defined type always creates a substream. We looked at the generated code and it looks like `raw_file_header` (a string) is what's needed to create `raw_file_header_io` (a kstream) which is what's used to fill the structure `file_header`.

**Amy**
- Will generate the C++ code for pixie4e.ksy and write C++ code that creates a histogram and prints it out using data from https://github.com/det-lab/xia
- Side note, Amy is having trouble installing Kaitai's dependencies so Manasvi has posted the generated C++ source code. Together with https://github.com/kaitai-io/kaitai_struct_cpp_stl_runtime.git I think I can work without installing the jre

**Manasvi**
- This is the C++ string generated for the builder definition of `lists.ksy` example:

RecordBuilder<
RecordField<Field0::lists, RecordBuilder<
  RecordField<Field1::sample_block, RecordBuilder<
    RecordField<Field2::num_sample, NumpyBuilder<uint32_t>>,
    RecordField<Field2::sample, ListOffsetBuilder<int64_t, NumpyBuilder<double>>>
  >>,
  RecordField<Field1::sample_block2, RecordBuilder<
    RecordField<Field3::block, RecordBuilder<
      RecordField<Field4::num_example, NumpyBuilder<uint32_t>>,
      RecordField<Field4::example, ListOffsetBuilder<int64_t, RecordBuilder<
        RecordField<Field5::sample1, NumpyBuilder<double>>
    >>>
   >>
  >>
 >>
> builder;

- The structure is correctly generated for all the example cases that I had created but I need to deal with some of the exceptions (like the `size` one) and edge cases in the larger .ksy files in the dataReader repository (Amy's histogram code will help in checking it).

Meeting Minutes July 18, 2023 - Kaitai-Awkward
===

**Manasvi**
- Changed the approach to separate recursive functions which uses the path to create the different levels of the nested structure and now there is no need for the RecordBuilder contents to have `Option[LayoutBuilder]` type.
- The structure is correct to certain level (even for the `RecordBuilder`), need to create more examples .ksy files to test it and fix accordingly.

**Amy**
- The ksy file and the data that goes with it are at https://github.com/det-lab/dataReaderWriter/tree/master
- Continuing to work on Manasvi's payments; she is getting paid now so this is all internal stuff.

Meeting Minutes July 11, 2023 - Kaitai-Awkward
===

**Manasvi**
- Tried to make `AwkwardCompiler` as a subclass of `CPPCompiler` but due to the constraints of the codebase regarding the need to send the parameters of `AwkwardCompiler` to `LanguageCompiler`, it will have to copy code from `CPPCompiler` instead.
- Still need to fix a few things to generate the correct structure (esp. for RecordBuilder)
- Added functions to add C++ strings for the different builders (might need to be modified a bit once the structure is fixed).

Meeting Minutes July 4, 2023 - Kaitai-Awkward
===

**Manasvi**

Building case classes and using `match case` statements instead of methods to generate C++ strings. So far, it's looking good.

The order of execution of Scala code is such that `RecordBuilder` needs to have mutable fields and contents: two lists of type `ListBuffer` (which has an `append` method). They don't have to be `Option` (which is like a list that can have only 0 or 1 items; `ListBuffer[Option[X]]` is like a nested list of lists).

Or not: Jim's wrong. There's a point in the code (`attrParse2`) in which you start with an empty `RecordBuilder` and are informed about

* all field names (strings)
* only primitive types (`NumpyBuilder`)
* and _not_ the distinction between sublists and subrecords

Elsewhere (multiple functions), you're informed about the distinction between sublists and subrecords, but not primitive types.

So yes: you either need

* `fields: ListBuffer[String], contents: ListBuffer[Option[LayoutBuilder]]` in order to use `None` as a placeholder for the types you can't _yet_ distinguish, or
* `fields: ListBuffer[String], contents: MutableMap[String, LayoutBuilder]` in order to use the `fields` as the one and only place where the order of the fields is maintained and the filled key-value pairs in the `MutableMap` are the types you know about, while the unfilled key-value pairs are the ones you don't know about yet.

PLUR

// maybe UpcastType is kaitai::struct_t
std::vector<UpcastType*> given_list;
UpcastType* given_data = given_list[i];

DowncastType1* downcast1;
DowncastType2* downcast2;
if (downcast1 = std::dynamic_cast<DowncastType1*>(given_data) != nullptr) {
  // handle downcast1
}
else if (downcast2 = dynamic_cast<DowncastType2*>(given_data) != nullptr) {
  // handle downcast2
}
else {
  // error!
}

__vtable

**Amy**

No news. Continuing to work on Manasvi's payment. Will write a check soon.

Meeting Minutes June 27, 2023 - Kaitai-Awkward
===

ksy for CDMS data: https://github.com/det-lab/dataReaderWriter/blob/master/kaitai/ksy/scdms_v8.ksy

spreadsheet drawing of CDMS data: https://docs.google.com/spreadsheets/d/18JFdAiGk3vv6xY_H2blqwkwoyIS1nLTkMQ6v6KmC4Ko/edit?pli=1#gid=1275782170

Want to use the public interface and stay away from the private members.

**Manasvi**

* Started on a Graph (actually tree) structure

Map(kaitai::kstruct* -> List(lists))
Map(kaitai::kstruct* -> List(lists), lists_t* -> List(sample_block))
Map(kaitai::kstruct* -> List(lists), lists_t* -> List(sample_block2))
Map(kaitai::kstruct* -> List(lists), lists_t* -> List(sample_block2), lists_t::sample_block2_t* -> List(block))

import scala.collection.mutable.ListBuffer

trait LayoutBuilder {}

case class NumpyBuilder(
    dtype: String
) extends LayoutBuilder {}

case class ListOffsetBuilder(
    offsets: String,
    var Option[content]: LayoutBuilder
) extends LayoutBuilder {}

case class RecordBuilder(
    fields: ListBuffer[String],
    contents: ListBuffer[LayoutBuilder]
) extends LayoutBuilder {}

val listoffset = ListOffsetBuilder("i32", None)
listoffset.content = Some(NumpyBuilder("float64"))

val unknowntype : LayoutBuilder = listoffset

unknowntype match {
  case NumpyBuilder(dtype) => println("Numpy " + dtype)
  case ListOffsetBuilder(offsets, None) => println("incomplete ListOffset")
  case ListOffsetBuilder(offsets, Some(content)) => println("ListOffset")
}

val a = ListBuffer[String]()
a.append("one")
a.append("two")
a.append("three")

// similarly for ListBuffer[LayoutBuilder]

// you'll probably want to make methods on these classes
// (between the curly brackets), maybe instead of the match-case.

// The "var", "Option", "ListBuffer" mutability is only if you need it.

**Amy**

* got approval from the NSF to pay Manasvi, still a few more steps but this should happen soon!
* am working on setting up a stipend for a student to work on

https://github.com/det-lab/dataReaderWriter/blob/master/kaitai/ksy/scdms_v8.ksy

https://docs.google.com/spreadsheets/d/18JFdAiGk3vv6xY_H2blqwkwoyIS1nLTkMQ6v6KmC4Ko/edit?pli=1#gid=1275782170

Meeting Minutes June 20, 2023 - Kaitai-Awkward
===

**Manasvi:**
Working on the Graph structure for the scala code to store the ID, names and type in a map and iterating over it to the get the builder definition and other strings for C++ code.

Meeting Minutes June 14, 2023 - Kaitai-Awkward
===

**Manasvi:**

3 options: going for the data structure option (Jim: I agree).

Agreed to structure and name the Scala classes one-to-one with LayoutBuilder _templates_.

Could be useful (asking for Amy to provide this): _partial_ KSYs that parse CDMS data up to a point and stop, but are simpler.

Meeting Minutes June 6, 2023 - Kaitai-Awkward
===

**Manasvi:**

Going to CERN on June 12.

Questions about how to generate the code because the order in which methods are called in the Scala are not the most conveninet for generating the C++ we want to generate. It seems to be walking from root to node of the structure.

From Jim: one thing is that the KSY we provided doesn't look like `ListOffsetBuilder(NumpyBuilder(double))`, it looks like

RecordBuilder(
    Field("num_sample", NumpyBuilder(uint32)),
    Field("sample", ListOffsetBuilder(NumpyBuilder(double)))
)

With this KSY, you're already forced to consider RecordBuilders. Maybe we can't write a KSY that is more complex than just NumpyBuilder and also doesn't have RecordBuilder. It might help to consider a raw file and KSY with a record of all simple fields (double, uint32, etc.).

Another thing: the order in which the outer Scala code chooses to walk over the data structure could probably be written into _enough_ streams of strings that go into C++. But that might be too complicated because every stream is constrained to have to be filled from left to right in C++ syntax. That might be too much of a constraint.

Another way around it (instead of many streams): make a temporary data structure in Scala that gets filled up with everything you need to know in order to generate the C++. Hopefully, there will be some function call that tells you the type declaration is complete, and at that time you can conver the data structure into the appropriate string with all information available.

A third way is to learn how Kaitai's own internal representation of KSY works and run your own iteration, in the order you need. But I think that could be too difficult.

**Amy:**

Working on paying Manasvi!

KSY is functional, no new progress since last week; next step is to find another dataset to verify that it's general.

**Yana:**

Pair programming when Manasvi arrives at CERN?

Meeting Minutes May 30, 2023 - Kaitai-Awkward
===

**Manasvi:**
- Finished the NumpyBuilder case, moving on to ListOffsetBuilder case.
- Now have a test file and KSY to develop with.

meta:
  id: test_arrays
  file-extension: test_arrays
  endian: le
  
seq:
  - id: sample_blk
    type: sample_block
    repeat: eos
    
types:
  sample_block:
    seq:
      - id: num_samples
        type: u4
      - id: sample
        type: f8
        repeat: expr
        repeat-expr: num_samples

To get a raw data file:

import random
import numpy as np

# array of 1000 lists of variable length
data = []
for _ in range(1000):
    data.append([random.gauss(0, 1) for _ in range(random.randint(0, 10))])

# accumulate binary output
binary_output = []
for some_list in data:
    # output size of the list in 32-bit unsigned (u4)
    binary_output.append(np.array(len(some_list), dtype="u4").tobytes())
    # for each element of the list...
    for some_value in some_list:
        # output the element as a 64-bit floating point (f8)
        binary_output.append(np.array(some_value, dtype="f8").tobytes())

# concatenate to get a single binary string
output = b"".join(binary_output)

# write it to a file
open("array_of_lists_of_floats.raw", "wb").write(output)

**Amy:**
- Not exactly the desired tree-structure, but all the data have been read in, and it can be restructured (zero-copy) by Awkward.
- Instead of using `optional` Kaitai uses null pointers for optional fields. Can see this by searching for "`= 0`" in the code. Can non-instance types (float, uint, etc.) be optional? If null pointers are used for optional data, what do they do for data that isn't pointed to? Can we ever know if a record/class instance is NOT optional (must always be present)?

> I'm working on a new backed with
> jpivarski (Jim Pivarski)
> and
> ManasviGoyal (ManasviGoyal)
> and I'm trying to write a data description that avoids dynamic casts in C++. I think that the "cases" data description method uses dynamic casts and am wondering if the conditional also uses dynamic casts?
>
> generalmimon (Petr Pučil)
> Frankly, I don't even know what you mean by "dynamic cast". If I take it literally, I'm quite sure that the generated C++ code never uses https://en.cppreference.com/w/cpp/language/dynamic_cast. Sometimes it uses https://en.cppreference.com/w/cpp/language/static_cast, but only in specific circumstances.
>
> The runtime library (kaitaistream.cpp) relies on https://en.cppreference.com/w/cpp/language/reinterpret_cast quite extensively, I don't know if that's a problem for you (I guess it shouldn't be, because according to the linked cppreference.com page: "It is purely a compile-time directive which instructs the compiler to treat expression as if it had the type new-type.").
>
> Back to https://en.cppreference.com/w/cpp/language/static_cast - it's generated when parsing enum fields (to cast the raw integer to the enum type), for the to_i method on floats (https://doc.kaitai.io/user_guide.html#float-methods) and of course when the specification explicitly requests type casting by using the .as<> operator (https://doc.kaitai.io/user_guide.html#typecast).
>
> By default, type switching itself does not use any casts (if autoRead is enabled, which is the default setting unless --no-auto-read or other option that implies it, e.g. --debug, is specified in the command line). It only does if you have disabled autoRead and the switch uses user-defined types, because in that case the generated code has to call _read() explicitly after instantiating the object and in order to convince the C++ compiler that _read() exists, cast must be used.
>
> Of course, one thing is that the generated code itself doesn't use casts in the type switching implementation in autoRead mode, another thing is that you typically have to use the casts in your own application code when processing the parsing results from the KS-generated code. If you want to avoid that as well, yes, the multiple-if approach avoids that, and it used to be the only approach that was available before the switch was introduced (https://github.com/kaitai-io/kaitai_struct/issues/10).
>
> It's been proposed to provide an alternative interface for type switching (https://github.com/kaitai-io/kaitai_struct/issues/493) that could eliminate the need for casts, but that's still just an idea.

Meeting Minutes May 23, 2023 - Kaitai-Awkward
===

**Amy:**
- Still need to find out if the conditionals in Kaitai avoid unions.
- Have completed the description of the CDMS data block and (I think) it involves only records and lists. Will confirm.
- The data structure has other data blocks besides the CDMS records and I'm currently working on those - can implement them with Kaitai cases (which I think involves unions) or Kaitai conditionals (trying to use conditionals)

**Manasvi:**
- Almost done with the Numpy case. Just need to fix some errors in the CMake.
- Will start working on the list/records case next.

Meeting Minutes May 16, 2023 - Katai-Awkward
===

**Amy:**
Do we add a conditional for blocks that can have zero elements like the SDU block? It depends on how it's implemented in Kaitai. We want to avoid unions.
- Do they use C++ std::optional
- Or is it an empty list when paired with repeat-expr?
- Or do they use unions with C++ dynamic casts?

**Manasvi:**
Asked Yana about something; it's fixed now.
How to identify which LayoutBuilder to use, given a KSY?

Goal: get AwkwardCompiler.scala to generate a LayoutBuilder in C++ for a simple NumPy case (no records, no lists, etc.).

**Yana:** will miss the next meeting; has updates on LayoutBuilder-Numba.

Meeting Minutes May 9, 2023 - Katai-Awkward
===

**Amy:** Got the repeated structure working, am near the end of the format! Should have this done by next week.

Meeting Minutes May 2, 2023 - Katai-Awkward
===

**Manasvi:** explored the parts of code where the RecordBuilder code can be added and how records can be filled using the fake.ksy file. Had some doubts which were clarified by Ianna.

**Amy:** working on the repeated structure

**Note**: there will be no meeting next week - due to the final exams

Meeting Minutes Apr 25, 2023 - Katai-Awkward
===

**Manasvi:** fully explored the animal KSY and where each part of the Scala code generates C++ code. Also noticed that we'll need a subclass of `ClassCompiler`. (It can be done in a more Scalarific way using `traits`!)

**Amy:** made progress; into the CDMS data now. Doing bit-shifts (have examples of that from before), will be getting into repeated structures next. Read out full header!

**Jim:** made a `fake.ksy` and `fake.raw` to test only class-generation so that Manasvi can focus on only the `Record` LayoutBuilder. (Shared in Slack.)

open("fake.raw", "wb").write(
    np.array(
        [(1, 2, 3), (10, 20, 30), (11, 22, 33)] * 1000,
        dtype=[("x", "u4"), ("y", "u4"), ("z", "u4")]
    ).tobytes()
)

Meeting Minutes Apr 18, 2023 - Katai-Awkward
===

**Amy:** working on understanding the "ODB" (XML record of the Online Database) part of the MIDAS data format, which doesn't seem documented on the MIDAS data format page, http://lmu.web.psi.ch/docu/manuals/bulk_manuals/software/midas195/html/AppendixA.html

**Manasvi:** Working on the CMake file. Figuring out how to use the code from the C++ runtime repo, rather than a copy in the Awkward repo. For next time: prepare a mini/informal presentation to us about which parts of the Scala code produce which parts of the C++.

Meeting Minutes Apr 4, 2023 - Katai-Awkward
===

**Amy:** no updates on the format, but likely to pick up again where left off. Instead of starting with a complete version and changing pieces, will start with a blank KSY and will merge things in.

**Manasvi:** still figuring out CMake, unclear about a few things. Planning to reach out to CMake experts: now have clear questions to ask. Specifically, "How to use the Awkward submodule?" #awkward-uproot and @angus. Also, about using git submodules in CMake.

**Yana:** planning to reproduce the setup (to be available for questions).

About CMake, Yana said,

$ cmake -B build -S header-only -DCMAKE_RUNTIME_OUTPUT_DIRECTORY=bin -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON
$ cmake --build build

**Jim:** What using should look like to users, at the end of the project:

1. User goes and gets kaitai_struct_compiler from the ordinary place (i.e. all of our Scala code is upstreamed).
2. ~~User goes and gets the Awkward runtime repo (`git clone --recursive`).~~
3. User already has a KSY file.
4. User runs `kaitai_struct_compiler -target awkward my_file.ksy` and gets a directory containing:
* CMakeLists.txt
* all the C++ code
* already wrapped up in pybind11 and Scikit-Build
* goes and gets what it needs from the Awkward runtime repo?
5. User runs `pip install .` in that new directory, gets a Python extension module.
6. In Python, user calls `import new_lib; awkward_array = new_lib.read("filename")`.

Meeting Minutes Mar 28, 2023 - Katai-Awkward
===

**Manasvi:** set up instructions: https://github.com/ManasviGoyal/awkward_kaitai_tests

https://cliutils.gitlab.io/modern-cmake/

Two areas to develop next:

1. an all-in-one build procedure, simplifying what's on awkward_kaitai_tests (that you'll eventually need anyway)
2. in the Scala: start instantiating LayoutBuilder templates

First subgoal: a short procedure that builds a usable Python module that can be imported into Python, but doesn't actually build any arrays—it just prints

BEGIN
str_len 3
age 5
weight 12
str_len 3
age 3
weight 43
str_len 6
age 10
weight 5
END

Short procedure is:

1. `git clone --recursive https://github.com/ManasviGoyal/awkward_kaitai_tests`
2. cmake (two steps: cmake configure & cmake build)
3. `python -c 'import new_module; new_module.do_it()'`

Want cmake to know how to run `sbt package`, pull in pybind11 and Awkward LayoutBuilder headers and cpp_stl code, compile it into `new_module`.

Meeting Minutes Mar 21, 2023 - Katai-Awkward
===

**Amy:** a bit of progress, moving slowly.

Broken the parsing; would be better to have the parsing working and then add features.

Maybe from beginning to end, sequentially/iteratively through the file.

**Manasvi:** got the backend working, starting from the example (CppCompiler.scala), can now use "awkward" as a language. Only had to change a few files.

Saw a demo.

Talked about next steps: should AwkwardCompiler.scala sub-class CppCompiler.scala or not? If so, what about the cpp_stl runtime? Manasvi will ask about this on Gitter.

Manasvi will also set up instructions so that we can reproduce her environment, to be in a better position to answer questions about her work, if she runs into something.

Meeting Minutes Feb 28, 2023 - Katai-Awkward
===

Follow-up from last week from Manasvi:

* Got the full workflow of kaitai-struct-compiler → C++ → reading data working (for the "animal" dataset).
* no KSY for new SDSS data yet
* most recent SCDMS format: https://github.com/det-lab/dataReaderWriter/tree/master/kaitai/ksy

For next week:

* Manasvi: in your fork of kaitai-struct-compiler, create a new "backend" using AwkwardCompiler.scala, which is a subclass of CppCompiler.scala.
* Amy: get the most recent SCDMS format to read the _small_ data file in the web browser: https://ide.kaitai.io/
* In writing KSY files, try to avoid branching/switching, because it leads to union-types in Awkward Array (the data analysis).

<!--
Note: if you end up with any JSON that should be KSY formatted, this is how to convert (in Python):

>>> import json
>>> import yaml  # package is "pyyaml"
>>> some_json_text = '''
{"first": [1, 2, 3], "second": {"x": 1, "y": 2.2, "z": "THREE"}}
'''
>>> print(yaml.dump(json.loads(some_json_text)))
first:
- 1
- 2
- 3
second:
  x: 1
  y: 2.2
  z: THREE

-->

Meeting Minutes Feb 21, 2023 - Katai-Awkward
===

End goal, what a successful conclusion looks like:

* Software: contribute a new backend to Katai, **KSY → compiled Python module**, and that module takes **data → Awkward Arrays**.
- this includes coordination with Katai authors for PR review and hopefully acceptance
- software scope?
- [Katai user manual](https://doc.kaitai.io/user_guide.html)
- copy [CppCompiler.scala](https://github.com/kaitai-io/kaitai_struct_compiler/blob/master/shared/src/main/scala/io/kaitai/struct/languages/CppCompiler.scala) with a new name
- modify it to emit LayoutBuilder-filling instructions
* Conference (talk or poster, maybe also proceedings)
- where? IEEE? FAIR data workshop/conference?
* Standalone paper
- CSBS: https://www.springer.com/journal/41781
- ~~JOSS~~

Resources

* https://osf.io/2sner/, a high-level overview of Kaitai
* CDSM datasets and data formats, https://osf.io/ybw7r/

Next week:

* M: Get the kB-sized file into the web IDE
* M: Get a katai-struct-compiler → C++ workbench set up
* A: Get the KSY definition working for the new format
* A: Start on the paper (intro, background)
* What conference? What journal?

Meeting Minutes Jan 24, 2023 - Katai-Awkward
===

###### tags: `PONDD` `Meeting` `Kaitai` `Awkward`

:::info
- **Location:** https://ucdenver.zoom.us/j/94702468227
- **Date:** March 3, 2022, 10 AM Mountain
- **Participants:**
- Amy Roberts (AR)
- Jim Pivarski (JP)
- Yana Osborne (YO)
- Manasvi Goyal (MG)
- **Meeting Goal(s)**

:::

## Summary
<!-- Please fill me in after the meeting -->

## Notes
<!-- Other important details discussed during the meeting can be entered here. -->
- Manasvi will go to CERN June 1 through August 31
- India Feb 6 through end of May
- Amy will find out how to handle finances for CERN and stipend
- Monthly!
- Minimum wage in switzerland is $25/hour
- 7200 USD for 12 weeks in switzerland is IRIS-HEP rate
- follow-up from Peter: for a student physically at CERN, IRIS-HEP is actually paying 3704 CHF per month (more)
- needs insurance while at CERN (what Aryan did was buy international insurance in India)
- 3600 USD for 12 weeks in india
- CERN summer students suggest *90 CHF* per day
- https://jobs.smartrecruiters.com/CERN/743999862723511-cern-openlab-summer-student-programme-2023
- Also need to include travel, IRIS-HEP directly pays for travel. Amy will find out how to handle travel at CU Denver. Roughly 1000 USD from Delhi, round trip.
- Amy will talk to Matt Turk about doing stipend reimbursements from his institution, maybe they will be more responsive!
- Estimated totals:
- 4800 USD for 4 months in India at 1200 USD per month
- 14407.2 USD for 3 months at CERN at 3704 CHF per month
- 1500 USD for travel
- 20707.02 total
- Ianna will find out what needs to happen from CERN to get a letter for the visa process
- Home institute: Princeton? The home institution doesn't need to pay!
- Do we ask CMS for a letter? Technically this doesn't have anything to do with CMS but that may not matter.
- Needed to get started
- SuperCDMS membership and account access: https://confluence.slac.stanford.edu/display/CDMSPUB/Home, want confluence access
- https://osf.io/ybw7r/ access
- data sets! Amy will put the data on OSF and ask the collaboration if we can make the dataset public. If the answer is no, then we could generate data!
- data format descriptions!
- Jim's fork of [kaitai_struct_compiler](https://github.com/jpivarski/kaitai_struct_compiler) and experiments with [awkward-kaitai](https://github.com/jpivarski/awkward-kaitai), which has copies of a few _small_ datasets and KSY files.
- https://gitter.im/kaitai_struct/Lobby
- What's the end goal? Will Manasvi go to a conference/write a conference proceedings?
- https://www.scipy2023.scipy.org/ this would entail writing a paper *before* going to the conference. Timescale is Feb 22, too early!
- CSBS: Computing and Software for Big Science article? https://www.springer.com/journal/41781
- https://www.software.ac.uk/which-journals-should-i-publish-my-software
- https://ieeebigdataservice.com/call-for-papers/
- FAIR data workshop/conference?
- Standing meeting time is 10:30 AM Mountain on Tuesdays