---
# System prepended metadata

title: ADIOS2 checkpointing I/O
tags: [dolfinx, FEniCSx]

---

# Objectives:
1. Support checkpointing, i.e. recovery of program state from file for restart after program failure.
2. Support file output of common objects that can be used as input for a program.


# Case 1: checkpointing

## Requirements
1. Very fast
2. Bit-wise same restart?
3. Robust with respect to program failures, e.g. avoid file corruptions, flushing.

## Questions
1. If writing on *m* ranks, should we support read on *n* ranks? (if $m \ne n$ is it really checkpointing?)
2. Should checkpointing be portable across versions/platforms? 
3. Should output mirror internal data structures closely? This would support fast read/write. Could we serialise?
4. Can typechecking, etc, be skipped since checkpointing is for program restart?


# Case 2: input/output of objects

## Requirements

1. Fast
2. Write with *m* ranks, read on *n* ranks.
3. Portable across platforms and versions
4. Support type introspection
- (JSD): Probably possible using: https://adios2.readthedocs.io/en/latest/api_full/api_full.html?highlight=list%20attributes#_CPPv4N6adios22IO18AvailableVariablesEb. However, it is unclear how this should work with our templated functions (read_mesh/read_function is a templated function in C++).
5. Robust file handling

## Questions
1. Use VTK rather than DOLFINx cell ordering for portability across platforms?
- (JSD): We do not support arbitrary order Lagrange elements in 3D for VTK ordering
2. Support all element types, or support DG (using interpolation to re-create Functions)
- (JSD): File size grows rapidly in 3D if we have to use DG to represent higher order Nedelec spaces.


# Misc notes

Need to consider backward compatibility (future-proofing). 

Writing to file:
- store Mesh
= geometry, topology, cell types
(support for Mixed Topology in future?)
How is the mesh represented in memory across processes? and in file format?

For an actual checkpoint, we would need to store alot more than just a single function. We would need to store all dependencies of the form (constants, functions etc). It could be a series of functions in a series of forms (say using PETSc nest or a splitting scheme).

## Notes
- (IAB) https://github.com/FEniCS/dolfinx/pull/2618 is only about mesh reading and writing, maybe we should discuss the requirements of mesh reading/writing first?