Proposal Enzyme is an LLVM incubator project that is able to vectorize and differentiate (in the calculus sense) a given function. Computing gradients efficiently is necessary for various algorithms such as backpropagation, Bayesian inference, uncertainty quantification, and probabilistic programming. The reason why this work should be integrated into the rust compiler is that it has been shown both in theory and in practise (PyTorch 2.0, Jax/XLA, ...) that Automatic Differentiation (AD) of code is hard to do efficiently on unoptimized source code like Rust. Enzyme has shown a geometric mean speedup of 4.5x over mature pre-optimization AD tools. Enzyme has also been successfully used in combination with parallel paradigms like CUDA, ROCm, MPI and OpenMP. Enzyme also supports other languages, e.g., Julia, Fortran, compiled Python and C++. Handling LLVM-IR generated by such a variety of compilers does help to recognize and fix bugs early which otherwise would have been hard to discover using only Rust test cases. The latest proof-of-concept rust-enzyme has demonstrated that our approach is mature enough and is able to solve all fundamental issues of earlier iterations. We, therefore, propose to conditionally enable Enzyme for nightly releases to allow more Rust developers to test the current state and provide valuable feedback. We do intend to stabilize this work at some point because we believe that good AD and vectorization support is a fundamental building block of scientific computing and machine learning. Working around the lack of such a tool in our opinion has similar consequences as working around the availability of async, asm, or simd support for the corresponding groups. User written implementations would either lack performance and test coverage as explained above, need to re-implement various LLVM/Enzyme optimizations or transpile the user code from Rust into other languages that have AD support. As a consequence, we hope to support both existing projects and maintainers, as well as lowering the barrier for new contributors in this field. We do intend to write a full RFC in more than a year from now, once we had sufficient users provide feedback on the Rust frontend of our work.

tags: RUST_ML_MEETING Meeting Info Date: 20200903 Start time: 1600 ET Agenda Rust-RL: Short presentation

Using this as a list of all the things I would like to understand better or things for which I had to ask people to understand them better. Please feel free to assign someone, add more todos and mark others as done / not needed. Document the C-API.h file. It probably deserves it's own page just beside of https://enzyme.mit.edu/getting_started/CallingConvention/ on top of the in-code documentation. I've also mostly used this part of Enzyme, so that's the largest part of my list. When do I care about it at all (instead of using _enzyme...) (probably for a nicer integration into a new language?) Group those functions. Language frontend devs probably want to expose some (EnzymeCreateForward/Reverse/...) functions to their users. Other ones (EnzymeTypeTreeOnlyEq) shall probably be hidden from enzyme users, instead frontend devs might want to generate those trees on behalf of the users from compiler information. Some informal guidance would be nice. Document illegal / allowed parameter combinations or make the types more specific:

Setting the activity of a single input variable has the following effects: Active = DFT_OUT_DIFF ==> f32/f64 only. Add a second f32/f64 as input (linear factor) and add one as output. Duplicated = DFT_DUP_ARG ==> ref / pointer only. Add a second (mutable) type as input and don't change output. For forward mode only dup-arg or const, also for f32/f64. Then second as in rev-active. If width(n) > 1 add n duplicates (not a vector of len n). Constant ==> Don't change IO parameters Gradient = DFT_DUP_NONEED ==> Like duplicated, but it's UB to use the first of the two parameters after the function. Allowed output combinations: