Enzyme Documentation

# Enzyme Documentation Using this as a list of all the things I would like to understand better or things for which I had to ask people to understand them better. Please feel free to assign someone, add more todos and mark others as done / not needed. - Document the C-API.h file. It probably deserves it's own page just beside of https://enzyme.mit.edu/getting_started/CallingConvention/ on top of the in-code documentation. I've also mostly used this part of Enzyme, so that's the largest part of my list. 1) When do I care about it at all (instead of using __enzyme_...) (probably for a nicer integration into a new language?) 2) Group those functions. Language frontend devs probably want to expose some (EnzymeCreateForward/Reverse/...) functions to their users. Other ones (EnzymeTypeTreeOnlyEq) shall probably be hidden from enzyme users, instead frontend devs might want to generate those trees on behalf of the users from compiler information. Some informal guidance would be nice. 3) Document illegal / allowed parameter combinations or make the types more specific: - There are 4 CDerivativeModes (more fwd-modes in progress). I can pass each of them to each of the EnzymeCreateX functions, but almost none of this combinations makes sense or is allowed. - There are 4 activity options, but for reverse-mode I can only use two for the return type. We should document it (or better, enable dup_x) - Explain what the EnzymeAugmentedReturnPtr does (what should I pass as input, when should I use it, what input parameters will it expect?). Also, mention that using split-mode reverse AD requires calling two EnzymeCreateX functions. - Explain when AtomicAdd should be set (If the ToDiff function has internal parallelism, e.g. calls #pragma omp parallel? Or if the Function which we generate will be called multiple times in parallel?). - Explain when args are uncacheable (in reverse split mode ad, when there are updates between the forward and the reverse pass). It would be even nicer if new users couldn't even set it in those cases where it doesn't make sense. But that would require us to re-factor those two reverse functions, with probably other implications. 4) Explain how the numbers and types of input / output parameters of our generated function depend on: - The activity of input / output parameters (Mostly covered in CallingConvention, put that's a different location) - returnValue - dretUsed - The mode: Forward/Reverse/Reverse-split Depending on how my RFC goes, Rust users might need to write the function declaration and Enzyme will just add the definition. So end-users should understand this. 5-10 simple example probably allow a faster scimming than acurate docs, so we might want to add that too. 5) Related to 4), document why we have retType, dretUsed, returnValue. 6) I guess that most Rust/Swift/C++ devs don't know how to create an LLVMBuilderRef to integrate a shadow variable for a global showing up somewhere. Is there something we can do to help them with CustomShadowAlloc? At least for my Rust frontend I'm not really sure how I can use it to help Rust devs. A similar thing goes for vTable based issues. We probably shouldn't expect that all users read the paper or understand what to do based on this instruction: - If you want Enzyme to differentiate a function with a virtual C++ class as an argument, however, you need to pass in a modiﬁed virtual method table in the shadow that conforms with Enzyme’s calling convention. - Some questions which came up, which are not only due to the c-api: 1) Related to 4), explain that we don't directly implemented Forward/Reverse, but more general cases. Howe they differ, why that's usefull. At least on the Rust side there had been some confusion. 2) How can we get higher-order derivatives? (Recursively). Two examples would be nice to show how users can get their Hessian. - For the *Dev-Docs*: Document the infrastructure provided to add additional parallel programming paradigms - Why do we have extra BLAS/Lapack support (being able to differentiate them even if their source code isn't available / slightly faster compile time). Why do we have extra support for parallel libraries (correctness). Which other libraries do we support (those based on our supported libraries), which ones are not supported (rayon). - explicit BLAS support: I think the goal is support for all lv2/lv3 functions, but we aren't there yet. For which of the following do we have explicit support? https://www.netlib.org/blas/#_blas_routines - explicit LAPACK support: Iirc. we wanted to add explicit LAPACK support similar to the BLAS one. What is the state here? - OpenMP support: We do support #pragma omp parallel, we don't support task based parallelism, as far as I know. We should document this somewhere and give some more details. @wsmoses - MPI support: We do support the common operations and skipped a few of the less used ones. I think we have a overview somewhere on the github project page, we probably want to move it. @Ludger - ROCm / cuda support. The Cuda guide looks quite ok to me, maybe add a note about AMD? Also, why do you declare it as highly experimental, when will it break? - Rust documentation / integration. - Swift documentation / integration. - Collect RFCs. There will probably be more languages which require larger or smaller changes and therefore an RFC. There is one for C++WG21, one for Rust is in progress. At least certain parts (motivation) will be quite similar across them, so people might benefit from being able to easily compare / copy them. Along the lines, add an informal list of failed / successful attempts to integrate enzyme into languages. Might help when people are looking to integrate enzyme into their language. E.g. the c-abi broke a lot of things for the Rust side, so now we aim for a rustc integration. - Finish splitting up the getting started docs based on the languages. - The EnzymeTutorial is quite hidden I think. - There are some (probably) outdated issues (some with scary names): https://github.com/wsmoses/Enzyme/issues/84, https://github.com/wsmoses/Enzyme/issues/28 - We should really do something about https://enzyme.mit.edu/charter/ - LLVM-Incubator documentation. Afaik it's a new process, but it would probably help the Rust side to get some information on the process like next steps, goals, rough timelines, open questions. - Fill https://llvm.discourse.group/t/about-the-enzyme-category/4878 - Integration into build-tools like cmake. - Some inconsistency in the examples. In the calling-convention we don't declare functions as extern, in the compiler explorer and the using-enzyme section we do. We should unify that or add a note. - Not directly docs, but how about the enzyme summit? - Question from the rust side: We don't use (not even offer) most fast-float/math flags. When does Enzyme use them? https://github.com/wsmoses/Enzyme/blob/dd3587e54a7f77acf01fbff2aa9e18368e135980/enzyme/Enzyme/EnzymeLogic.cpp#L3365 - For contributors. Document, that branches should be made on the enzyme propper repo, creating PRs from forks will break CI. Kurze frage, was muss ich an `-Wl,` anhängen, damit er globals ohne shaddow akzeptiert?