AiiDA Team Meeting 2023-06-15

# AiiDA Team Meeting 2023-06-15 ###### tags: `team meetings` ###### time: 10am CET [TOC] ### Present * Chris * Marnik * Francisco * Giovanni * Miki * Xing * Edan * Kristjan ### Catch-up round *Max. 3 minutes each* Sebastiaan _(won't be able to join)_: * Straightforward PRs. Who can have a look?: * [`verdi process status`: Add `call_link_label` to stack entries](https://github.com/aiidateam/aiida-core/pull/6056) * [`QueryBuilder`: Deprecate `debug` argument and use logger](https://github.com/aiidateam/aiida-core/pull/6055) * [DevOps: Update requirement pylint~=2.17.4](https://github.com/aiidateam/aiida-core/pull/6054) [CJS review] * Origin of the [`RecursionError`](https://github.com/aiidateam/aiida-core/issues/4876) is now known. The way process functions are implemented, they need to reenter the event-loop when they are launched, which is explicitly forbidden by `asyncio`. It is necessary to reenter the loop to prevent users from having to write `async` code but can simply write and call normal Python functions. As a workaround, we use the `nest_asyncio` library which monkeypatches `asyncio` to remove the limitation of the event loop not being reentrant. The problem with this approach is that each process function execution adds frames to the stack, which don't get popped until the execution finishes. What we were seeing is many workchains being submitted that run a process function, and all process functions would be started before being finished, causing the recursion error to be hit. With the cause known, a number of workarounds have been proposed: 1. [Add configuration for recursion limit in workers](https://github.com/aiidateam/aiida-core/pull/6044): This allows users to manually configure a higher recursion limit that will be used by daemon workers. 2. [Engine: Dynamically update maximum stack size close to overflow](https://github.com/aiidateam/aiida-core/pull/6052): Same principle of increasing recursion limit but have the daemon worker do it automatically whenever it notices that it is coming close to the limit. Advantage is that it "fixes" the problem automatically, downside is that if the load is significant, the daemon worker might hit a real stackoverflow. 3. [Use `greenlet` library](https://github.com/sphuber/aiida-core/tree/fix/recursion-error-greenlet): Use the [`greenlet`](https://pypi.org/project/greenlet/) library to provide in-process multitasking. Solution sketched by Giovanni in the original issue. I have integrated in `aiida-core` and for the daemon test it seems to work, however, as pointed out by Giovanni in a Jupyter notebook (and maybe other interactive shells) `nest_asyncio` still needs to be enabled manually for it to work. I haven't yet been able to do a performance test of this solution. These are all workarounds that don't really addres the underlying problem or introduce even more complexity to get around it. Neverthelesss I would be tempted to maybe add 1 and 2 and release it. Although they do not solve the problem, they push it forward and will allow users to run a lot more before hitting a problem, without adding any additional complexity. But I am interested to hear your thoughts on how to proceed. - GP: merge and release 1+2 (including dynamic), but continue discussing and possibly merging also greenlets if there are no major issues we see? Marnik: * Coding week: * :calendar: Shifting week one earlier (3-7 July): possible for everyone? * [Working document on coding week topics/tasks](https://hackmd.io/Hg2nbnUmSfinIx4WagjJfw). Chris: * Updated aiida-firecrest to Sebs branch of aiida-core and latest firecrest/pyfirecrest: https://github.com/aiidateam/aiida-firecrest/pull/4 * Looks to be working, but just working more on test infrastructure * Upstream some improvements to aiida-core: https://github.com/aiidateam/aiida-core/pull/6059 * Meeting with Edan, will start on PRs Xing: * Work on the XPS workchain to support calculating binding energy of specific atoms, use case: large system with low sysmetry, e.g. supported nanoparticle. * Work on Materials cloud archive and OPTIMADE integration script. Jason: * Test the "maximum recursion" issue. We can give the hotfix a patch release? * Working on OPTIMADE workshop nothing more to report. Francisco: * No news on aiida-core. Kristjan: * Nothing aiida-related to report. Giovanni: * Can we move the coding week 1 week before? Agree on tasks and who does it. * For coding week, there are a number of open PRs important for usability on disk-objectstore. If we agree, I'll probably work on those (support for compression, support for whence to loop from the back of a file, imporve archival format). * See discussion above on asyncio/greenlets ### Specific Agenda Items 1. Coding week: [Topics and tasks](https://hackmd.io/Hg2nbnUmSfinIx4WagjJfw)