owned this note
owned this note
Published
Linked with GitHub
# General policy for interpreted language environments
Two primary goals:
1. Avoid needing extra duplicate -bare/-minimal variants of Perl/Python
2. Avoid having many optional nice-to-have packages that bloat the installations (primarily R)
We want to have clear policy on what goes where to enforce this consistently.
## Rules for base easyconfig (Perl, Python, R, ...)
There will be a single easyconfig providing the interpreter of the language and it will be minimal in scope:
* include dependencies needed to build/install and run the barebones interpreter
* include extensions needed to install extra packages on top of base installation (*i.e.* packaging tools, maybe some build tools)
* keep runtime dependencies to a minimum to avoid circular dependencies
## Perl
Needed to install Autotools, which makes it a very early dependency.
A completely bare Perl isn't enough for Automake/conf (see [here](https://github.com/easybuilders/easybuild-easyconfigs/issues/8704)), so options are:
1. Adding a few extra extensions that Autotools need.
2. Bundle some of the extra extensions that Autotools need in a seperate bundle.
Open question: what to do with the many many more perl extensions (of which some will introduce indirect dependency on Autotools). Make one or more bundle? What should these be named?
- (suggestion by Kenneth) What about `Perl-bundle-CPAN` (since all Perl modules come from CPAN), equivalent to `R-bundle-Bioconductor`?
- Alex: I would start by putting all current extensions in `Perl-bundle-CPAN`
## Python
Must make sure to not include Rust or any other extra dependency.
As of 3.11 the list of dependencies for just bare Python would be
```python
builddependencies = [
('UnZip', '6.0'),
('pkgconf', '1.9.4'),
]
dependencies = [
('binutils', '2.40'),
('bzip2', '1.0.8'), # required for bz2 package in Python stdlib
('zlib', '1.2.13'),
('libreadline', '8.2'),
('ncurses', '6.4'),
('SQLite', '3.41.2'),
('XZ', '5.4.2'),
('libffi', '3.4.4'),
('OpenSSL', '1.1', '', SYSTEM),
]
```
Some possible options are
1. Keep Python completely bare
2. Add just a minimal set of: pip, setuptools (this is what venv gives you)
3. Add just a minimal set of: pip, wheel, setuptools (these is what virtualenv gives you)
4. All of the above plus virtualenv + deps
5. All of the above plus pytest, cython, poetry and any other very common build tool.
Risk of including more stuff is that the risk that one of them will switch out their core stuff to Rust in the future will increase, forcing is to exclude them then, which again introduces more changes we have to explain to users.
Build tools should be safe to have as separate packages, doesn't cause much harm with temporarily having extra PYTHONPATH's. Might increase workload with easyconfig PRs if we have to many small packages or bundles though.
Alex: I like option 4, including virtualenv seems a minimum to provide a working vanila environment for end-users
Mikael: virtualenv does explode into quite a few extra extensions as dependencies, but a lot of it the build step ones can be avoided using the wheels
Kenneth: easyconfigs currently included with Python can be moved into a new easyconfig `Python-bundle-PyPI`?
Build tools like `poetry`, `hatchling`, etc. should be split off into separate easyconfigs, so they can be included only as build dependency?
## R
Options are
1. Keep R completely bare
2. Add a few key libraries, perhaps `devtools` or similar, if R has that? (I (Mikael) don't know enough about R's ecosystem)
and move the rest of the extensions into one or more bundles.
Mikael suggests: Keep R bare and make something like `R-common` for things that we need as deps for other easyconfigs. Keep `R-contrib` that each centre can add their specific stuff to. Not sure if we want that upstream, not sure if there really is anything we should put in `R-common`.
Kenneth: Extensions currently included with `R` can be moved into a new easyconfig `R-bundle-CRAN` (naming matches well with current `R-bundle-Bioconductor`)
John Dey (fizwit): Can we drop `-bundle-` from the name since it is impylied? Can we aggree on some package names; R-Tidyverse-4.3.x, R-Shiny-4.3.x, R-Bioconductor-4.3.x etc. I can not think of case where I would want an R easyconfig with a single library from CRAN.