# Building and distributing

###### tags: `book`

Some introduction

The end goal is to be able to make our code installable with package managers such as `conda` or `pip` without the end user having to worry about compiling code, setting up paths for shared libraries etc.

## f2py

Packaging Fortran code with `f2py` is embarrasingly easy thanks to [`numpy.distutils`](https://numpy.org/doc/stable/reference/distutils.html). We can tell setuptools to build Fortran source files by using the `numpy.distutils` drop-in replacements for the regular `Extension` and `setup`.

Let's start with our function that does addition.

! file: add.f90
integer function add(a, b)
  integer :: a, b
  add = a + b
end function add

We now write a minimal `setup.py` that will install our function within a Python package.

from numpy.distutils.core import Extension, setup

setup(name='f2py-example',
      ext_modules = [Extension("f2py_example.add", sources=["add.f90"])]
      )

We install with `pip install .` and can now import into Python:

>>> from f2py_example.add import add
>>> add(1, 2)
3

It does not get easier than this.

> :warning: Do not forget to uninstall with `pip uninstall f2py-example` when finished.

## ctypes

`ctypes` does not provide any tools for building or distribution.
The most pragmatic solution is to package the shared library directly with the code using the `package_data` keyword in `setup.py`.
We use the following package structure:

.
├── ctypes_example
│   ├── __init__.py
│   ├── add.py
│   └── lib
│       └── mod_add.so
└── setup.py

and this `setup.py`:

from setuptools import setup

setup(
    name='ctypes-example',
    packages=["ctypes_example"],
    package_data={"ctypes_example": ["lib/mod_add.so"]},
)

The `package_data` keyword will instruct setuptools to copy the `lib/mod_add.so` into the built Python package.

We have created the `add.py` module that loads `mod_add.so` using the `pathlib` library and exports the addition function to the module namespace:

from ctypes import CDLL, c_int
from pathlib import Path
import sys

file_path = Path(__file__).parent / "lib" / "mod_add.so"
lib = CDLL(file_path)

add = lib.add

After installing with `pip install .`, we can import the function from our new Python package.

>>> from ctypes_example.add import add
>>> add(1, 2)
3

### Supporting different platforms

Shared libraries need to be compiled separately for each platform that you want to support.
One option is to distribute all of them with the package and use [`sys.platform`](https://docs.python.org/3/library/sys.html#sys.platform) to load the right one at runtime. Alternatively, tools like [cibuildwheel](https://github.com/pypa/cibuildwheel) can be used to set up an automatic build of (1) the shared library and (2) the Python package for multiple platforms.

## cffi

## Cython

We start with the following project structure.

.
├── cython_example
│   ├── __init__.py
│   ├── c_mod_sum.pxd
│   └── mod_sum.pyx
├── include
│   └── mod_sum.h
├── lib
│   └── libmod_sum.dylib
├── pyproject.toml
└── setup.py

The files `c_mod_sum.pxd`, `mod_sum.pyx`, `mod_sum.h` and `libmod_sum.so` are the same files that we have worked with [previously](/UsaVP1D4QZu2tTUz8D8T8Q#Cython). Here is the `setup.py`:

from setuptools import setup, Extension
from Cython.Build import cythonize

setup(
    name='cython-example',
    packages=["cython_example"],
    install_requires=["numpy"],
    ext_modules = cythonize([
        Extension(
            name="cython_example.sum",
            sources=["cython_example/mod_sum.pyx"],
            libraries=["mod_sum"],
            library_dirs=["lib"],
            include_dirs=["include"],
        ),
    ]),
)

There are some changes to have we worked previously:

* `library_dirs` is set to `lib`
* We have put the C header file in the `include` directory and link to it via the `include_dirs` keyword.
* The extension is installed as a module, `sum`, in the `cython_example` package.

The `pyproject.toml` now also lists Cython as a dependency:

[build-system]
requires = ["setuptools", "wheel", "Cython"]
build-backend = "setuptools.build_meta"

With everything set up, we can create a binary wheels file with setuptools:

$ python setup.py bdist_wheel

If we try to install the wheels with

pip install dist/cython_example-0.0.0-cp39-cp39-macosx_10_9_x86_64.whl

and import the package, we encounter the following error:

>>> from cython_example.sum import sum
ImportError: dlopen(/../sum.cpython-39-darwin.so, 2): Library not loaded: mod_sum.so
  Referenced from: /.../sum.cpython-39-darwin.so
  Reason: image not found

The problem is that the shared library is not available to Python at runtime. We could solve it by setting the `LD_LIBRARY_PATH` variable before running Python, but that option would not be available to the end user who installs our package from PyPi. Instead, we will patch the wheels.

### Patching the wheels

To package the shared library with the wheel file, we need to use external and platform-specific tools.

| OS | Tool |
| ------- | -------------------------------------------------------------------- |
| Linux | [auditwheel](https://github.com/pypa/auditwheel) |
| macOS | [delocate](http://github.com/matthew-brett/delocate) |
| Windows | [delvewheel](https://github.com/adang1345/delvewheel) (experimental) |

Patching the wheels is a very platform-specific procedure, and we illustrate it here exemplified using `delocate` on MacOS.
Guides for how to use the other tools for Linux and Windows can reached through the links in the table above.

First we need to change the *install name* of our shared library if it was generated with `gfortran`:

$ install_name_tool -id @rpath/libmod_sum.dylib lib/libmod_sum.dylib

> :bulb: We switched the suffix from `.so` to `.dylib` as we are working on MacOS here

We then go into the `dist` directory and patch the wheel file with `delocate`.
We use the `DYLD_LIBRARY_PATH` environment variable to tell `delocate` where it can find our shared library.

$ cd dist
$ DYLD_LIBRARY_PATH="$PWD/../lib/" delocate-wheel -w fixed cython_example-0.0.0-cp39-cp39-macosx_10_9_x86_64.whl

Now we are ready to install the patched wheels:

$ cd fixed
$ pip install cython_example-0.0.0-cp39-cp39-macosx_10_9_x86_64.whl

The Python package is now ready to use and could be uploaded to PyPi.

>>> from cython_example.sum import sum_columns
>>> import numpy as np
>>> a = np.array([[1, 2, 3], [1, 2, 3]])
>>> sum_columns(a)
array([2., 4., 6.])