Building and distributing

tags: book

Some introduction

The end goal is to be able to make our code installable with package managers such as conda or pip without the end user having to worry about compiling code, setting up paths for shared libraries etc.

f2py

Packaging Fortran code with f2py is embarrasingly easy thanks to numpy.distutils. We can tell setuptools to build Fortran source files by using the numpy.distutils drop-in replacements for the regular Extension and setup.

Let's start with our function that does addition.

! file: add.f90
integer function add(a, b)
  integer :: a, b
  add = a + b
end function add

We now write a minimal setup.py that will install our function within a Python package.

from numpy.distutils.core import Extension, setup

setup(name='f2py-example',
      ext_modules = [Extension("f2py_example.add", sources=["add.f90"])]
      )

We install with pip install . and can now import into Python:

>>> from f2py_example.add import add
>>> add(1, 2)
3

It does not get easier than this.

:warning: Do not forget to uninstall with pip uninstall f2py-example when finished.

ctypes

ctypes does not provide any tools for building or distribution. The most pragmatic solution is to package the shared library directly with the code using the package_data keyword in setup.py. We use the following package structure:

.
├── ctypes_example
│   ├── __init__.py
│   ├── add.py
│   └── lib
│       └── mod_add.so
└── setup.py

and this setup.py:

from setuptools import setup

setup(
    name='ctypes-example',
    packages=["ctypes_example"],
    package_data={"ctypes_example": ["lib/mod_add.so"]},
)

The package_data keyword will instruct setuptools to copy the lib/mod_add.so into the built Python package.

We have created the add.py module that loads mod_add.so using the pathlib library and exports the addition function to the module namespace:

from ctypes import CDLL, c_int
from pathlib import Path
import sys

file_path = Path(__file__).parent / "lib" / "mod_add.so"
lib = CDLL(file_path)

add = lib.add

After installing with pip install ., we can import the function from our new Python package.

>>> from ctypes_example.add import add
>>> add(1, 2)
3

Supporting different platforms

Shared libraries need to be compiled separately for each platform that you want to support. One option is to distribute all of them with the package and use sys.platform to load the right one at runtime. Alternatively, tools like cibuildwheel can be used to set up an automatic build of (1) the shared library and (2) the Python package for multiple platforms.

cffi

Cython

We start with the following project structure.

.
├── cython_example
│   ├── __init__.py
│   ├── c_mod_sum.pxd
│   └── mod_sum.pyx
├── include
│   └── mod_sum.h
├── lib
│   └── libmod_sum.dylib
├── pyproject.toml
└── setup.py

The files c_mod_sum.pxd, mod_sum.pyx, mod_sum.h and libmod_sum.so are the same files that we have worked with previously. Here is the setup.py:

from setuptools import setup, Extension
from Cython.Build import cythonize

setup(
    name='cython-example',
    packages=["cython_example"],
    install_requires=["numpy"],
    ext_modules = cythonize([
        Extension(
            name="cython_example.sum",
            sources=["cython_example/mod_sum.pyx"],
            libraries=["mod_sum"],
            library_dirs=["lib"],
            include_dirs=["include"],
        ),
    ]),
)

There are some changes to have we worked previously:

• library_dirs is set to lib
• We have put the C header file in the include directory and link to it via the include_dirs keyword.
• The extension is installed as a module, sum, in the cython_example package.

The pyproject.toml now also lists Cython as a dependency:

[build-system]
requires = ["setuptools", "wheel", "Cython"]
build-backend = "setuptools.build_meta"

With everything set up, we can create a binary wheels file with setuptools:

$ python setup.py bdist_wheel

If we try to install the wheels with

pip install dist/cython_example-0.0.0-cp39-cp39-macosx_10_9_x86_64.whl

and import the package, we encounter the following error:

>>> from cython_example.sum import sum
ImportError: dlopen(/../sum.cpython-39-darwin.so, 2): Library not loaded: mod_sum.so
  Referenced from: /.../sum.cpython-39-darwin.so
  Reason: image not found

The problem is that the shared library is not available to Python at runtime. We could solve it by setting the LD_LIBRARY_PATH variable before running Python, but that option would not be available to the end user who installs our package from PyPi. Instead, we will patch the wheels.

Patching the wheels

To package the shared library with the wheel file, we need to use external and platform-specific tools.

OS	Tool
Linux	auditwheel
macOS	delocate
Windows	delvewheel (experimental)

Patching the wheels is a very platform-specific procedure, and we illustrate it here exemplified using delocate on MacOS. Guides for how to use the other tools for Linux and Windows can reached through the links in the table above.

First we need to change the install name of our shared library if it was generated with gfortran:

$ install_name_tool -id @rpath/libmod_sum.dylib lib/libmod_sum.dylib

:bulb: We switched the suffix from .so to .dylib as we are working on MacOS here

We then go into the dist directory and patch the wheel file with delocate. We use the DYLD_LIBRARY_PATH environment variable to tell delocate where it can find our shared library.

$ cd dist
$ DYLD_LIBRARY_PATH="$PWD/../lib/" delocate-wheel -w fixed cython_example-0.0.0-cp39-cp39-macosx_10_9_x86_64.whl

Now we are ready to install the patched wheels:

$ cd fixed
$ pip install cython_example-0.0.0-cp39-cp39-macosx_10_9_x86_64.whl

The Python package is now ready to use and could be uploaded to PyPi.

>>> from cython_example.sum import sum_columns
>>> import numpy as np
>>> a = np.array([[1, 2, 3], [1, 2, 3]])
>>> sum_columns(a)
array([2., 4., 6.])