PyMutex

Scope

  • Non-limited C API: excluded from the limited C API and stable ABI.
  • API part of accepted PEP 703 (PyObject.ob_mutex).

API

typedef struct { uint8_t _bits; } PyMutex;
#define PyMutex_STATIC_INIT { 0 }
static inline void PyMutex_Init(PyMutex *m) { *m = PyMutex_STATIC_INIT; }
static inline void PyMutex_Lock(PyMutex *m) { ... }
static inline void PyMutex_Unlock(PyMutex *m) { ... }

These functions cannot fail to make them convenient to use.

We can additionally provide these functions as an actual exported function (regular "opaque" function) for non-C languages such as Rust.

Maybe add also this function for debugging:

int PyMutex_IsLocked(PyMutex *m)

Sam: It's not useful for making control flow decisions, but it's useful for assertions.

Usage

Static mutex

static PyMutex lock = PyMutex_STATIC_INIT;

void use(void)
{
    PyMutex_Lock(&lock);
    // ... protected against race conditions ...
    PyMutex_Unlock(&lock);
}

Allocated on the heap memory

typedef struct {
    PyObject_HEAD
    PyMutex lock;
} MyObject;

PyObject* new_object(void)
{
    // allocate uninitialized memory
    MyObject *obj = PyObject_GC_New(MyObject, MyType);
    PyMutex_Init(&obj->lock);
    return obj;
}

void use_myobject(MyObject *obj)
{
    PyMutex_Lock(&obj->lock);
    // ... use the object safely ...
    PyMutex_Unlock(&obj->lock);
}

It's an artificial example since all objects already contain a mutex in PyObject header (PyObject.ob_mutex).

Errors

Sam: The only possible failures are due to incorrect usage, like unlocking a mutex that is not locked, but returning an error code in that case isn't a good design because they are likely to be ignored.

Sam: Adding error codes complicates the use of the API to the point where callers are best served by just ignoring them.

Background

The idea is proven. "Parking lot" implementation used in WebKit for 9 years (since Aug 2015): wtf/Lock.h. The major change since then was in July 2016 when it became "eventually fair", which entailed going from three states (two bits in one byte) to four states (still two bits in one byte); the major implementation changes happen at a lower levels of the API.

typedef struct {
    uint8_t _v;
} PyMutex;

Internally, there's a hash table that maintains a mapping of PyMutex address to struct mutex_entry* for mutexes with threads waiting on them.

The static inline functions call private functions such as:

  • _Py_atomic_compare_exchange_uint8()
  • _PyMutex_LockSlow()
  • _PyMutex_UnlockSlow()

API Stability

Petr: I don't think API stability is an issue. Changing the PyMutex size will break ABI, not API (i.e. we can do it a new feature release).

Steve: I also want the fact that both the address and the value of the mutex matters to be clearly specified, along with whatever constraints that implies (for languages other than C).

  • Petr: Docs for “both the address and the value” can be added. Some non-C languages call this “pinning” (rust, golang).

ABI Stability

Out of the scope, since PyMutex is excluded from the stable ABI.

Reserve space in PyMutex structure

Steve proposes:

struct PyMutex {
    union {
        uint8_t _v;
        void *_reserved;
    };
};

This change would make PyObject header bigger (+7 bytes) in Free Threaded build (PEP 703).

Sam: It's incompatible with the one-byte internal definition and doesn't adhere to the "one definition rule."

Note: nameless unions are compiler extensions.

Destroy

Add function:

static inline void PyMutex_Destroy(PyMutex *m) { ... }

Sam: I think that PyMutex_Destroy is not a good addition:

  • It doesn't do what the name suggests – there is nothing to destroy and the primary suggested purpose is something else (asserting that there are no waiters).
  • It does not let you implement a non-trivial destroy in the future without breaking users – you would almost certainly break existing users in weird edge cases, like after fork.
Select a repo