###### tags: `library-overrides`

# Library Overrides

*This page gives a technical overview of how library overrides are implemented. The goal is to only give an overview of what is already in the code-base, but does not go into the exact details how it is implemented.*

*More details on current development can be found on the [Overrides project page](https://developer.blender.org/project/profile/141/).*

This documentation uses the terms 'datablock' and 'ID' interchangeably.

---

Library overrides are essentially local, and thus editable, datablocks copied from linked ones. They are defined at a datablock level, in the `ID` data structure. This defines the granularity: a datablock is either overridden or it is not -- there are no partial overrides.

However, Blender keeps track of which part of the datablock was edited. That way, changes to the originally linked datablock can be merged with the overrides. This process happens every time when opening a `.blend` file. As a result, library overrides behave like a hybrid of local data (it can be edited) and linked data (changes in the library file get pulled in).

It is possible to have multiple, distinct overrides for the same linked datablock. For example, multiple overrides of the same fully rigged character can be added to the same scene. This is made possible by the concept of an 'override hierarchy', representing a single 'asset' with all its dependencies. Hierarchies are defined by a single 'root' datablock, and the dependencies are explicitly part of the hierarchy by having a pointer back to the root datablock. This is part of the override data structure.

A more complex process is needed when the relationships between the linked datablocks change. For example, an object may get a different material, some new objects may be added to a collection, etc. This process is called 'resynchronisation', or 'resync' for short. It essentially rebuilds part of the override hierarchy from scratch from the linked data, and then transfers existing edits from the old overrides to the new ones.

## Data Structure & Key Concepts

Library overrides are defined and stored at the ID level, in their own sub-struct `IDOverrideLibrary`. They are essentially made of a reference to their linked ID source (`IDOverrideLibrary.reference`), and of a list of 'override properties'.

Each of these override properties represent a different RNA property in the owner ID, identified by the RNA path. An override property is stored as an `IDOverrideLibraryProperty` struct, containing a list of `IDOverrideLibraryPropertyOperation` structs that define specific override operations to apply to their property. Typically, there is one operation per property (usually simply replacing the value of that property), but in some cases, like RNA collections that support insertion of new items, there can be several ones.

**Note: the library overrides strucuture as described above only stores override operations, not override operands.** In other words, they do not actually store any data. Data is stored in the local override ID, and this ID is used as source of the override operations when re-generating the override from its reference linked data (e.g. on file load).

What can or cannot be overridden is defined as part of the RNA properties definition, in their own set of flags `PropertyOverrideFlag`. Note that those flags also control the [diffing behavior](#Detecting-Local-Changes-Diffing).

### Embedded and other non-linkable IDs

Embedded (root node groups, master scene collections…) and non-linkable (shape keys…) IDs do not have their own library override struct. Instead, they are considered (from the overrides' point of view) as sub-data of their owner ID. This owner ID will also store their overridden properties. Such embedded IDs are flagged with `LIB_EMBEDDED_DATA_LIB_OVERRIDE`, and are also sometimes referred as 'virtual overrides' in the code.

### Override Hierarchies

Usually more than one ID needs to be overridden in some form of group. A typical production character for example is made of hundreds of datablocks, and a lot of them need to be overridden in order to have a fully working override. Since there is a clear 'root' to this dependency tree, like a single collection or object, it is also called an 'override hierarchy'.

All overrides in a hierarchy must have a valid (non-`NULL`) `IDOverrideLibrary.hierarchy_root` pointer (roots point to themselves). Overrides that are **not** part of a hierarchy are flagged with `IDOVERRIDE_LIBRARY_FLAG_NO_HIERARCHY`. This helps with checking data consistency.

This hierarchy definition allows overrides from different hierarchies to have relationships with each other. For example, one override of a linked character can be parented to another override of the same linked character. Storing these override hierarchies explicitly also helps with complex hierarchical processes like [resyncing](#Resync).

### User Overrides vs. System Overrides

User and system overrides can be distinguished by respectively being editable by the user and being read-only.

When a user creates a library override, this becomes a 'user override'. It can trigger the creation of 'system overrides' when they are needed to build the whole hierarchy. As an example, posing a linked character requires overriding the Object (because that contains the pose data). However, the mesh may contain drivers that need to refer to the overridden object, and thus the mesh itself also needs an override. The former is a user override, whereas the latter becomes a system override.

System overrides are flagged with `IDOVERRIDE_LIBRARY_FLAG_SYSTEM_DEFINED` in `IDOverrideLibrary->flag`. This flag is used (among others) in the editing code (UIs, operators, RNA access, etc) to determine whether a datablock is editable or read-only.

Since system overrides are non-editable in the UI, they are not expected to have any override properties/operations besides the system-required ID pointer ones.

:::warning
**TODO:** describe 'system override properties'.
(i.e. it has an operation tagged with `IDOVERRIDE_LIBRARY_FLAG_IDPOINTER_MATCH_REFERENCE`)
:::

## File Layout

This section provides some pointers to start digging into the implementation details of library overrides.

- The core of library overrides are implemented in BKE's `BKE_lib_overrides.h` and `lib_overrides.c`.
- The DNA structures are defined in `DNA_ID.h`.
- The RNA-level property flags are defined in `RNA_types.h`, and the relevant API, in `RNA_define.h` and `RNA_access.h`.
- The RNA-level diffing and applying functionality is implemented in `rna_access_compare_override.c`.

## Processes Overview

This section briefly describes the process of the most common library overrides operations.

It is possible that a library `.blend` file contains an overridden datablock from yet another file. In this section, such cases are ignored, and 'override' is understood to be an override defined in the current `.blend` file.

### Making Library Override

To make an override of a linked datablock, a local copy is created and a library override structure is initialized (see `BKE_lib_override_library_create_from_id`).

In almost all cases, overrides are not created in isolation, but as part of an override hierarchy (either the whole hierarchy, or some sub-tree being added to an existing override hierarchy).

:::info
This process effectively always makes the linked data 'directly linked', i.e. its `LIB_TAG_INDIRECT` `ID.tag` is always cleared.
:::

### Making Local Changes

A user can work with the override in a similar, though restricted, way as with a regular local ID (using the UI, operators, the python API, etc.).

One of the main restrictions currently is that Edit modes are not allowed on overrides. Another example of such a restriction is that overrides cannot reorder or remove modifiers or constraints coming from the linked data.

To enforce these restrictions, many operators are checking for overrides in their `poll` callback, similar to checking for linked data.

#### Changes to Lists (RNA Collections)

The general case is that lists (a.k.a. RNA collections) are not editable for overrides, so you cannot add, remove, or re-order their items.

There are a few exceptions (modifiers, constraints and NLA tracks), where users can add new items, and subsequently re-organize those new items. Deletion or re-organization of *existing* items from linked data is currently forbidden.

Items that have been inserted in the override (and are therefore 'purely local') must be tagged as such, so that editing code can distinguish them from those coming from the linked ID. There is no common way to do it currently, all use different flags (`eModifierFlag_OverrideLibrary_Local` for modifiers' flag, `CONSTRAINT_OVERRIDE_LIBRARY_LOCAL` for constraints' flags, etc.).

### Detecting Local Changes: Diffing

Local changes are detected when saving the file to disk and during undo step generation. This is called the 'diffing' process.

The diffing that is part of the undo step is used to provide feedback in the UI about what is overridden. `BKE_lib_override_library_main_operations_create` is called from `BKE_undosys_step_push_with_type`.

For each RNA property that is different between the original linked data and the override, an `IDOverrideLibraryProperty` with one or more operations are stored in the override structure.

For simple data properties (like float, int, string, enum) a single replace operation (`IDOVERRIDE_LIBRARY_OP_REPLACE`) is added. For vector types (color, coordinates), there can be several operations for some individual elements, or a single one for the whole array.

In case the diffing process finds modified RNA properties that are not allowed to be overridden, it will attempt to restore the value(s) from the linked reference data.

The default diffing process is implemented by `rna_property_override_diff_default`. In case a specific RNA property require non-standard diffing process, it can define its own callback, and pass it to `RNA_def_property_override_funcs`.

#### Diffing Lists (RNA collections)

To insert a new item to a list the `IDOVERRIDE_LIBRARY_OP_INSERT_AFTER` operation is used. It stores two names:
- the name of the item to insert, from the override data, and
- the name of the preceeding item, after which it will be inserted (the 'reference' or 'anchor' item).

The anchor item may be from the original linked data or an item previously added to the override. Because of this, the order of insertion operations is important as the anchor does need to exist at the time the operation is executed.

### Writing Local Changes to Disk

When writing a `.blend` file, the override ID is stored just like a regular local datablock, with one key difference. The data stored on disk is size-optimized by removing some potentially heavy data sets that will never be overridden. E.g. with meshes, no geometry is written to disk (neither vertices, edges and faces, nor their custom data layers, can be overridden, see `mesh.cc::mesh_blend_write`).

The referenced linked ID is stored like any other directly linked datablock (i.e. its data is not written on disk, only its name and source library file).

During writing the override is checked to ensure that it contains all the necessary override operations (`BKE_lib_override_library_main_operations_create` is called from `wm_file_write`).

### Reading Local Changes from Disk

When reading a `.blend` file, one of the last steps is to update the overrides.

This is done by calling `BKE_lib_override_library_main_update`, at the end of `blo_read_file_internal`. This function will go over all the IDs that have a library override structure, and:

- Make a new local copy from their linked reference.
- Apply the override operations stored in the old override (just read from disk) to the new copy, using the old override as source data. This is handled in RNA code (see `RNA_struct_override_apply`).

This ensures that the override data is always following as close as possible the data from their linked reference.

If an override property's RNA path is not valid anymore, it is ignored (a message is printed in the console), and will be cleaned up during the next diffing operation.

If the linked reference ID is not found, there will be no update and the override is kept as-is. This preserves the override data as well as possible when some library goes missing.

:::info
The library overrides can be chained: it is possible o link an override from another `.blend` file, and then create another override on top of this linked data. This is handled in `BKE_lib_override_library_update`, by checking whether the reference linked data is also an override itself, and if so, recursively calling `BKE_lib_override_library_update` on it first.
:::

### Resync

Resync is an operation to update the override hierarchy when it does not match the hierarchy of its linked reference data anymore (i.e. when relations between IDs have been added, changed or removed in the linked library data).

Resyncing can happen automatically on all data on file load ('auto-resync'), or on a per-datablock level from the outliner ('manual resync').

See `BKE_lib_override_library_resync` for manual resync, and `BKE_lib_override_library_main_resync` for auto-resync. Auto-resync is by default run as part of `setup_app_data`, after reading the `.blend` file.

#### Resync Conflicts & Resolution

Resync conflicts will happen when reloading the working file, and the reference linked data does not match the overrides anymore. This section lists the possible kind of conflicts, and describes their resolutions.

The linked datablock remains the same, but its content changed (e.g. some modifiers were added or removed).
: The situation is similar to animation's FCurves or drivers: some override properties may become invalid (their RNA path would not match anything in override data). Currently, this is simply ignored, and invalid override properties are removed on next file save.

A new linked datablock is added, or its relationships to others are modified, and it should now become overridden as well.
: This is fully automatically handled by the resync code, roughly:
- The affected parts of the hierarchy are re-created from the linked data.
- Overrides from the old pre-resync data are ported over to the post-resync data. If this is not possible, the old override data is lost.

The usage of a linked datablock is reassigned to another one, or is cleared.
: Example: an object and its material are overridden, and the last time the file was saved, the linked object used Material A. Now that it is reopened, the linked object is using material B.
: **System override:** If the override property is referencing the replaced/deleted ID, and it has a ['system override' operation](#User-Overrides-vs-System-Overrides), the handling is similar as the "new linked datablock" case above.
: **User override:** If the existing override was user-edited and diverges from the linked-defined hierarchy, it will remain unchanged, unless `do_hierarchy_enforce` is specified (only possible for manual resync operations).

:::warning
**TODO:** add a flow diagram of this process.
:::