tags: library-overrides

Library Overrides

This page gives a technical overview of how library overrides are implemented. The goal is to only give an overview of what is already in the code-base, but does not go into the exact details how it is implemented.

More details on current development can be found on the Overrides project page.

This documentation uses the terms 'datablock' and 'ID' interchangeably.

---

Library overrides are essentially local, and thus editable, datablocks copied from linked ones. They are defined at a datablock level, in the ID data structure. This defines the granularity: a datablock is either overridden or it is not – there are no partial overrides.

However, Blender keeps track of which part of the datablock was edited. That way, changes to the originally linked datablock can be merged with the overrides. This process happens every time when opening a .blend file. As a result, library overrides behave like a hybrid of local data (it can be edited) and linked data (changes in the library file get pulled in).

It is possible to have multiple, distinct overrides for the same linked datablock. For example, multiple overrides of the same fully rigged character can be added to the same scene. This is made possible by the concept of an 'override hierarchy', representing a single 'asset' with all its dependencies. Hierarchies are defined by a single 'root' datablock, and the dependencies are explicitly part of the hierarchy by having a pointer back to the root datablock. This is part of the override data structure.

A more complex process is needed when the relationships between the linked datablocks change. For example, an object may get a different material, some new objects may be added to a collection, etc. This process is called 'resynchronisation', or 'resync' for short. It essentially rebuilds part of the override hierarchy from scratch from the linked data, and then transfers existing edits from the old overrides to the new ones.

Data Structure & Key Concepts

Library overrides are defined and stored at the ID level, in their own sub-struct IDOverrideLibrary. They are essentially made of a reference to their linked ID source (IDOverrideLibrary.reference), and of a list of 'override properties'.

Each of these override properties represent a different RNA property in the owner ID, identified by the RNA path. An override property is stored as an IDOverrideLibraryProperty struct, containing a list of IDOverrideLibraryPropertyOperation structs that define specific override operations to apply to their property. Typically, there is one operation per property (usually simply replacing the value of that property), but in some cases, like RNA collections that support insertion of new items, there can be several ones.

Note: the library overrides strucuture as described above only stores override operations, not override operands. In other words, they do not actually store any data. Data is stored in the local override ID, and this ID is used as source of the override operations when re-generating the override from its reference linked data (e.g. on file load).

What can or cannot be overridden is defined as part of the RNA properties definition, in their own set of flags PropertyOverrideFlag. Note that those flags also control the diffing behavior.

Embedded and other non-linkable IDs

Embedded (root node groups, master scene collections…) and non-linkable (shape keys…) IDs do not have their own library override struct. Instead, they are considered (from the overrides' point of view) as sub-data of their owner ID. This owner ID will also store their overridden properties. Such embedded IDs are flagged with LIB_EMBEDDED_DATA_LIB_OVERRIDE, and are also sometimes referred as 'virtual overrides' in the code.

Override Hierarchies

Usually more than one ID needs to be overridden in some form of group. A typical production character for example is made of hundreds of datablocks, and a lot of them need to be overridden in order to have a fully working override. Since there is a clear 'root' to this dependency tree, like a single collection or object, it is also called an 'override hierarchy'.

All overrides in a hierarchy must have a valid (non-NULL) IDOverrideLibrary.hierarchy_root pointer (roots point to themselves). Overrides that are not part of a hierarchy are flagged with IDOVERRIDE_LIBRARY_FLAG_NO_HIERARCHY. This helps with checking data consistency.

This hierarchy definition allows overrides from different hierarchies to have relationships with each other. For example, one override of a linked character can be parented to another override of the same linked character. Storing these override hierarchies explicitly also helps with complex hierarchical processes like resyncing.

User Overrides vs. System Overrides

User and system overrides can be distinguished by respectively being editable by the user and being read-only.

When a user creates a library override, this becomes a 'user override'. It can trigger the creation of 'system overrides' when they are needed to build the whole hierarchy. As an example, posing a linked character requires overriding the Object (because that contains the pose data). However, the mesh may contain drivers that need to refer to the overridden object, and thus the mesh itself also needs an override. The former is a user override, whereas the latter becomes a system override.

System overrides are flagged with IDOVERRIDE_LIBRARY_FLAG_SYSTEM_DEFINED in IDOverrideLibrary->flag. This flag is used (among others) in the editing code (UIs, operators, RNA access, etc) to determine whether a datablock is editable or read-only.

Since system overrides are non-editable in the UI, they are not expected to have any override properties/operations besides the system-required ID pointer ones.

TODO: describe 'system override properties'.
(i.e. it has an operation tagged with IDOVERRIDE_LIBRARY_FLAG_IDPOINTER_MATCH_REFERENCE)

File Layout

This section provides some pointers to start digging into the implementation details of library overrides.

• The core of library overrides are implemented in BKE's BKE_lib_overrides.h and lib_overrides.c.
• The DNA structures are defined in DNA_ID.h.
• The RNA-level property flags are defined in RNA_types.h, and the relevant API, in RNA_define.h and RNA_access.h.
• The RNA-level diffing and applying functionality is implemented in rna_access_compare_override.c.

Processes Overview

This section briefly describes the process of the most common library overrides operations.

It is possible that a library .blend file contains an overridden datablock from yet another file. In this section, such cases are ignored, and 'override' is understood to be an override defined in the current .blend file.

Making Library Override

To make an override of a linked datablock, a local copy is created and a library override structure is initialized (see BKE_lib_override_library_create_from_id).

In almost all cases, overrides are not created in isolation, but as part of an override hierarchy (either the whole hierarchy, or some sub-tree being added to an existing override hierarchy).

This process effectively always makes the linked data 'directly linked', i.e. its LIB_TAG_INDIRECT ID.tag is always cleared.

Making Local Changes

A user can work with the override in a similar, though restricted, way as with a regular local ID (using the UI, operators, the python API, etc.).

One of the main restrictions currently is that Edit modes are not allowed on overrides. Another example of such a restriction is that overrides cannot reorder or remove modifiers or constraints coming from the linked data.

To enforce these restrictions, many operators are checking for overrides in their poll callback, similar to checking for linked data.

Changes to Lists (RNA Collections)

The general case is that lists (a.k.a. RNA collections) are not editable for overrides, so you cannot add, remove, or re-order their items.

There are a few exceptions (modifiers, constraints and NLA tracks), where users can add new items, and subsequently re-organize those new items. Deletion or re-organization of existing items from linked data is currently forbidden.

Items that have been inserted in the override (and are therefore 'purely local') must be tagged as such, so that editing code can distinguish them from those coming from the linked ID. There is no common way to do it currently, all use different flags (eModifierFlag_OverrideLibrary_Local for modifiers' flag, CONSTRAINT_OVERRIDE_LIBRARY_LOCAL for constraints' flags, etc.).

Detecting Local Changes: Diffing

Local changes are detected when saving the file to disk and during undo step generation. This is called the 'diffing' process.

The diffing that is part of the undo step is used to provide feedback in the UI about what is overridden. BKE_lib_override_library_main_operations_create is called from BKE_undosys_step_push_with_type.

For each RNA property that is different between the original linked data and the override, an IDOverrideLibraryProperty with one or more operations are stored in the override structure.

For simple data properties (like float, int, string, enum) a single replace operation (IDOVERRIDE_LIBRARY_OP_REPLACE) is added. For vector types (color, coordinates), there can be several operations for some individual elements, or a single one for the whole array.

In case the diffing process finds modified RNA properties that are not allowed to be overridden, it will attempt to restore the value(s) from the linked reference data.

The default diffing process is implemented by rna_property_override_diff_default. In case a specific RNA property require non-standard diffing process, it can define its own callback, and pass it to RNA_def_property_override_funcs.

Diffing Lists (RNA collections)

To insert a new item to a list the IDOVERRIDE_LIBRARY_OP_INSERT_AFTER operation is used. It stores two names:

• the name of the item to insert, from the override data, and
• the name of the preceeding item, after which it will be inserted (the 'reference' or 'anchor' item).

The anchor item may be from the original linked data or an item previously added to the override. Because of this, the order of insertion operations is important as the anchor does need to exist at the time the operation is executed.

Writing Local Changes to Disk

When writing a .blend file, the override ID is stored just like a regular local datablock, with one key difference. The data stored on disk is size-optimized by removing some potentially heavy data sets that will never be overridden. E.g. with meshes, no geometry is written to disk (neither vertices, edges and faces, nor their custom data layers, can be overridden, see mesh.cc::mesh_blend_write).

The referenced linked ID is stored like any other directly linked datablock (i.e. its data is not written on disk, only its name and source library file).

During writing the override is checked to ensure that it contains all the necessary override operations (BKE_lib_override_library_main_operations_create is called from wm_file_write).

Reading Local Changes from Disk

When reading a .blend file, one of the last steps is to update the overrides.

This is done by calling BKE_lib_override_library_main_update, at the end of blo_read_file_internal. This function will go over all the IDs that have a library override structure, and:

• Make a new local copy from their linked reference.
• Apply the override operations stored in the old override (just read from disk) to the new copy, using the old override as source data. This is handled in RNA code (see RNA_struct_override_apply).

This ensures that the override data is always following as close as possible the data from their linked reference.

If an override property's RNA path is not valid anymore, it is ignored (a message is printed in the console), and will be cleaned up during the next diffing operation.

If the linked reference ID is not found, there will be no update and the override is kept as-is. This preserves the override data as well as possible when some library goes missing.

The library overrides can be chained: it is possible o link an override from another .blend file, and then create another override on top of this linked data. This is handled in BKE_lib_override_library_update, by checking whether the reference linked data is also an override itself, and if so, recursively calling BKE_lib_override_library_update on it first.

Resync

Resync is an operation to update the override hierarchy when it does not match the hierarchy of its linked reference data anymore (i.e. when relations between IDs have been added, changed or removed in the linked library data).

Resyncing can happen automatically on all data on file load ('auto-resync'), or on a per-datablock level from the outliner ('manual resync').

See BKE_lib_override_library_resync for manual resync, and BKE_lib_override_library_main_resync for auto-resync. Auto-resync is by default run as part of setup_app_data, after reading the .blend file.

Resync Conflicts & Resolution

Resync conflicts will happen when reloading the working file, and the reference linked data does not match the overrides anymore. This section lists the possible kind of conflicts, and describes their resolutions.

The linked datablock remains the same, but its content changed (e.g. some modifiers were added or removed).

The situation is similar to animation's FCurves or drivers: some override properties may become invalid (their RNA path would not match anything in override data). Currently, this is simply ignored, and invalid override properties are removed on next file save.

A new linked datablock is added, or its relationships to others are modified, and it should now become overridden as well.

This is fully automatically handled by the resync code, roughly:

• The affected parts of the hierarchy are re-created from the linked data.
• Overrides from the old pre-resync data are ported over to the post-resync data. If this is not possible, the old override data is lost.

The usage of a linked datablock is reassigned to another one, or is cleared.

Example: an object and its material are overridden, and the last time the file was saved, the linked object used Material A. Now that it is reopened, the linked object is using material B.

System override: If the override property is referencing the replaced/deleted ID, and it has a 'system override' operation, the handling is similar as the "new linked datablock" case above.

User override: If the existing override was user-edited and diverges from the linked-defined hierarchy, it will remain unchanged, unless do_hierarchy_enforce is specified (only possible for manual resync operations).

TODO: add a flow diagram of this process.