---
tags: Asset management, Developer documentation
---
# Asset Indexing

Asset indexing provides a optimized way to load asset representations into the asset system. With asset indexing users have to spend significantly less time waiting for asset libraries to be loaded and ready to use.

Information about the asset like its asset meta-data (link), name and data-block type is extracted from asset files and stored in asset index files. Asset indexing can observe the content of asset libraries to perform minimize updates to the asset index files.

The asset index files are stored in a local cache of the users operating system.

:::warning
sub page of https://wiki.blender.org/wiki/Source/Architecture/Asset_System
:::

## Glossary

Glossary used within the scope of this page.

**Asset**
: Data-blocks that have been tagged by users to be an asset.

**Asset File**
: Blend file which is located inside an asset library.

**Asset Library**
: Entry point and container for asset representations and asset catalogs. https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Back_End#Asset_Libraries

**Asset Index File**
: JSON file containing information limited to selecting and browsing about assets contained inside a single asset file.

**Asset Catalog**
: Catalogs where assets can be organized in.

**Asset representation**
: An asset representation holds the necessary information about an asset so that the asset system can work with it, and the user interact with it as per design, without requiring the represented asset data to be available. It holds information like the asset's name, traits and other asset metadata. Asset representations are organized into asset libraries. [source](https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Back_End#Asset_Representation).

**File Browser**
: Editor in blender used for navigating file system and finding data-blocks in blend files. Used for linking, appending, file open, import, save and asset browsing functionalities.

**Data-block**
: manual https://wiki.blender.org/wiki/Source/Architecture/ID/ID_Type

## Logical Overview

object Asset
object "Blend file" as BlendFile
object "Asset file" as AssetFile
object "Asset library" as AssetLibrary
object "Asset index file" as AssetIndexFile {
    + version: int
}
object "Asset catalog" as AssetCatalog
object "Asset representation" as AssetRepresentation
object "Data-block" as DataBlock {
    + marked as asset: bool
}


BlendFile <|.. AssetFile
note right: An blend file is an asset file\nwhen part of an asset library\nIt may or may not contain assets.
BlendFile *- DataBlock
Asset ..|> DataBlock 
note left of Asset: An data-block is an asset\nwhen the data-block is marked as an asset.
AssetRepresentation -- Asset
AssetIndexFile *- AssetRepresentation

AssetLibrary o- AssetFile
Asset -o AssetCatalog

## Problem statement

Assets are stored in a blend file as part of an asset library. In order to browse through the assets of an asset library Blender has to parse each asset file in order to know which data-blocks have been tagged as assets.
Doing this each time when browsing assets and having large asset libraries extracting the assets can take a long time.

- Blender loads asset files sequentially in a background thread, which is a slow operation.
- Asset libraries can be located on network shares and version controlled repositories.
- Asset libraries can have a large number of files.
- Asset files can be large is size.
- Asset files can be compressed.
- Asset files may not actually contain assets.

## Solution

To reduce the time needed to browse asset libraries we want to extract the assets representation from the asset files and store them in the asset index. When the asset file isn't changed the asset index file would contain the required information needed to be able to reconstruct the [asset representation](https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Back_End#Asset_Representation).

### Requirements

**Each asset file has an asset index file.**

Each asset file has an asset index file even when the asset file has no assets. The reason is that asset files can be part of an asset library without containing assets. For example when an asset file contains materials which are linked from assets inside a different asset file, but those materials are not assets in them selves.

Not having an asset index file would parse the file each time.

**Icons of assets are not part of the asset index file.**

Icons compared to the rest of the asset representation can use a lot of disk space. Blender already has an icon manager that utilizes the OS to store and cache icons. Therefore we chose not to include the icon inside the asset representation.

**Empty asset files can be detected without opening the asset index file.**

Opening, reading and parsing asset index files has latency. To detect if there are actually asset representation stored inside the asset index file after parsing isn't efficient. By checking the file size on disk the asset indexer can already check if there are any asset representations inside the asset index file, removing the need for opening, reading and parsing the file.

:::info
The asset index is implemented as a [file indexer](https://hackmd.io/4PDN4EhISB2mwZFkIUtFFg) (`FileIndexerType`). Browsing assets shares logic with the file browser editor. File browser uses file indexers to read data from folders on a file-system, but also to list data-blocks located inside blend-files or data-blocks from asset libraries.

The usage of the File browser for asset listing and `FileIndexerType` is planned to be refactored. (ref #T999999)
:::

## Asset library loading process

@startuml name: Asset Indexer

AssetBrowser -> FileList
activate FileList
== Reading asset file index ==
    FileList -> AssetFileIndexer: read_index(filepath_to_asset_file)

    activate AssetFileIndexer
        alt asset index file is up to date
            FileList <-- AssetFileIndexer:  NeedsUpdate
            note right: Asset index file does not exist.\nIs asset file index created before asset file.\nAsset index file version is latest
        end

        alt asset index file is empty
            FileList <-- AssetFileIndexer: EntriesLoaded
        end

        AssetFileIndexer -> AssetIndexFile : read_contents
        activate AssetIndexFile
        deactivate AssetIndexFile

        AssetFileIndexer -> AssetIndexFile : extract_entries
        activate AssetIndexFile
        deactivate AssetIndexFile

        FileList <-- AssetFileIndexer: EntriesLoaded
    deactivate AssetFileIndexer

== Updating asset file index ==
    alt NeedsUpdate
        FileList -> FileList: read_entries_from_asset_file
        activate FileList
        deactivate FileList

        FileList -> AssetFileIndexer: update_entries(filepath_to_asset_file, entries)
        activate AssetFileIndexer
            AssetFileIndexer -> AssetIndexFile: write_contents
            activate AssetIndexFile
            deactivate AssetIndexFile
        FileList <-- AssetFileIndexer

        deactivate AssetFileIndexer
end

deactivate AssetFileIndexer

AssetBrowser <-- FileList
deactivate FileList
@enduml

The sequence diagram shown above describes an overview control flow of the asset indexer. When the asset browser needs to know the assets representations that are inside an asset library it uses a `FileList`. The file list walks over the file inside the asset library and when an asset file is detected it invokes `read_index` on the `AssetFileIndexer`. `read_index` returns if the found asset representations (`EntriesLoaded`) or that the asset index file needs to be updated for the given asset file.

In the case that the asset file index requires an update, the file list would extract the asset representations from the asset file and invoke `update_index`. Update entries would create update or create the asset index file with the given asset representations.

## Asset Index

When browsing assets, asset indexes are created when they don't exist or are out of date. The asset browser does not access asset files directly, but uses asset indexes.

### Storage

The asset index files are stored in the [local cache directory](https://docs.blender.org/manual/en/latest/advanced/blender_directory_layout.html#local-cache-directory). Within the local cache directory the indexes are stores using the following pattern:
`/asset-library-indices/<asset-library-hash>/<asset-index-hash>_<asset_file>.index.json`.

`asset-library-hash`
: Hash of the absolute file path of the asset library.

`asset-index-hash`
: Hash of the absolute file path of the asset file.

`asset_file`
: Filename of the asset file. Not used by Blender, but is added for discoverability convenience.

### Content

{
  "version": <file version number>,
  "entries": [{
    "name": "<asset name>",
    "catalog_id": "<catalog_id>",
    "catalog_name": "<catalog_name>",
    "description": "<description>",
    "author": "<author>",
    "tags": ["<tag>"],
    "properties": [..]
  }]
}

`version`
: the version of the asset index file. It is used to identify the structure of the content. Asset indexes with a different version than used by Blender would be regenerated. Blender 3.1-3.4 expect version attribute to be `1`.). Later versions might require to change it.

`entries`
: list of assets in the asset file related to the asset index.

`entries[].name`
: ID name of the asset, including its ID type prefix. (`OBSuzanne`).

`entries[].catalog_id`
: UUID of the catalog associated with the asset.

`entries[].catalog_name`
: Name of the catalog associated with the asset.

`entries[].description`
: Human readable description of the asset.

`entries[].author`
: Information about the author of the asset.

`entries[].tags`
: List of tags associated with this asset.

`entries[].properties`
: List of ID Properties associated with this asset.

## Performance

Performance is optimized for reading and parsing asset indexes.

- Asset libraries can have blend files that doesn’t contain any assets. To reduce opening and parsing an asset index that doesn’t contain any entries the size on disk of the asset index is checked. Asset indexes smaller than `MIN_FILE_SIZE_WITH_ENTRIES` would not be parsed and assumed to contain no entries.
- Asset indexes older than asset files are assumed to be out of date and should be regenerated. This can be determined without opening the asset index.
- Asset indexes of asset files on network shares are stored on your local system to reduce latency. Although this could depends on how your OS has been configured.

## Future expected changes

:::warning
To be added
:::

## References

- Technical overview of asset catalogs: `https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Catalogs`.
- The main source code is located at `source/blender/editors/asset/intern/asset_indexer.cc` and `source/blender/editors/asset/ED_asset_indexer.h`.
- End user documentation about assets library system `https://docs.blender.org/manual/en/latest/files/asset_libraries/introduction.html`, `https://docs.blender.org/manual/en/latest/files/asset_libraries/catalogs.html`