owned this note
owned this note
Published
Linked with GitHub
---
tags: Asset management, Developer documentation
---
# Asset Indexing
Asset indexing provides a optimized way to load asset representations into the asset system. With asset indexing users have to spend significantly less time waiting for asset libraries to be loaded and ready to use.
Information about the asset like its asset meta-data (link), name and data-block type is extracted from asset files and stored in asset index files. Asset indexing can observe the content of asset libraries to perform minimize updates to the asset index files.
The asset index files are stored in a local cache of the users operating system.
:::warning
sub page of https://wiki.blender.org/wiki/Source/Architecture/Asset_System
:::
## Glossary
Glossary used within the scope of this page.
**Asset**
: Data-blocks that have been tagged by users to be an asset.
**Asset File**
: Blend file which is located inside an asset library.
**Asset Library**
: Entry point and container for asset representations and asset catalogs. https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Back_End#Asset_Libraries
**Asset Index File**
: JSON file containing information limited to selecting and browsing about assets contained inside a single asset file.
**Asset Catalog**
: Catalogs where assets can be organized in.
**Asset representation**
: An asset representation holds the necessary information about an asset so that the asset system can work with it, and the user interact with it as per design, without requiring the represented asset data to be available. It holds information like the asset's name, traits and other asset metadata. Asset representations are organized into asset libraries. [source](https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Back_End#Asset_Representation).
**File Browser**
: Editor in blender used for navigating file system and finding data-blocks in blend files. Used for linking, appending, file open, import, save and asset browsing functionalities.
**Data-block**
: manual https://wiki.blender.org/wiki/Source/Architecture/ID/ID_Type
## Logical Overview
```plantuml
object Asset
object "Blend file" as BlendFile
object "Asset file" as AssetFile
object "Asset library" as AssetLibrary
object "Asset index file" as AssetIndexFile {
+ version: int
}
object "Asset catalog" as AssetCatalog
object "Asset representation" as AssetRepresentation
object "Data-block" as DataBlock {
+ marked as asset: bool
}
BlendFile <|.. AssetFile
note right: An blend file is an asset file\nwhen part of an asset library\nIt may or may not contain assets.
BlendFile *- DataBlock
Asset ..|> DataBlock
note left of Asset: An data-block is an asset\nwhen the data-block is marked as an asset.
AssetRepresentation -- Asset
AssetIndexFile *- AssetRepresentation
AssetLibrary o- AssetFile
Asset -o AssetCatalog
```
## Problem statement
Assets are stored in a blend file as part of an asset library. In order to browse through the assets of an asset library Blender has to parse each asset file in order to know which data-blocks have been tagged as assets.
Doing this each time when browsing assets and having large asset libraries extracting the assets can take a long time.
- Blender loads asset files sequentially in a background thread, which is a slow operation.
- Asset libraries can be located on network shares and version controlled repositories.
- Asset libraries can have a large number of files.
- Asset files can be large is size.
- Asset files can be compressed.
- Asset files may not actually contain assets.
## Solution
To reduce the time needed to browse asset libraries we want to extract the assets representation from the asset files and store them in the asset index. When the asset file isn't changed the asset index file would contain the required information needed to be able to reconstruct the [asset representation](https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Back_End#Asset_Representation).
### Requirements
**Each asset file has an asset index file.**
Each asset file has an asset index file even when the asset file has no assets. The reason is that asset files can be part of an asset library without containing assets. For example when an asset file contains materials which are linked from assets inside a different asset file, but those materials are not assets in them selves.
Not having an asset index file would parse the file each time.
**Icons of assets are not part of the asset index file.**
Icons compared to the rest of the asset representation can use a lot of disk space. Blender already has an icon manager that utilizes the OS to store and cache icons. Therefore we chose not to include the icon inside the asset representation.
**Empty asset files can be detected without opening the asset index file.**
Opening, reading and parsing asset index files has latency. To detect if there are actually asset representation stored inside the asset index file after parsing isn't efficient. By checking the file size on disk the asset indexer can already check if there are any asset representations inside the asset index file, removing the need for opening, reading and parsing the file.
:::info
The asset index is implemented as a [file indexer](https://hackmd.io/4PDN4EhISB2mwZFkIUtFFg) (`FileIndexerType`). Browsing assets shares logic with the file browser editor. File browser uses file indexers to read data from folders on a file-system, but also to list data-blocks located inside blend-files or data-blocks from asset libraries.
The usage of the File browser for asset listing and `FileIndexerType` is planned to be refactored. (ref #T999999)
:::
## Asset library loading process
```plantuml
@startuml name: Asset Indexer
AssetBrowser -> FileList
activate FileList
== Reading asset file index ==
FileList -> AssetFileIndexer: read_index(filepath_to_asset_file)
activate AssetFileIndexer
alt asset index file is up to date
FileList <-- AssetFileIndexer: NeedsUpdate
note right: Asset index file does not exist.\nIs asset file index created before asset file.\nAsset index file version is latest
end
alt asset index file is empty
FileList <-- AssetFileIndexer: EntriesLoaded
end
AssetFileIndexer -> AssetIndexFile : read_contents
activate AssetIndexFile
deactivate AssetIndexFile
AssetFileIndexer -> AssetIndexFile : extract_entries
activate AssetIndexFile
deactivate AssetIndexFile
FileList <-- AssetFileIndexer: EntriesLoaded
deactivate AssetFileIndexer
== Updating asset file index ==
alt NeedsUpdate
FileList -> FileList: read_entries_from_asset_file
activate FileList
deactivate FileList
FileList -> AssetFileIndexer: update_entries(filepath_to_asset_file, entries)
activate AssetFileIndexer
AssetFileIndexer -> AssetIndexFile: write_contents
activate AssetIndexFile
deactivate AssetIndexFile
FileList <-- AssetFileIndexer
deactivate AssetFileIndexer
end
deactivate AssetFileIndexer
AssetBrowser <-- FileList
deactivate FileList
@enduml
```
The sequence diagram shown above describes an overview control flow of the asset indexer. When the asset browser needs to know the assets representations that are inside an asset library it uses a `FileList`. The file list walks over the file inside the asset library and when an asset file is detected it invokes `read_index` on the `AssetFileIndexer`. `read_index` returns if the found asset representations (`EntriesLoaded`) or that the asset index file needs to be updated for the given asset file.
In the case that the asset file index requires an update, the file list would extract the asset representations from the asset file and invoke `update_index`. Update entries would create update or create the asset index file with the given asset representations.
## Asset Index
When browsing assets, asset indexes are created when they don't exist or are out of date. The asset browser does not access asset files directly, but uses asset indexes.
### Storage
The asset index files are stored in the [local cache directory](https://docs.blender.org/manual/en/latest/advanced/blender_directory_layout.html#local-cache-directory). Within the local cache directory the indexes are stores using the following pattern:
`/asset-library-indices/<asset-library-hash>/<asset-index-hash>_<asset_file>.index.json`.
`asset-library-hash`
: Hash of the absolute file path of the asset library.
`asset-index-hash`
: Hash of the absolute file path of the asset file.
`asset_file`
: Filename of the asset file. Not used by Blender, but is added for discoverability convenience.
### Content
```json
{
"version": <file version number>,
"entries": [{
"name": "<asset name>",
"catalog_id": "<catalog_id>",
"catalog_name": "<catalog_name>",
"description": "<description>",
"author": "<author>",
"tags": ["<tag>"],
"properties": [..]
}]
}
```
`version`
: the version of the asset index file. It is used to identify the structure of the content. Asset indexes with a different version than used by Blender would be regenerated. Blender 3.1-3.4 expect version attribute to be `1`.). Later versions might require to change it.
`entries`
: list of assets in the asset file related to the asset index.
`entries[].name`
: ID name of the asset, including its ID type prefix. (`OBSuzanne`).
`entries[].catalog_id`
: UUID of the catalog associated with the asset.
`entries[].catalog_name`
: Name of the catalog associated with the asset.
`entries[].description`
: Human readable description of the asset.
`entries[].author`
: Information about the author of the asset.
`entries[].tags`
: List of tags associated with this asset.
`entries[].properties`
: List of ID Properties associated with this asset.
## Performance
Performance is optimized for reading and parsing asset indexes.
- Asset libraries can have blend files that doesn’t contain any assets. To reduce opening and parsing an asset index that doesn’t contain any entries the size on disk of the asset index is checked. Asset indexes smaller than `MIN_FILE_SIZE_WITH_ENTRIES` would not be parsed and assumed to contain no entries.
- Asset indexes older than asset files are assumed to be out of date and should be regenerated. This can be determined without opening the asset index.
- Asset indexes of asset files on network shares are stored on your local system to reduce latency. Although this could depends on how your OS has been configured.
## Future expected changes
:::warning
To be added
:::
## References
- Technical overview of asset catalogs: `https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Catalogs`.
- The main source code is located at `source/blender/editors/asset/intern/asset_indexer.cc` and `source/blender/editors/asset/ED_asset_indexer.h`.
- End user documentation about assets library system `https://docs.blender.org/manual/en/latest/files/asset_libraries/introduction.html`, `https://docs.blender.org/manual/en/latest/files/asset_libraries/catalogs.html`