Catalogs Plan of Attack

Context: 2021-08-20 Asset Browser and 3.0 meeting.

This is a draft planning for tasks regarding the Catalogs.

UPDATE: This document is superseded by T90066: Design Catalog system

Design Considerations

C vs C++

The asset catalog system will be C++ with a C interface where necessary.

Since all the catalog information is runtime-only, there is no need for DNA.

File Format

A catalog definition file (CDF) is a plain text (UTF-8) file.

Empty lines are ignored.
Lines starting with # are ignored.
The file should start with a declaration of what it is (human-readable), and a version number indicating the file format revision (human- and machine-readable).

Any other line should contain a catalog definition:

CATALOG_ID posix/style/path

The CATALOG_ID may not contain spaces, as a space is used as separator between it and the path. The path can contain spaces.

Multiple definitions

A library could have zero, one, or multiple catalog definition files.

One CDF: simple configuration. All new catalogs are stored here.
Zero CDFs: unexpected configuration. Should not break Blender.
- Maybe show assets under a ${CATALOGID} (unknown catalog) label? Renaming that catalog should create its path. This could also work for other cases where unknown catalog IDs are used.
Multiple CDFs: complex configuration. Renames could influence multiple CDFs simultaneously. Changed catalogs should be written back to the file they came from.

Deduplication

Multiple definitions of a CATALOG_ID in one file should be avoided. A warning could be logged (on the terminal, for example) and the first occurrence of the catalog wins.

Defining the same catalog in multiple files is fine, as long as they don't confict. In case of conflict, the one defined in the top-most file wins, and the rest is synchronized when it changes (i.e. when a write is triggered).

File Locations

Multiple locations are supported:

Top level: ${ASSET_LIBRARY_ROOT}/blender-asset-library.cats.txt
Per blend file: blendfile-path-without-extension.cats.txt
Maaaaybe arbitrary locations? Like a blender-asset-library.cats.txt in each subdirectory? Will become relevant when subdirs are possible with the Asset Browser.

Big chunks to do

Decide on line endings

What happens when writing the file? Probably best to keep the line ending that's in use (to stay friendly with versioning systems).

First step: use platform-native line endings when writing.
Second step: remember line endings used, and use those when writing.

Decide how to keep track of which catalog is defined in which CDF

A catalog can be defined in multiple files, and when updated should be written to all those files. This can be done by keeping track of a set<AssetCatalogDefinitionFile *> in each catalog.

Decide UX for editing catalog IDs

The catalog paths are what users will be primarily interacting with. The catalog IDs allow for nifty tricks (like moving assets without updating their blend file), but do make things harder to explain/use. This should be clear in the UI.

Decide UX for creating new Catalogs

Users have to choose a catalog ID, which could default to the slugified last element of the catalog path.

Something (user or Blender) has to choose where to save the catalog info:

If there is a CDF for the current blend file, store there, otherwise
if there is a top-level CDF, store there, otherwise
if the current blend file is in an asset library, create a top-level CDF, otherwise
create a CDF for the current blend file.

Add Catalog ID to `AssetMetaData`

A symbolic identifier of which catalog path this asset is stored in.

typedef struct AssetMetaData {
  ...
  char catalog_id[64];
}

TBD: the max length. 64 was just chosen arbitrarily.

Add `AssetLibraryCatalogs` class

Manages catalogs of a single asset library.

Given a Catalog ID, can look up the path.
Can give the Catalog ID of a given path (f.e. when moving assets)
Can rename paths (keeping track of other paths to change as well, when the changed part is shared by other paths).
Can read CDFs.
Can write CDFs.
Maybe: can work as back-end for tree view.

TBD: this needs to be split up into smaller TODOs.

Reading, writing, tracking


using CatalogID = std::string;
using CatalogPath = std::string;
using CatalogPathComponent = std::string;
using CatalogFilePath = std::string;

class AssetCatalogDefinitionFile;

class AssetCatalogTreeNode {

  /* Pointer to the parent tree node. */
  AssetCatalogTreeNode *parent = nullptr;

  /* Child nodes, ordered by their name. */
  std::set<AssetCatalogTreeNode *> children;
  
  /* Catalogs that have this path. */
  Map<CatalogID, AssetCatalog *> catalogs;

  /* The path component that is associated with this catalog. Note that this may contain separators
   * when ancestors don't exist. This can happen when there are catalog definitions like this:
   *
   * PARENT       parent                        -> name = "parent"
   * THIS_CATALOG parent/nonexistant/thisone    -> name = "nonexistant/thisone"
   */
  CatalogPathComponent name;
  
  /* Set the catalog path to `parent->catalog_path() + separator + node_name` and update all child catalogs. */
  void set_catalog_name(const CatalogPathComponent &node_name);

  /* Return the full catalog path, defined as the name of this catalog prefixed by the full
   * catalog path of its parent and a separator. */
  std::string catalog_path() const;

};

class AssetCatalog {
  /* Symbolic ID, "THIS_CATALOG" in the above example. */
  CatalogID catalog_id;

  /* Return the full catalog path, defined as the name of this catalog prefixed by the full
   * catalog path of its parent and a separator. */
  std::string catalog_path

  /* Which definition files this catalog was defined in. Can be multiple, which means that multiple
   * files will have to be updated when this catalog is edited. */
  Set<AssetCatalogDefinitionFile *> defined_in;
};


class AssetCatalogDefinitionFile {
  CatalogFilePath file_path;
  Map<CatalogID, AssetCatalog *> catalogs;
  VectorSet<AssetCatalog *> root_catalogs;

  void write_to_disk() const;

  // Defined here instead of on AssetCatalog itself for reasons:
  // - Caller doesn't have to know whether this catalog is even defined in this file.
  // - Allows for keeping track of dirty files (i.e. where write_to_disk() has to
  //   actually do something).
  // - Allows for keeping the internal bookkeeping up to date.
  // - Allows for path changes to be reflected in more than one catalog.
  // No-op if this file doesn't have this catalog.
  void change_catalog_path(const CatalogID &catalog_id, const CatalogPath &new_catalog_path);
  void change_catalog_id(const CatalogID &catalog_id, const CatalogID &new_catalog_id);

  // Return nullptr if not found.
  AssetCatalog *find_catalog(const CatalogID &catalog_id);
};

TBD: internal data structure. What is written above is just one simple idea. It's probably better to have the internal structure stored in a tree, such that a rename of a path just requires moving some elements of the tree around. Might be over-engineered though, as the catalogs are meant to be human-manageable and thus there won't be thousands of them anyway.

Add Tree view of available catalogs

Make things collapsible, selectable, determines which assets are shown in the asset browser.

Allows renaming and moving of catalogs. These operations are the same from the CDF perspective, but are different operations for the user interface.

Add current catalog (ID or path) to asset browser

The asset browser should be able to show assets from a single catalog path + its children.