# Asset System: Data Storage, Reading & UI Access

## Current Design

The current Asset Browser design is based on the File Browser. This already has a system for data storage, reading and UI access: `FileList`

### About `FileList`

Each File Browser editor stores a `FileList`. It does a fair bit of heavy lifting. Its main responsibilities are:
* Concurrent, possibly recursive loading of files from disk (including .blend libraries).
* Reading of different kinds of "files" (general files, .blend libray data, current `Main`).
* Storing all necessary data on the read files (name, path, preview image, file type, ...)
* Lazy-cached, concurrent loading of preview images. (Previews around the current view rectangle are loaded into cache.)
* Consistently storing selection state.
* Filtering and sorting.

There is also a fair amount of code for the old asset engine design.

All of these features are good to have in general, we should not remove them.

### Its Problems

* `FileList` does too many things; there is no separation of concerns.\
On its own that already makes maintainance hard. For the asset system we need something well designed, or we'll quickly run into problems when designing further functionality, like the Python API.
* Code is very hard to follow and reason about.
* The asset system requires a storage that is independent of the File Browser.\
We need global access for Python and the asset views (UI template to show a mini asset browser in regular layouts, e.g. panels, menus, regions, etc).
* `FileList` does UI specific things that are already done elsewhere in the UI.\
E.g. the filtering and sorting it does are also done for `UIList`s, Outliners, animation channel lists, etc. This redundency is unnecessary overhead.
The asset view UI template uses `FileList` via an asset specific abstraction. So the redundancy is quite ugly here.

----

## Proposal

We do not need to start entirely from scratch. We should be able to pick apart `FileList` and re-build it into a new design. Here is how that could look.

### Separation of Concerns

The different responsibilities from above can be split into a few basic concerns:
* **Reading:** Advanced reading of different kinds of data.
* **Storing:** Storing the data as some abstraction (e.g. file or asset).
* **Viewing:** Provide a view into the data for the UI (includes preview image loading/caching, filtering, sorting, etc.)

These are the main concernes that should be separated clearly, with well defined interfaces in-between.

### Reading

`AbstractFileReader`? TBD.
*(A bit awkward to call assets "files", besides that it makes sense. Better naming idea welcome.)*.

*TODO. Basically: Recursive, concurrent loading of files from disk (including .blend libraries). Different kinds of files, so OOP style inheritance makes sense.*

In future we might want a file-reading implementation for asset caches too. These would be simple .json files with meta-data stored inside an asset library.

Could be exposed to Python somehow, so Add-ons can implement custom asset reading, e.g. via the web.

### Storing

Base class `AbstractFileList`.

The asset specific implementation would look like this:

class AssetList : public AbstractFileList {
  // ...
  // Iterators, queries to get meta-data like names, previews and `AssetData`,
  // etc.
};

There could be a separate asset list for each library. That way we can cache that data as we like. Note that we can also lazy-load parts of an asset library into the list.

We'll need some way to identify an asset library (also needed in DNA, so uses C here):

struct AssetLibraryReference {
  /* ... */
  /* Type information and custom library identifier(s). */
};

Sybren had the interesting idea of using URIs for this.

There would be a global storage for assets:

class AssetListStorage {
  /* Global map as storage of asset-lists, per library. */
  static AssetListMap global_storage_;
  
public:
  // ...
  // Functions to fetch libraries into storage. All functions `static`.
};

> **Note 1:**<br/>
> In principle, asset storage for external asset libraries could be kept alive over file reads.

> **Note 2:**<br/>
> The File Browser could use some global storage system as well. So that loading the same directory in multiple File Browsers will share data.

This essentially is a "model" in the model-view-controller pattern.

### Viewing

If the storage part represents the *model*, this part represents the *view*. Idea is a general abstraction: `uiModelView`. It gives a view into a (potentially big) data-set, i.e. a model. This could be useful for the rest of Blender too, but we can keep it very simple for the start.

For assets, there would be the following model-view:

class AssetView : public uiModelView {
  // ...
  // Methods for lazy-caching, filtering, and querying information about the 
  // model (e.g. the name and preview of an item).
}

UIs like the `UIList` or the File Browser can get their view onto data via the `uiModelView` interface. Once the data is fetched into such a view, they do not need to know about the exact implementation (e.g. if this is an asset list, a file list, an RNA collection, ...).

### Putting it all Together

Who is responsible for putting this all together?

Few things:
* An Add-on can register custom asset libraries, using an `AssetLibraryReference` (could be a URI).
* Other libraries are registered by Blender, as is already done.
* If the Asset Library is activated in the UI, the UI or editor asks the `AssetListStorage` to fetch the needed assets. For example the asset-view UI template does this before creating the list.
* The `AssetList` or `AssetListStorage` creates the appropriate reader and invokes it via the `AbstractFileReader` interface. The reading can be threaded for non-local data.
* The UI/editor requests an `AssetView` from the read `AssetList`. This way it gets a view into the data that it can work with for drawing, interaction, filtering, etc.
* Special attention needs to be put on data-updates. Especially local data needs to be represented in a consistent state, or things can go bad. E.g. deletion of local asset data, undo/redo, file reading (with and without UI), threaded loading of preview images, ... (See [3d706bb0f3dd: Do proper updates when local asset data changes](https://developer.blender.org/rB3d706bb0f3dd04f7f1d01dbe008463648a6f66f2)).

### The File Browser

The File Browser should be ported to using this new design as well. I.e. `FileList` should be entirely replaced by the new design.

## Workload

All of this sounds like a lot of work. Things can be kept very simple for the start though. We don't need rich APIs yet. This is just about thinking carefully about the general direction, which will have long-term impact. Once we know which direction to go, we can add just the bits necessary for what we need for the current project goals.