###### tags: `Design`

# High level controls for node assets

A proposal for making complex node trees customizable using node groups parameters.

Node assets that are supposed to be used as building blocks require detail knowledge to tweak their internals. Some parameters can be exposed on the modifier level, but extending a system requires a different approach, as will be demonstrated below.

This proposal describes a mechanism to allow users to specify node groups at the highest level of a node asset, which are then used inside the asset without requiring users to touch the asset internals themselves.

## The modifier stack for node assets and its limitations

Simon Thommes recently published the first set of hair hair assets to be shipped with Blender.

<iframe width="420" height="315" src="https://www.youtube.com/embed/gCQN5vNgHiI" title="How to Make Procedural Fur in Blender 3.5" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

Simon breaks down the hair workflow into separate modifiers. This relies a lot on the fact that modifiers can be applied as a single self-contained step one after the other.

*Modifier stack in Simon's hair assets*

Simulations won't be able to use this organizational trick so easily. That's because simulations contain multiple loops, nested inside each other, and the parts that users would be interested in modifying and swapping out are located within those loops.

[](https://i.imgur.com/j9YSNPW.png)
*Particle simulation prototype with a large simulation loop (click to enlarge)*

The most obvious loop is the simulation stepping itself, sandwiched between the _Simulation Input_ and _Simulation Output_ nodes. This cannot be split into multiple node groups which then each go into their own modifier. In addition to this outer loop there may also be inner loops, depending on the purpose and complexity of the simulation.
* A simulation may want to break the 1 frame steps into smaller substeps for greater accuracy, motion blur, etc.
* Constraint solvers typically work iteratively, meaning the solver has to perform several iterations in each time step for a good solution.
[<img src="https://i.imgur.com/iLWFq0h.png" width="400">](https://i.imgur.com/iLWFq0h.png)
*Constraint solver pseudo-loop, nested inside the simulation loop*

In addition to having customizable parts inside loops there may also be initialization steps _before_ or cleanup steps _after_ the loop, which have to match parts inside the loop body.
.

## Node groups for customizing assets

Let's use particle emitters as a case study. A particle system should support an arbitrary number of emitters. There are many different ways a particle emitter might be implemented:
- A continuous stream of particles with a variable rate
- Emission of a fixed particle count over some interval
- Burst of particles at a single point in time

We can imagine an emitter as a node group that simply outputs a point cloud as geometry. Users could create such a node group quite easily without knowing a lot about the rest of the particle system.


Users should not need to dive deep into some obscure particle system to insert a new emitter. Instead, we want to decouple the _declaration_ of such node groups from their _evaluation_.

## Function sockets

A "function socket" encapsulates a node tree as a data type. It can be passed around a tree like other data types. It is ultimately passed to an _Evaluate Function_ node, which takes the place of a regular node group. In place of the usual node group button it has a Function input socket.

The function can be generated by _binding_ a node tree: If a node asset exposes a Function socket, users can select a node group as a default value. This provides an straightforward way to use node groups as a configuration tool for assets:
1. Asset authors decide where customization is needed and provide function parameters for those cases (e.g. particle emitters).
2. An _Evaluate Function_ node is added where the outputs of such custom groups are used (e.g. the emission stage at the beginning of a simulation loop).
3. The Function input socket gets exposed through the node group interfaces, right to the top level of the asset.
4. Users can define one or more groups to use as emitters with the resulting node tree button.

## How it works (the short version)

What function sockets are **not**:
- Function sockets are single-valued (no fields). This should keep overhead from type checking and dispatching to a minimum. The _Evaluate Function_ node can still accept and return fields, it just means the function itself is the same for all elements of a field.
- Functions are never stored in blend files (serialized). There is no "function attribute". They are purely a runtime feature to customize a node tree.

The data type of a function socket is a _closure_: A combination of the node graph and a set of default values for inputs. The closure can be copied and passed through the node graph until it is evaluated.

flowchart
    G[Node Tree]:::data
    F[Default Bindings]:::data
    G --> E([bind]):::op
    F --> E
    E --> D[Closure]:::data
    D -->|passed through the node tree| B([evaluate]):::op
    B --> A[Result]:::data

    classDef data fill:#f96
    classDef op fill:#6bf

The inputs and outputs of the _Evaluate_ node are not directly related to any node tree. Users can freely chose which inputs to provide and which outputs are expected from an evaluation.

This means that a node tree can be unsuitable for a given evaluation! Specifically, inputs and outputs of the graph are optional: The Evaluate Function node can specify a subset of the graph inputs and any unspecified graph inputs will use a default or bound value. Likewise any unspecified graph outputs will simply be regarded as unused.

The evaluation node must, however, make sure that any specified outputs are provided by the graph it evaluates. Passing in a graph that does not have all required inputs or outputs constitutes a runtime error.

<!---

## Node groups as unit of customization

We want node group assets to work as building blocks, similar to modifiers: users should not have to connect nodes to be able to use assets. Jacques Lucke made a prototype for _node layers_ a while ago, which would simplify the top level UI ([#95472](https://projects.blender.org/blender/blender/issues/95472)). A similar idea is behind the material tree view in shader nodes (although shader nodes don't use a clearly defined node group interface yet, so the shader view contains a lot more parameters).

<p align="center">
<img src="https://i.imgur.com/UpGv461.png"/>
&nbsp; &nbsp; &nbsp; &nbsp;
<img alt="aaaa" src="https://i.imgur.com/JoFPgnm.png"/>
</p>

_Node Layers prototype and the Material tree view_

The advantages of such high-level systems are:

1. All important parameters visible on the same level in the UI.
In a simulation, customizable nodes may be hidden deep inside a complex node tree. But users should be able to specify and configure a feature at the top level.
1. Users can add and remove them with just a few clicks.
The new asset system provides a convenient way of adding nodes from a library.
1. Everything related to a feature is contained within the same node group.
If a node group is used in different places inside a simulation it may require different inputs and outputs. Declaring node group _contracts_ could be a way to ensure a node group has all necessary inputs and outputs.

## Node functions as data

Going a step further from node trees, there has been the idea of using functions as data types in nodes. In programming this is sometimes expressed as "functions are first class citizens", meaning that a function can be assigned to variables and parameters like regular data types.

[<img src="https://i.imgur.com/Llq5dBz.png" width="400">](https://i.imgur.com/Llq5dBz.png)
_Initial mockup by Jacques_

There are a number of implications if such a system were to be used:
- Because functions are handled as data types, the UI integration follows known concepts. Lists of functions can be supported as multi-value sockets (cf. "Join Geometry" node).
- Functions should always be "single" data types, like Geometry, instead of fields. The function itself should not vary per element of a domain (it's easy enough to do this kind of branching within a function).
- Functions could be _static_ or _dynamic_ types:
- Static function types would come with a fixed signature (inputs + outputs), and only same signature functions can be connected. This performs validity checks at "compile time" before the tree is evaluated. Doing this with current socket type definitions would be challenging though.
- Dynamic function types would represent a generic function and all type checking and error reporting happens during evaluation.
Some programming languages ([GDScript](https://docs.godotengine.org/en/stable/tutorials/scripting/gdscript/static_typing.html), C#) allow both static and dynamic types.
Because functions are not fields the overhead of dynamic type checking and dispatching is minimal. The implementation of dynamic function types would be much simpler than for static types. The downside is that errors could occur far away from the place where a function is actually defined, so reporting errors upstream is needed somehow.
-->

<!---
## Proposal: Node Tree Contracts

The goal of this concept is to decouple node trees from their immediate use.

In a regular node group the inputs and outputs of the group tree are immediately reflected by the node and connected to its surroundings. When node groups are specified at the top level but used deeper inside a system, there needs to be some agreement on what inputs and outputs are expected.

The system using a node tree can define a _contract_, and only node trees fulfilling the contract can be used for the purpose. For example:
- Modifier nodes must have a geometry input and output. This is an existing contract for top-level geometry nodes that is checked by the modifier. Using node trees without a geometry output will cause an error (geometry input is optional).
- A particle emitter must have a geometry output for points (other components would be ignored).
- Constraints are formulated using distance and gradient outputs. These are used by a constraint solver to find a solution satisfying all constraints in the system.

The definition of a contract consists simply of a set of input and output types and metadata (e.g. UI subtypes and limits). This is what node trees already do for their "interfaces", so it would be easy to verify that a tree fulfils a contract.
> :warning: The code for node tree interfaces is [here](https://projects.blender.org/blender/blender/src/commit/dc63f75837f254a0c5d0dcab39af562c1d1621e0/source/blender/makesdna/DNA_node_types.h#L577), but it should be noted that the C code leaves room for improvement. A distinct struct for `bNodeSocketType` lists instead of re-using `bNodeSocket` would avoid much confusion.

### Duck Typing or Nominative Typing?

There are two possible ways i can think of for ensuring a node group fulfils a contract:
1. [Duck typing](https://en.wikipedia.org/wiki/Duck_typing): The user must specify inputs and outputs matching the contract. If a node tree is used without the required sockets there will be an error message and the tree may not work.
2. Nominative Typing: The tree is declared to support a contract. Required sockets for the contract are automatically added to the node. Inside the node tree these sockets can be accessed through their own input output nodes (or should they be added to group input/output nodes?).

Duck typing has the advantage that it doesn't require many changes to node groups and only needs some verification and error handling. Modifier nodes are currently verified in this way. It puts the burden on the user to ensure a node group can be used for the respective contract.

Nominative typing might be easier to use because adding a tag to the tree is all that is needed to fulfil the contract. The respective input/output nodes are then provided automatically.

A hybrid approach could also be duck typing with additional support operators, such as an operator to "fix up" sockets for a given contract.

### Defining node contracts

In the beginning there would only be built-in contracts defined in C/C++ code, such as
* Modifier nodes
* Operator/Brush nodes
* Particle Emitters, Renderers, etc.
* Force fields for physics
* Constraint functions

Eventually users may be able to define contracts in nodes themselves. The details need to be worked out, but it could work like this:
1. User creates a node group, which is a placeholder for a user-provided datablock.
2. The node group is marked as a _template_. This defines the contract through the inputs and outputs of the template node tree.
3. The template itself may or may not implement useful behavior, but it could a reasonable default implementation.
4. A top-level node tree containing such a template node group exposes a tree selection button on its instances. This is where users can set the actual implementation of the contract.

Many examples mentioned before (e.g. particle emitters) use not just one but an arbitrary number of node groups for a contract. How this might work on a node level is unclear and requires further design, which is why focussing first on built-in contracts should be easier.

### Programming Language Equivalents

The contract concept can be found in programming languages in different forms:
- [C++ _templates_](https://en.cppreference.com/w/cpp/language/templates) also allow customization of a class based on compile-time parameters. There is no checking of contracts other than compile time errors, so users are responsible for passing in correct "duck types".
Virtual functions are also similar, but node group contracts would be resolved at "compile time" so there is no dynamic dispatch. We also don't have any form of inheritance, so function overrides are not a useful concept for nodes.
- [C# _generics_](https://learn.microsoft.com/en-us/dotnet/csharp/fundamentals/types/generics) serve the same purpose. They can also have [_constraints_](https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/types#845-satisfying-constraints) to make sure a generic parameter fulfils certain conditions. These can be much more detailed than what we need for nodes, where only matching output types are required.
-->