<!--
---
tags: fbc
title: Coordinating multiple veneers: veneergroups
---
-->
# Coordinating multiple veneers: veneergroups

Previously, we've talked about how a [basic veneer](https://hackmd.io/J6PQYDnLTfqYGLWpOs44aQ) for File Based Catalogs would looks like, as well as a [semver based veneer](https://hackmd.io/q3_f5fnlTqGB9hOCTqZTiw?view) for going further with automated generation of channels based on bundle versions.

This document details a first look at a veneergroup that can be used to automate the processing of multiple sets of veneers. This is useful for generating multiple versions of a File Based Catalog, which in turn is useful for managing multiple delivery pipelines.

# The veneergroup:
A veneergroup is simply a collection of sources from which catalogs can be generated, which can then be written to specified output locations. It can have multiple contributor entries, each having their own output and sources. The contributor entries are processed independent of each other, and the veneergroup reports the success or failure of each set independently. Any failing contributor entry will cause the veneergroup to fail, reporting a status on the failing entries.

The overall schema of the veneergroup would look like:

---
schema: olm.veneergroup
- contributor: <contributor-label>
  output:
    type: file
    file:
      path: <fully-qualified path to output file --- all directories along path must exist>
  sources:
    - type: exec
      exec:
        - command: <fully-qualified path to executable>
          args: [<... list of args for command ...>]
        ...
   post-process:
       command: <fully-qualified path to executable>
       args: [<... list of args for command ...>]
        

The fields of these contributor entries are as described below:
- **contributor**: The contributor provides a label to identify each entry in the veneergroup. This is useful for identifying the set of contributors to be executed and for reporting the states of each contributor when the veneergroup is applied. There may be one or more contributors, with unique identifiers. Each contributor defines one or more `sources` objects, one `output` object, and may include a single `post-process` object. Execution of contributors occurs in parallel.
- **sources**: Each individual veneer in the veneergroup mentioned can use different kinds of inputs to generate its final output FBC. Each such input acts as a source for the component FBC. These are typically set of executables run in sequence that ultimately output a File Based Catalog at stdout. Each executable is checked for its output status after a run, much like a piped set of commands. Failure of any source step will be reported on completion of that contributor and will not impact other contributors.
- **output**: The catalog generated by each contributor entry of the veneergroup can then be written to a location. This must be a file in a specified non-existent or empty directory. This ensures the veneergroup is idempotent across runs.

## use-cases include:
- indv operator, turning veneers into FBC along release-based rails
- consolidating FBCs into CoC

### One-to-one example:
Illustrating that we can have a discrete input/output for each contributor
*Note that each release pipeline has a specific version of opm it uses for generating its catalog.*

---
schema: olm.veneergroup
- contributor: 4.12
  sources:
    - type: exec
      exec: 
        - command: /pipeline/operator-framework/4.12/bin/opm
          args: ["alpha", "render-veneer", "semver", "--output=yaml", "/pipeline/assets/4.12/my-package-veneer.yaml"]
  output:
    type: file
    path: /pipeline/outputs/4.12/my-package-catalog/catalog.yaml
  post-process:
    command: /pipeline/operator-framework/4.12/bin/opm
    args: ["validate", "/pipeline/outputs/4.12/my-package-catalog/"]
    
- contributor: 4.11
  sources:
    - type: exec
      exec: 
        - command: /pipeline/operator-framework/4.11/bin/opm 
          args: ["alpha", "render-veneer", "basic", "--output=yaml", "/pipeline/assets/4.11/my-package-veneer.yaml"]
  output:
    type: file
    path: /pipeline/outputs/4.11/my-package-catalog/catalog.yaml
  post-process:
    command: /pipeline/operator-framework/4.11/bin/opm
    args: ["validate", "/pipeline/outputs/4.11/my-package-catalog/"]

### One-to-many example:
Demonstrating that a common asset may be processed by discrete opm instances to generate outputs for different releases

---
schema: olm.veneergroup
- contributor: 4.12
  sources:
    - type: exec
      exec: 
        - command: /pipeline/operator-framework/4.12/bin/opm
          args: ["alpha", "render-veneer", "semver", "--output=yaml", "/pipeline/assets/4.X/my-package-veneer.yaml"]
  post-process:
    command: /pipeline/operator-framework/4.12/bin/opm
    args: ["validate", "/pipeline/outputs/4.12/my-package-catalog/"]
  output:
    type: file
    path: /pipeline/outputs/4.12/my-package-catalog/catalog.yaml
    
- contributor: 4.11
  sources:
    - type: exec
      exec: 
        - command: /pipeline/operator-framework/4.11/bin/opm 
          args: ["alpha", "render-veneer", "semver", "--output=yaml", "/pipeline/assets/4.X/my-package-veneer.yaml"]
  post-process:
    command: /pipeline/operator-framework/4.11/bin/opm
    args: ["validate", "/pipeline/outputs/4.11/my-package-catalog/"]
  output:
    type: file
    path: /pipeline/outputs/4.11/my-package-catalog/catalog.yaml

When applying the veneergroup, we can either allow it to generate every contributor within its config file, or limit the generation to a specified set of contributors, identified by their labels.

This veneergroup then allows the user to:
- Manage multiple File-Based Catalog pipelines
- Execute subsets of pipelines as needed
- Handle catalog generation requiring multiple steps
- Generate output catalogs at a desired location
- Verify the outputs of generated File-based Catalogs
- Use different versions of executables (like opm) for different pipelines
- Ensure idempotent catalog generation, provided the inputs remain the same

### Example of using the VofV to accumulate multiple operator author indices in a CoC

---
schema: olm.veneergroup
- contributor: foo
  output:
    type: file
    path: /pipeline/outputs/4.12/my-package-catalog/foo/catalog.yaml
  sources:
    - type: exec
      exec: 
        - command: /bin/cat
          args: ["/pipeline/outputs/4.12/my-package-catalog/catalog.yaml"]
schema: olm.veneergroup
- contributor: bar
  output:
    type: file
    path: /pipeline/outputs/4.12/my-package-catalog/bar/catalog.yaml
  sources:
    - type: exec
      exec: 
        - command: /bin/cat
          args: ["/pipeline/outputs/4.12/my-package-catalog/catalog.yaml"]

[comment]: <Future work:>
[comment]: < templating for source/output/pre-processing>
[comment]: < allow specifying a directory for output location in future iterations>
[comment]: < have the output struct have multiple fields of which at exactly one must be populated ala kube volumes (eg file, dir, image)>