Proposal to Simplify the OCM library

Context
Proposal
2.1 Core
2.2 Providers
2.3 Structure
2.4 Examples
Consequences

Context

The current state of the OCM library poses significant challenges to developers. The integration process is cumbersome, primarily due to the complex nature of the codebase and the extensive list of dependencies. The library has over 65 direct dependencies, making the setup and maintenance arduous. The opaque relationship between spec concepts and their corresponding implementations in code further complicates the matter. Moreover, incremental adoption of features is nearly impossible under the current structure.

Project maintenance is far from efficient, a reflection of the inherent complexities and extensive dependencies. We are witnessing a low contributor engagement, and the code review process is a daunting task. The size of the changesets is often overwhelming, making thorough review and timely feedback an uphill battle.

Addressing these issues is paramount. This ADR aims to restructure the OCM library to enhance understandability, maintainability, and contribute-ability. Our objectives are concise:

Simplify the Codebase: Refactor and reorganize the code to improve readability and comprehension. The spec concepts should align intuitively with the code, making navigation and understanding seamless.
Reduce Dependencies: Streamline the number of direct dependencies. A leaner, more manageable list will facilitate easier integration and maintenance, aligning with Go's philosophy of simplicity and efficiency.
Facilitate Incremental Adoption: The restructured library should support the gradual integration of features, allowing developers to adopt and implement specific elements effortlessly.
Enhance Contribution Experience: By simplifying the codebase and reducing dependencies, we aim to lower the entry barrier for contributors. Smaller, more manageable PRs will make the code review process more efficient and inviting.

We are dedicated to transforming OCM into a toolset that is not only powerful but also user-friendly and maintainable.

Prior Art

The OCM library could glean insights from the structure and simplicity of the crane project, a renowned tool for handling OCI images, underscored by its widespread adoption. The project’s architecture is a confluence of the OCI Spec, the google/go-containerregistry Go library, and crane itself, each playing a distinct yet complementary role.

OCI Spec: It lays the foundation, offering types that adhere strictly to the OCI specifications.
google/go-containerregistry: This library, building atop the OCI Spec, outlines clear interfaces and mechanisms to facilitate interactions with OCI images.
crane: It harnesses the google/go-containerregistry, offering a user-friendly tool that simplifies the manipulation of OCI images.

The elegance of the google/go-containerregistry lies in its simplicity. The codebase is structured with clarity, making it a breeze for developers to discern the operations it supports. Its focused scope and minimized dependencies - numbering around 13 - are instrumental in its wide adoption. This streamlined approach not only simplifies integration but also enhances the library’s appeal, as evidenced by its use in over 4,984 public projects.

In reimagining the OCM library, adopting a similar ethos - one of reduced complexity, clear structuring, and minimized dependencies - could be the key to unlocking enhanced usability, maintenance, and integration.

Proposal

To enhance the Open Component Model (OCM) project, we focus on enabling easier incremental adoption and simplifying the path from engaging with the OCM specification to utilizing the library.

Core

Three core strategies are proposed:

Embed OCM Golang Types in the OCM-Spec:
Incorporating OCM Golang types directly into the OCM-spec allows developers to immediately use OCM types in their projects, reducing the integration complexity.
Simplify the Core Library:
The core library should be stripped down and focused strictly on the concepts outlined in the specification, excluding additional features like configuration and plugins. The introduction of providers will help implement technology-specific functionalities without complicating the core.
Separate Advanced and Specific Requirements:
Backward compatibility and specific requirements should be addressed separately, ensuring that the core OCM library remains uncluttered and generic.

Providers

The concept of providers is akin to the modular approach seen in Terraform or Crossplane. Providers can offer technology-specific functionalities while ensuring the core library remains lightweight and focused.

Key attributes of providers include:

Modular Design: Each provider is a distinct Go module, allowing for specific versioning and integration.
Access Methods: Providers support varied technology types and facilitate streamlined access to diverse objects.
Repository Features: Providers are equipped to store objects in different repositories, enhancing flexibility.

Providers must fulfil the following criteria:

A struct type satisfying the Access interface is essential for a provider to facilitate byte access.
Access types offered by providers need to be registered with the core library.
Although repositories aren’t required to register with the core library, they must implement the library’s Repository interface.
Providers with repository support must ensure both read and write access to stored blobs.

Structure

The proposed core-library structure would look similar to the following:

api/
├── v2/
│   ├── provider/       # provider interface and registration functions
│   ├── mutate/         # functionality for performing operations on elements
│   ├── build/          # enables building components from scratch
│   ├── query/          # generic querying functions for looking up components & resources
│   ├── reference.go    # the .go files below all define interfaces 
│   ├── descriptor.go   # that capture behaviors described in the OCM spec
│   ├── repository.go
│   ├── signature.go
│   ├── access.go
│   ├── component.go
│   ├── resource.go
│   └── source.go
├── go.mod
└── go.sum

Providers could have the following structure (note how each provider is a dedicated go module):

providers/
├── helm/           # helm provider, note: only access is provided
│   ├── access.go
│   ├── chart.go
│   ├── encoding.go
│   ├── go.mod
│   └── go.sum
├── oci/            # oci provider, note: repository implementation is also provided
│   ├── repository.go
│   ├── component.go
│   ├── access.go
│   ├── image.go
│   ├── encoding.go
│   ├── go.mod
│   └── go.sum
└── filesystem/     # filesystem provider, note: repository implementation is also provided
    ├── repository.go
    ├── component.go
    ├── access.go
    ├── encoding.go
    ├── go.mod
    └── go.sum

Providers enable consolidated and logical grouping of functionality in a way that encapsulates volatility (dependencies) and clarifies the available functionality.

By simplifying the core library and introducing modular providers, we aim to make the OCM more accessible, easier to maintain, and straightforward to adopt.

Examples

For all the following examples the source is available in context here: https://github.com/phoban01/ocm-v2-poc/cmd/

(note: for the purposes of the proof-of-concept the ocm types have been included in the api.)

1. Simple component stored on the filesystem

In the following example we create a component that contains a single resource. The resource data is accessed using the filesystem provider.

The component is written to disk using the filesystem provider's Repository implementation.

package main

import (
	"log"

	"github.com/phoban01/ocm-spec/types"
	"github.com/phoban01/ocm-v2/api/v2/build"
	"github.com/phoban01/ocm-v2/api/v2/mutate"
	"github.com/phoban01/ocm-v2/providers/filesystem"
)

func main() {
	// define metadata for the resource
	meta := types.ObjectMeta{
		Name: "config",
		Type: types.ResourceType("file"),
	}

	// create an access method for a file on disk
	// notice the filesystem provider helper methods to access resources
	// filesystem.ReadFile returns v2.Access
	access, err := filesystem.ReadFile("config.yaml", filesystem.WithMediaType("application/x-yaml"))
	if err != nil {
		log.Fatal(err)
	}

	// build the config resource using the metadata and access method
	config := build.NewResource(meta, access)

	// create the component
	cmp := build.New("ocm.software/test", "v1.0.0", "acme.org")

	// add resources to the component using the mutate package
	cmp = mutate.WithResources(cmp, config)

	// setup the repository using the filesystem provider
	repo, err := filesystem.Repository("./transport-archive")
	if err != nil {
		log.Fatal(err)
	}

	// write the component to the repository
	if err := repo.Write(cmp); err != nil {
		log.Fatal(err)
	}
}

Here is the same code flow depicted as a diagram:

2. Build a component with multiple resources & write to OCI Repository

The following example builds a component with three resources:

configuration file
oci image
helm chart

The component is written to an OCI Repository where the configuration file and Helm Chart are stored as local blobs.

package main

import (
	"fmt"
	"log"

	v2 "github.com/phoban01/ocm-v2/api/v2"
	"github.com/phoban01/ocm-v2/api/v2/build"
	"github.com/phoban01/ocm-v2/api/v2/mutate"
	"github.com/phoban01/ocm-v2/api/v2/types"
	"github.com/phoban01/ocm-v2/providers/filesystem"
	"github.com/phoban01/ocm-v2/providers/helm"
	"github.com/phoban01/ocm-v2/providers/oci"
)

func main() {
	config, err := NewFileResource("config", "config.yaml", "application/x-yaml")
	if err != nil {
		log.Fatal(err)
	}

	image, err := NewImageResource("nginx", "docker.io/nginx", "1.25.2")
	if err != nil {
		log.Fatal(err)
	}

	chart, err := NewChartResource("chart", "nginx-stable/nginx-ingress", "0.17.1")
	if err != nil {
		log.Fatal(err)
	}

	resources := []v2.Resource{config, image, chart}

	// create a new component
	cmp := build.New("ocm.software/v2/server", "v1.0.0", "acme.org")

	// add the resources to the component
	cmp = mutate.WithResources(cmp, resources...)

	// setup the repository (replace "ghcr.io/phoban01" with a repository you have access to)
	repo, err := oci.Repository("ghcr.io/phoban01")
	if err != nil {
		log.Fatal(err)
	}

	// write the component to the archive
	if err := repo.Write(cmp); err != nil {
		log.Fatal(err)
	}
}

func NewFileResource(name, path, mediaType string) (v2.Resource, error) {
	meta := types.ObjectMeta{
		Name: name,
		Type: types.Blob,
		Labels: map[string]string{
			"ocm.software/filename": path,
		},
	}

	access, err := filesystem.ReadFile(path, filesystem.WithMediaType(mediaType))
	if err != nil {
		return nil, err
	}

	return build.NewResource(meta, access), nil
}

func NewImageResource(name, ref, version string) (v2.Resource, error) {
	meta := types.ObjectMeta{
		Name:    name,
		Type:    types.OCIImage,
		Version: version,
	}

	access, err := oci.FromImage(fmt.Sprintf("%s:%s", ref, version))
	if err != nil {
		return nil, err
	}

	return build.NewResource(meta, access, build.Deferrable(true)), nil
}

func NewChartResource(name, ref, version string) (v2.Resource, error) {
	meta := types.ObjectMeta{
		Name:    name,
		Type:    types.HelmChart,
		Version: version,
	}

	access, err := helm.FromChart(ref, version)
	if err != nil {
		return nil, err
	}

	return build.NewResource(meta, access), nil
}

3. View Component Descriptor

The following example retrieves the component descriptor from and OCI repository and prints it to stdout:

package main

import (
	"encoding/json"
	"fmt"
	"log"

	"github.com/phoban01/ocm-v2/providers/oci"
)

func main() {
	repo, err := oci.Repository("ghcr.io/phoban01")
	if err != nil {
		log.Fatal(err)
	}

	cmp, err := repo.Get("ocm.software/v2/server", "v1.0.0")
	if err != nil {
		log.Fatal(err)
	}

	desc, err := cmp.Descriptor()
	if err != nil {
		log.Fatal(err)
	}

	out, err := json.MarshalIndent(desc, " ", " ")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(string(out))
}

4. List resources

The following example retrieves a component from an OCI repository and prints the resource metadata:

package main

import (
	"fmt"
	"log"

	_ "github.com/phoban01/ocm-v2/providers/helm"
	"github.com/phoban01/ocm-v2/providers/oci"
)

func main() {
	repo, err := oci.Repository("ghcr.io/phoban01")
	if err != nil {
		log.Fatal(err)
	}

	cmp, err := repo.Get("ocm.software/v2/server", "v1.0.0")
	if err != nil {
		log.Fatal(err)
	}

	res, err := cmp.Resources()
	if err != nil {
		log.Fatal(err)
	}

	var table [][]string
	table = append(table, []string{"TYPE", "NAME", "MEDIA TYPE", "DIGEST"})
	for _, v := range res {
		dig, err := v.Digest()
		if err != nil {
			log.Fatal(err)
		}

		acc, err := v.Access()
		if err != nil {
			log.Fatal(err)
		}
		row := []string{string(v.Type()), v.Name(), acc.MediaType(), dig.Value}
		table = append(table, row)
	}

	printTable(table)
}

func printTable(data [][]string) {
	if len(data) == 0 {
		fmt.Println("Table is empty")
		return
	}

	// Calculate the maximum width for each column
	colWidths := make([]int, len(data[0]))
	for _, row := range data {
		for i, cell := range row {
			if len(cell) > colWidths[i] {
				colWidths[i] = len(cell)
			}
		}
	}

	// Print the table header
	for i, cell := range data[0] {
		fmt.Printf("%-*s", colWidths[i]+2, cell) // +2 for padding
	}
	fmt.Println()

	// Print the table data
	for _, row := range data[1:] {
		for i, cell := range row {
			fmt.Printf("%-*s", colWidths[i]+2, cell) // +2 for padding
		}
		fmt.Println()
	}
}

5. Read resource data

This example retrieves the bytes for the configuration resource from an OCI repository and prints to standard out:

package main

import (
	"fmt"
	"io"
	"log"

	"github.com/phoban01/ocm-v2/api/v2/query"
	"github.com/phoban01/ocm-v2/api/v2/types"
	"github.com/phoban01/ocm-v2/providers/oci"
)

func main() {
	repo, err := oci.Repository("ghcr.io/phoban01")
	if err != nil {
		log.Fatal(err)
	}

	cmp, err := repo.Get("ocm.software/v2/server", "v1.0.0")
	if err != nil {
		log.Fatal(err)
	}

	config, err := query.GetResourceData(cmp, types.ObjectMeta{
		Name: "config",
		Type: types.Blob,
	})

	buffer := make([]byte, 4096)
	for {
		n, err := config.Read(buffer)
		if err != nil && err != io.EOF {
			fmt.Println("Error:", err)
			return
		}

		if n == 0 {
			break
		}

		fmt.Print(string(buffer[:n]))
	}
}

6. Transfer component between repositories

This example transfers a component from a remote OCI repository to the local filesystem:

package main

import (
	"log"

	"github.com/phoban01/ocm-v2/providers/filesystem"
	"github.com/phoban01/ocm-v2/providers/oci"
)

func main() {
	repo, err := oci.Repository("ghcr.io/phoban01")
	if err != nil {
		log.Fatal(err)
	}

	cmp, err := repo.Get("ocm.software/v2/server", "v1.0.0")
	if err != nil {
		log.Fatal(err)
	}

	archive, err := filesystem.Repository("local-copy")
	if err != nil {
		log.Fatal(err)
	}

	if err := archive.Write(cmp); err != nil {
		log.Fatal(err)
	}
}

7. Use GitHub as an OCM Repository

This example demonstrates how GitHub can be used as a storage repository for components. Note: the provider implementation is currently very limited and likely cannot handle anything more than simple text files.

package main

import (
	"log"

	"github.com/phoban01/ocm-v2/api/v2/build"
	"github.com/phoban01/ocm-v2/api/v2/mutate"
	"github.com/phoban01/ocm-v2/api/v2/types"
	"github.com/phoban01/ocm-v2/providers/filesystem"
	"github.com/phoban01/ocm-v2/providers/github"
)

func main() {
	// define metadata for the resource
	meta := types.ObjectMeta{
		Name: "config",
		Type: types.ResourceType("file"),
	}

	// create an access method for a file on disk
	// notice the filesystem provider helper methods to access resources
	// filesystem.ReadFile returns v2.Access
	access, err := filesystem.ReadFile(
		"config.yaml",
		filesystem.WithMediaType("application/x-yaml"),
	)
	if err != nil {
		log.Fatal(err)
	}

	// build the config resource using the metadata and access method
	config := build.NewResource(meta, access)

	// create the component
	cmp := build.New("ocm.software/test", "v1.0.0", "acme.org")

	// add resources to the component using the mutate package
	cmp = mutate.WithResources(cmp, config)

	// setup the repository using the github provider
	// arguments are owner and repository
	repo, err := github.Repository("phoban01", "ocm-github-repository")
	if err != nil {
		log.Fatal(err)
	}

	// write the component to the repository
	if err := repo.Write(cmp); err != nil {
		log.Fatal(err)
	}
}

Consequences

Providers Isolation
The introduction of providers will segregate technology-specific code from the core library. This change will result in the reduction of the core library size.
OCM Type Management
OCM types should be defined and versioned as part of the ocm-spec, and published as a go-module, so that when the spec changes so do the types.
Library Repository
Library code should live in a dedicated repository such as ocm-lib or ocm-core and consume types from ocm-spec.
OCM CLI Adaptation
The OCM CLI can follow an incremental migration path to the new library.
Provider Versioning
Providers will be versioned using Go modules and it may be helpful to track this information as part of the component descriptor.
Core Library Simplification
Configuration, secrets, substitution, contexts etc… are not part of the core library and must be added by the consuming client.
Descriptor Access Efficiency
In order to read a component descriptor from a repository it is only neccessary to import the provider for the specific repository type. This feature could be useful for reporting tools that do not need access to the resource data.
Provider Creation Process
The adoption of code-generation tools could reduce boilerplate requirements in provider creation, but the practicality and efficiency gains will be evaluated upon implementation.
To be addressed
In an effort to reduce the scope of this initial phase the poc has not considered in depth digesting, signing and validation. However it is hopefully clear that these aspects will be quite compatible with the proposed design. For example, the mutate package enables the core library to enforce constraints at various points in the component management lifecycle. Likewise implementing the MarshalJSON interface for the Descriptor type enables valdiation of the component descriptor prior to serialization and storage.

Piaras Hoban

2023/10/16 09:48:55

Comment from Ilya: > One question I have is where does`context.Context` go? I would imagine some of these functions should have it. I think crane hid contexts because they didn't want to break the API, but for a new design, I'd much rather see explicit context arguments. (Edited)

2023/10/16 09:53:56

I agree, we should have explicit context args. We also need to standardize the auth config settings. We will update the proof-of-concept with these items this week.

Table of contents