---
# System prepended metadata

title: 'Marlin artifacts management, delivery channels and marlinctl'

---

# Marlin artifacts management, delivery channels and marlinctl
In order to formalize and streamline releases - it's build automation, delivery to end user machines and finally process management at the end user machine, the following document describes the ideas and key implementations relevant to this release pipeline for use starting larvanet.

## Important concepts
- **Projects**: Project refer to a set of binary artifacts used in a specific configuration to allow implementation of a use case. For example. `gateway_near`, `relay_cosmos` are projects which consist of gateway + bridge / relay + abci bundled for specific use cases of setting up near node's gateway processes / cosmos node's relay processes. 
- **Channels**: Release channels such as `public`, `beta`, `alpha` etc
- **Registry**: Registry refers to a set of metadata containing release information of various projects. A registry is defined by two parameters: `upstream_VCS_URL` and `upstream_VCS_BRANCH`. The head of the two is the current state of given registry. Different registries service different channels.
- **Runtime**: Specifies the runtime configuration of the system. Is generally of the form `OS-ARCH.PROCESSMANAGEMENT`. For example: `linux-amd64.supervisor` and `linux-amd64.systemd`. However, this parameter is project dependent and may not have general formats as described above.
- **Runner**: Implementation of runner interface tasked with heavy lifting of start / stop / configuration etc for given project version's runtime. Explained later.
- **Semver**: Semantic versioning - in our case we append `-chan.build` for non public channel. Described below.
- **S3 Buckets**: S3 buckets hold artifacts to be available for download by end users
- **CDN Service**: Provides a cached proxy for faster downloads globally. Put in front of S3

## Explanation
There will be four channels for artifact delivery (all follow semver):
1. **Public**: This channel consists of stable binaries available for public. Releases are of the form `X.Y.Z`. Registry that implements this channel is `https://github.com/marlinprotocol/releases.git` @ branch `public`. Hence, this is a protected branch on VCS with release policies attached. Build automation does not support writing builds directly to public bucket, only beta channel's release candidates can be upgraded to a public release.
2. **Beta**: Public facing unstable channel tasked with delivery of beta artifacts, release candidates and support releases. Releases are of the form `X.Y.Z-beta.U` where `U` is the build number.  Registry that implements this channel is `https://github.com/marlinprotocol/releases.git` @ branch `beta`. Build automation allows writing builds directly to beta bucket. Relevant beta artifacts of form `X.Y.Z-beta.U` can later be "upgraded" to `X.Y.Z` public release.
3. **Alpha**: Development / Programmatic unstable channel tasked with deliver of artifacts to any upcoming regression test service etc. Currently not in use. Registry that implements this channel is `https://github.com/marlinprotocol/releases.git` @ branch `alpha`.
4. **Dev**: Local development channel to mock any of the other channels during development in localhost. Registry that implements this channel is `/home/<exampleuser>/examplereleases` @ branch `dev`.

Consequently `https://github.com/marlinprotocol/releases.git` always implements three branches: `public`, `beta`, `alpha`. Head of any of these branch at any given time hold the metadata the current state of their release channel.

In essence, schema for a registry looks as follows:
```
- projects
    - beacon
        - releases.json
        - changelogs.md
    - gateway_iris
        - releases.json
        - changelogs.md
    - relay_near
        - releases.json
        - changelogs.md
    ...
```
Any VCS that implements this schema and plays well with git is a candidate for registry.

`releases.json` hold information in the following format:
```
{
    "json_version": 1,
    "data" : {
        "0": {
            "0": {
                "1": {
                    "1": {
                        "time": "12 Dec 20 15:06 +0530",
                        "description": "Beta test for 0.0.1-beta.1",
                        "bundles": {
                            "linux-amd64.supervisor": {
                                "runner": "linux-amd64.supervisor.runner01",
                                "data": {
                                    "gateway": "http://beta.artifacts.marlin.pro/projects/gateway_iris/0.0.1-beta.1/iris_gateway-0.0.1-beta.1-linux-amd64",
                                    "gateway_checksum": "42e29d7b223de75400d0e05b58274c78",
                                    "bridge": "http://beta.artifacts.marlin.pro/projects/gateway_iris/0.0.1-beta.1/iris_bridge-0.0.1-beta.1-linux-amd64",
                                    "bridge_checksum": "243af76921feca3356a9a1024ef1dca1"
                                }
                            }
                        }
                    },
                    "2": {
                        "time": "13 Dec 20 15:06 +0530",
                        "description": "Beta test for 0.0.1-beta.2",
                        "bundles": {
                            "linux-amd64.supervisor": {
                                "runner": "linux-amd64.supervisor.runner01",
                                "data": {
                                    "gateway": "http://beta.artifacts.marlin.pro/projects/gateway_iris/0.0.1-beta.1/iris_gateway-0.0.1-beta.1-linux-amd64",
                                    "gateway_checksum": "42e29d7b223de75400d0e05b58274c78",
                                    "bridge": "http://beta.artifacts.marlin.pro/projects/gateway_iris/0.0.1-beta.1/iris_bridge-0.0.1-beta.1-linux-amd64",
                                    "bridge_checksum": "243af76921feca3356a9a1024ef1dca1"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```

We come to the following points hence:
1. Artifact URLs are described as `http://beta.artifacts.marlin.pro/...`. This is our cloudflare CDN in front of our `beta.artifacts.marlin.pro` s3 bucket hosted on AWS. Hence, users are routed through a CDN whenever they intend to download new binaries. **It is hence extremely important that we use the buckets as append only system. Changing existing artifacts in buckets creates complications relating to cache invalidation at CDN level**. With this, both `beta.artifacts.marlin.pro` and `public.artifacts.marlin.pro` s3 buckets will be enforced with append only policies in the future. In total, we will have three s3 buckets: `beta.artifacts.marlin.pro`, `alpha.artifacts.marlin.pro` and `public.artifacts.marlin.pro`. Two of these will be provided using cloudflare CDN: `beta` and `public`. Alpha since is internal does not use the CDN service.
2. Every release describes multiple runtimes and each runtime tags a runner. It is extremely important to understand this design. Please read below.

### Runtimes and runners
With every release, the information under "bundle" is as follows:
```
"bundles": {
    "linux-amd64.supervisor": {
        "runner": "linux-amd64.supervisor.runner01",
        "data": {
            "gateway": "URLX",
            "gateway_checksum": "CHKX",
            "bridge": "URLY",
            "bridge_checksum": "CHKY"
        }
    },
    "linux-amd64.systemd": {
        "runner": "linux-amd64.systemd.runner01",
        "data": {
            "gateway": "URLX",
            "gateway_checksum": "CHKX",
            "bridge": "URLY",
            "bridge_checksum": "CHKY"
        }
    },
    "darwin-x64.supervisor": ...
    ...
}
```
It is immediately clear why releases will have multiple bundles for different runtimes - since build artifacts may be different for different architectures (in case they are similar - for example bundles `linux-amd64.supervisor` and `linux-amd64.systemd` should mostly ship same artifacts, urls can be reused for them).

Every release in turn describes information as follows:
```
"runtime" : {
    "runner": <string>
    "data": <runnerdata{}>
}
```

A runner is a piece of code that implements the following interface:
```
type Runner interface {
	PreRunSanity() error
	Download() error
	Prepare() error
	Create(runtimeArgs map[string]string) error
	Destroy() error
	PostRun() error
	Status() error
	Logs() error
}
```

This has been done to solve the following problem. Let's say at version `X.0.0` of `iris_gateway`, things were handled a certain way: Download two executables, create two supervisor processes, run the processes. Monitoring for process uptime also constitutes monitoring two supervisor processes.

Later at version `X+1.0.0` of `gateway_iris`, the two executables are merged into one and now, marlinctl only requires setting up one process, destroy one process etc.

These two needs to play well, since handling of `X+1.0.0` is very different to handling `X.0.0`. We can't update marlinctl's iris gateway existing handlers directly since if we now support the newer version, we won't be backwards compatible. This creates a trouble when user wants to downgrade to a lower version since then, we can't guarantee if that will run the way it should have run when the particular version was released. THIS IS UNDESIRABLE.

Runners are description of how a certain version is to be handled on the system. For example when a certain version of `gateway_iris` describes that for runtime `linux-amd64.supervisor`, the `runner` is `linux-amd64.supervisor.runner01`, it describes the implementation of handling logic that should come at play during runtime to handle this version. Hence, version `X.0.0` can be implemented using `linux-amd64.supervisor.runner01` runner for `linux-amd64.supervisor` which handles two executable / two process paradigm but newer version `X+1.0.0` can be implemented using `linux-amd64.supervisor.runner02` which describes one executable / one process paradigm. These runners are written into marlinctl and new marlinctl artifact gets released with any new additional runner.

**Note**: Runners are project specific. `relay_near`'s `linux-amd64.supervisor.runner01` is not the same as `gateway_iris`'s `linux-amd64.supervisor.runner01`. Both are independent and can be upgraded separately.

Over the long run hence, we may tend to see releases holding information such as following:
```
{
    "json_version": 1,
    "data" : {
        "0": {
            "0": {
                "1": {
                    "1": {
                        "time": "12 Dec 20 15:06 +0530",
                        "description": "Beta test for 0.0.1-beta.1",
                        "bundles": {
                            "linux-amd64.supervisor": {
                                "runner": "linux-amd64.supervisor.runner01",
                                "data": ...
                            }
                        }
                    },
                    "2": {
                        "time": "13 Dec 20 15:06 +0530",
                        "description": "Beta test for 0.0.1-beta.2",
                        "bundles": {
                            "linux-amd64.supervisor": {
                                "runner": "linux-amd64.supervisor.runner02",
                                "data": ...
                            }
                        }
                    }
                }
            }
        }
    }
}
```

### marlinctl 2.0.0
marlinctl 2.0.0 (currently ctl2) marks the implementation of the above model. It consists of the following components:
1. marlinctl config: This describes the projects information along with global registries subsribed to. An example config is as follows:
```
config_version: 1
gateway_iris:
  subscription:
  - public
  version: latest
  storage: /root/.marlinctl/storage/projects/gateway_iris
  runtime: linux-amd64.supervisor
  forcedruntime: false
homedir: /root/.marlinctl/storage
marlinctl:
  forcedruntime: false
  runtime: linux-amd64
  storage: /root/.marlinctl/storage/projects/marlinctl
  subscription:
  - public
  version: latest
registries:
- branch: public
  enabled: true
  link: https://github.com/marlinprotocol/releases.git
  local: /root/.marlinctl/registries/public
  name: public
- branch: beta
  enabled: true
  link: https://github.com/marlinprotocol/releases.git
  local: /root/.marlinctl/registries/beta
  name: beta
- branch: alpha
  enabled: false
  link: https://github.com/marlinprotocol/releases.git
  local: /root/.marlinctl/registries/alpha
  name: alpha
- branch: dev
  enabled: false
  link: https://github.com/marlinprotocol/releases.git
  local: /root/.marlinctl/registries/dev
  name: dev
```
This is internally managed using `spf13/viper`. Project's channel subscription is supposed to be a subset of registries subscribed to (hence, project `gateway_iris` cannot subscribe to `alpha` since alpha registry is not enabled). For end users, by default `public` and `beta` registries are enabled globally and each project by default only subscribes to `public`

**Note**: While providing additional support to end users, we can ask them to explicitly enable beta channel for their project. This they can do using:
```
marlinctl gateway iris configure --enable-beta --runtime linux-amd64.supervisor --version 1.0.0-beta.3
```
Later, for coming back to release channel only, they can do
```
marlinctl gateway iris configure --runtime linux-amd64.supervisor --version latest
```

### Dynamic runtime-args
Every runner implements the runner interface:
```
type Runner interface {
	PreRunSanity() error
	Download() error
	Prepare() error
	Create(runtimeArgs map[string]string) error
	Destroy() error
	PostRun() error
	Status() error
	Logs() error
}
```
The method `Create(runtimeArgs map[string]string)` takes runtimeargs as map[string]string - offloading cli parsing at cobra level.

For example, when running
```
marlinctl gateway iris create --loglevel debug --skip-update-check
```
The output generated is as follows:
```
DEBUG Remote registeries pulled in 1.569560485s
DEBUG Project config found. Not creating defaults.
DEBUG Resolving "latest" project version to: 0.0.1-beta.1
INFO Successully verified gateway's integrity
INFO Successully verified bridge's integrity
INFO Keyfile information
 Key     Value                                    
 NodeId  5311a871331d37fe8cec9ed4e90a40afa4a568ac 
INFO Running configuration
 Key                    Value                                                                                
 GatewayProgram         irisgateway                                                                          
 GatewayUser            root                                                                                 
 GatewayRunDir          /                                                                                    
 GatewayExecutablePath  /root/.marlinctl/storage/projects/gateway_iris/0.0.1-beta.1/iris_gateway_linux-amd64 
 GatewayKeyfile         /root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json                   
 GatewayListenPortPeer  21900                                                                                
 GatewayMarlinIp        127.0.0.1                                                                            
 GatewayMarlinPort      21901                                                                                
 BridgeProgram          irisbridge                                                                           
 BridgeUser             root                                                                                 
 BridgeRunDir           /                                                                                    
 BridgeExecutablePath   /root/.marlinctl/storage/projects/gateway_iris/0.0.1-beta.1/iris_bridge_linux-amd64  
 BridgeBootstrapAddr    127.0.0.1:8002                                                                       
INFO Trigerred bridge run
INFO Trigerred gateway run
INFO Waiting 10 seconds to poll for status
INFO Process status
 Key          Value                                 
 irisbridge   RUNNING   pid 2049759, uptime 0:00:10 
 irisgateway  RUNNING   pid 2049760, uptime 0:00:10 
```

Any of the above running configurations can be changed using runtime-arguments as follows:

```
marlinctl gateway iris create --loglevel debug --skip-update-check --runtime-arguments "GatewayMarlinIp=89.91.22.214,BridgeRunDir=/root/.marlinctl,BridgeBootstrapAddr=127.0.0.1:9001"
```
to get
```
...
INFO Running configuration
 Key                    Value                                                                                
 GatewayProgram         irisgateway                                                                          
 GatewayUser            root                                                                                 
 GatewayRunDir          /                                                                                    
 GatewayExecutablePath  /root/.marlinctl/storage/projects/gateway_iris/0.0.1-beta.1/iris_gateway_linux-amd64 
 GatewayKeyfile         /root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json                   
 GatewayListenPortPeer  21900                                                                                
 GatewayMarlinIp        89.91.22.214                                                                         
 GatewayMarlinPort      21901                                                                                
 BridgeProgram          irisbridge                                                                           
 BridgeUser             root                                                                                 
 BridgeRunDir           /root/.marlinctl                                                                     
 BridgeExecutablePath   /root/.marlinctl/storage/projects/gateway_iris/0.0.1-beta.1/iris_bridge_linux-amd64  
 BridgeBootstrapAddr    127.0.0.1:9001  
...
```
This offloading of runtime-arguments to runners is the right way to offload cli parsing burden across runners.

## Advanced usage
**Multi App support**:
Users may want to run multiple instances of the same project on their machines. This they can now do by providing `instance-id` flags. For example:
```
marlinctl gateway iris create --instance-id 001 --loglevel debug
```
This gives:
```
DEBUG Remote registeries pulled in 1.672679544s
DEBUG Latest marlinctl described upstream is current marlinctl's version. No updates to do.
DEBUG Project config found. Not creating defaults.
DEBUG Successully verified gateway's integrity
DEBUG Successully verified bridge's integrity
INFO Keyfile information
 Key     Value                                    
 NodeId  162ba52fd948c14cb2bf803178aaa85cbda93d8c 
INFO Running configuration
 Key                    Value                                                                         
 Runner                 linux-amd64.supervisor.runner01                                               
 Version                1.0.2                                                                         
 StartTime              01 Jan 21 10:06 +0530                                                         
 GatewayProgram         gatewayiris001                                                                
 GatewayUser            root                                                                          
 GatewayRunDir          /                                                                             
 GatewayExecutablePath  /root/.marlinctl/storage/projects/gateway_iris/1.0.2/gateway_iris_linux-amd64 
 GatewayKeyfile         /root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json            
 GatewayListenPortPeer  21900                                                                         
 GatewayMarlinIp        127.0.0.1                                                                     
 GatewayMarlinPort      21901                                                                         
 BridgeProgram          bridgeiris001                                                                 
 BridgeUser             root                                                                          
 BridgeRunDir           /                                                                             
 BridgeExecutablePath   /root/.marlinctl/storage/projects/gateway_iris/1.0.2/bridge_iris_linux-amd64  
 BridgeBootstrapAddr    127.0.0.1:8002                                                                
DEBUG Trigerred bridge run
DEBUG Trigerred gateway run
INFO Waiting 10 seconds to poll for status
INFO Process status
 Key             Value                                
 bridgeiris001   RUNNING   pid 281776, uptime 0:00:11 
 gatewayiris001  RUNNING   pid 281777, uptime 0:00:11 
```

**Resource files**
Every instance run of a project has a resource file which keeps information of the runtime invocation in a json format. Here is the resource file for above run:
```
cat /root/.marlinctl/storage/projects/gateway_iris/common/project_gateway_iris_instance001.resource
```
This gives:
```
{
 "Runner": "linux-amd64.supervisor.runner01",
 "Version": "1.0.2",
 "StartTime": "01 Jan 21 10:06 +0530",
 "GatewayProgram": "gatewayiris001",
 "GatewayUser": "root",
 "GatewayRunDir": "/",
 "GatewayExecutablePath": "/root/.marlinctl/storage/projects/gateway_iris/1.0.2/gateway_iris_linux-amd64",
 "GatewayKeyfile": "/root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json",
 "GatewayListenPortPeer": "21900",
 "GatewayMarlinIp": "127.0.0.1",
 "GatewayMarlinPort": "21901",
 "BridgeProgram": "bridgeiris001",
 "BridgeUser": "root",
 "BridgeRunDir": "/",
 "BridgeExecutablePath": "/root/.marlinctl/storage/projects/gateway_iris/1.0.2/bridge_iris_linux-amd64",
 "BridgeBootstrapAddr": "127.0.0.1:8002"
}
```

Status command also shows information from resource file
```
marlinctl gateway iris status --instance-id 001 --loglevel debug
```

```
DEBUG Remote registeries pulled in 1.468580033s
DEBUG Latest marlinctl described upstream is current marlinctl's version. No updates to do.
DEBUG Project config found. Not creating defaults.
DEBUG Resource metadata: {linux-amd64.supervisor.runner01 1.0.2}
INFO Project configuration
 Key             Value                                          
 Subscription    [public]                                       
 UpdatePolicy    minor                                          
 CurrentVersion  1.0.2                                          
 Storage         /root/.marlinctl/storage/projects/gateway_iris 
 Runtime         linux-amd64.supervisor                         
 ForcedRuntime   false                                          
 AdditionalInfo  map[]                                          
INFO Keyfile information
 Key     Value                                    
 NodeId  162ba52fd948c14cb2bf803178aaa85cbda93d8c 
INFO Resource information
 Key                    Value                                                                         
 Runner                 linux-amd64.supervisor.runner01                                               
 Version                1.0.2                                                                         
 StartTime              01 Jan 21 10:06 +0530                                                         
 GatewayProgram         gatewayiris001                                                                
 GatewayUser            root                                                                          
 GatewayRunDir          /                                                                             
 GatewayExecutablePath  /root/.marlinctl/storage/projects/gateway_iris/1.0.2/gateway_iris_linux-amd64 
 GatewayKeyfile         /root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json            
 GatewayListenPortPeer  21900                                                                         
 GatewayMarlinIp        127.0.0.1                                                                     
 GatewayMarlinPort      21901                                                                         
 BridgeProgram          bridgeiris001                                                                 
 BridgeUser             root                                                                          
 BridgeRunDir           /                                                                             
 BridgeExecutablePath   /root/.marlinctl/storage/projects/gateway_iris/1.0.2/bridge_iris_linux-amd64  
 BridgeBootstrapAddr    127.0.0.1:8002                                                                
INFO Process status
 Key             Value                                
 gatewayiris001  RUNNING   pid 281777, uptime 0:03:50 
 bridgeiris001   RUNNING   pid 281776, uptime 0:03:50 
```

**UpdatePolicies, 0.0.0 and defualt runtime and updatepolicies**

Update policies have been enforced for semver in projects. Updates follows update policies which can be one of the four:

- major
- minor
- patch
- frozen

By default it will be set to `minor` for all projects. Marlinctl project defines the information now for default setting:
```
marlinctl:
  additionalinfo:
    defaultprojectruntime: linux-amd64.supervisor
    defaultprojectupdatepolicy: minor
  currentversion: 2.0.0
  forcedruntime: false
  runtime: linux-amd64
  storage: /root/.marlinctl/storage/projects/marlinctl
  subscription:
  - public
  updatepolicy: minor
```
Each project now has config as follows:
```
gateway_iris:
  subscription:
  - public
  updatepolicy: minor
  currentversion: 1.0.2
  storage: /root/.marlinctl/storage/projects/gateway_iris
  runtime: linux-amd64.supervisor
  forcedruntime: false
  additionalinfo: {}
```
Defaults come from additional info fields of marlinctl project

Explicit major upgrade: TODO