# Marlin artifacts management, delivery channels and marlinctl
In order to formalize and streamline releases - it's build automation, delivery to end user machines and finally process management at the end user machine, the following document describes the ideas and key implementations relevant to this release pipeline for use starting larvanet.
## Important concepts
- **Projects**: Project refer to a set of binary artifacts used in a specific configuration to allow implementation of a use case. For example. `gateway_near`, `relay_cosmos` are projects which consist of gateway + bridge / relay + abci bundled for specific use cases of setting up near node's gateway processes / cosmos node's relay processes.
- **Channels**: Release channels such as `public`, `beta`, `alpha` etc
- **Registry**: Registry refers to a set of metadata containing release information of various projects. A registry is defined by two parameters: `upstream_VCS_URL` and `upstream_VCS_BRANCH`. The head of the two is the current state of given registry. Different registries service different channels.
- **Runtime**: Specifies the runtime configuration of the system. Is generally of the form `OS-ARCH.PROCESSMANAGEMENT`. For example: `linux-amd64.supervisor` and `linux-amd64.systemd`. However, this parameter is project dependent and may not have general formats as described above.
- **Runner**: Implementation of runner interface tasked with heavy lifting of start / stop / configuration etc for given project version's runtime. Explained later.
- **Semver**: Semantic versioning - in our case we append `-chan.build` for non public channel. Described below.
- **S3 Buckets**: S3 buckets hold artifacts to be available for download by end users
- **CDN Service**: Provides a cached proxy for faster downloads globally. Put in front of S3
## Explanation
There will be four channels for artifact delivery (all follow semver):
1. **Public**: This channel consists of stable binaries available for public. Releases are of the form `X.Y.Z`. Registry that implements this channel is `https://github.com/marlinprotocol/releases.git` @ branch `public`. Hence, this is a protected branch on VCS with release policies attached. Build automation does not support writing builds directly to public bucket, only beta channel's release candidates can be upgraded to a public release.
2. **Beta**: Public facing unstable channel tasked with delivery of beta artifacts, release candidates and support releases. Releases are of the form `X.Y.Z-beta.U` where `U` is the build number. Registry that implements this channel is `https://github.com/marlinprotocol/releases.git` @ branch `beta`. Build automation allows writing builds directly to beta bucket. Relevant beta artifacts of form `X.Y.Z-beta.U` can later be "upgraded" to `X.Y.Z` public release.
3. **Alpha**: Development / Programmatic unstable channel tasked with deliver of artifacts to any upcoming regression test service etc. Currently not in use. Registry that implements this channel is `https://github.com/marlinprotocol/releases.git` @ branch `alpha`.
4. **Dev**: Local development channel to mock any of the other channels during development in localhost. Registry that implements this channel is `/home/<exampleuser>/examplereleases` @ branch `dev`.
Consequently `https://github.com/marlinprotocol/releases.git` always implements three branches: `public`, `beta`, `alpha`. Head of any of these branch at any given time hold the metadata the current state of their release channel.
In essence, schema for a registry looks as follows:
```
- projects
- beacon
- releases.json
- changelogs.md
- gateway_iris
- releases.json
- changelogs.md
- relay_near
- releases.json
- changelogs.md
...
```
Any VCS that implements this schema and plays well with git is a candidate for registry.
`releases.json` hold information in the following format:
```
{
"json_version": 1,
"data" : {
"0": {
"0": {
"1": {
"1": {
"time": "12 Dec 20 15:06 +0530",
"description": "Beta test for 0.0.1-beta.1",
"bundles": {
"linux-amd64.supervisor": {
"runner": "linux-amd64.supervisor.runner01",
"data": {
"gateway": "http://beta.artifacts.marlin.pro/projects/gateway_iris/0.0.1-beta.1/iris_gateway-0.0.1-beta.1-linux-amd64",
"gateway_checksum": "42e29d7b223de75400d0e05b58274c78",
"bridge": "http://beta.artifacts.marlin.pro/projects/gateway_iris/0.0.1-beta.1/iris_bridge-0.0.1-beta.1-linux-amd64",
"bridge_checksum": "243af76921feca3356a9a1024ef1dca1"
}
}
}
},
"2": {
"time": "13 Dec 20 15:06 +0530",
"description": "Beta test for 0.0.1-beta.2",
"bundles": {
"linux-amd64.supervisor": {
"runner": "linux-amd64.supervisor.runner01",
"data": {
"gateway": "http://beta.artifacts.marlin.pro/projects/gateway_iris/0.0.1-beta.1/iris_gateway-0.0.1-beta.1-linux-amd64",
"gateway_checksum": "42e29d7b223de75400d0e05b58274c78",
"bridge": "http://beta.artifacts.marlin.pro/projects/gateway_iris/0.0.1-beta.1/iris_bridge-0.0.1-beta.1-linux-amd64",
"bridge_checksum": "243af76921feca3356a9a1024ef1dca1"
}
}
}
}
}
}
}
}
}
```
We come to the following points hence:
1. Artifact URLs are described as `http://beta.artifacts.marlin.pro/...`. This is our cloudflare CDN in front of our `beta.artifacts.marlin.pro` s3 bucket hosted on AWS. Hence, users are routed through a CDN whenever they intend to download new binaries. **It is hence extremely important that we use the buckets as append only system. Changing existing artifacts in buckets creates complications relating to cache invalidation at CDN level**. With this, both `beta.artifacts.marlin.pro` and `public.artifacts.marlin.pro` s3 buckets will be enforced with append only policies in the future. In total, we will have three s3 buckets: `beta.artifacts.marlin.pro`, `alpha.artifacts.marlin.pro` and `public.artifacts.marlin.pro`. Two of these will be provided using cloudflare CDN: `beta` and `public`. Alpha since is internal does not use the CDN service.
2. Every release describes multiple runtimes and each runtime tags a runner. It is extremely important to understand this design. Please read below.
### Runtimes and runners
With every release, the information under "bundle" is as follows:
```
"bundles": {
"linux-amd64.supervisor": {
"runner": "linux-amd64.supervisor.runner01",
"data": {
"gateway": "URLX",
"gateway_checksum": "CHKX",
"bridge": "URLY",
"bridge_checksum": "CHKY"
}
},
"linux-amd64.systemd": {
"runner": "linux-amd64.systemd.runner01",
"data": {
"gateway": "URLX",
"gateway_checksum": "CHKX",
"bridge": "URLY",
"bridge_checksum": "CHKY"
}
},
"darwin-x64.supervisor": ...
...
}
```
It is immediately clear why releases will have multiple bundles for different runtimes - since build artifacts may be different for different architectures (in case they are similar - for example bundles `linux-amd64.supervisor` and `linux-amd64.systemd` should mostly ship same artifacts, urls can be reused for them).
Every release in turn describes information as follows:
```
"runtime" : {
"runner": <string>
"data": <runnerdata{}>
}
```
A runner is a piece of code that implements the following interface:
```
type Runner interface {
PreRunSanity() error
Download() error
Prepare() error
Create(runtimeArgs map[string]string) error
Destroy() error
PostRun() error
Status() error
Logs() error
}
```
This has been done to solve the following problem. Let's say at version `X.0.0` of `iris_gateway`, things were handled a certain way: Download two executables, create two supervisor processes, run the processes. Monitoring for process uptime also constitutes monitoring two supervisor processes.
Later at version `X+1.0.0` of `gateway_iris`, the two executables are merged into one and now, marlinctl only requires setting up one process, destroy one process etc.
These two needs to play well, since handling of `X+1.0.0` is very different to handling `X.0.0`. We can't update marlinctl's iris gateway existing handlers directly since if we now support the newer version, we won't be backwards compatible. This creates a trouble when user wants to downgrade to a lower version since then, we can't guarantee if that will run the way it should have run when the particular version was released. THIS IS UNDESIRABLE.
Runners are description of how a certain version is to be handled on the system. For example when a certain version of `gateway_iris` describes that for runtime `linux-amd64.supervisor`, the `runner` is `linux-amd64.supervisor.runner01`, it describes the implementation of handling logic that should come at play during runtime to handle this version. Hence, version `X.0.0` can be implemented using `linux-amd64.supervisor.runner01` runner for `linux-amd64.supervisor` which handles two executable / two process paradigm but newer version `X+1.0.0` can be implemented using `linux-amd64.supervisor.runner02` which describes one executable / one process paradigm. These runners are written into marlinctl and new marlinctl artifact gets released with any new additional runner.
**Note**: Runners are project specific. `relay_near`'s `linux-amd64.supervisor.runner01` is not the same as `gateway_iris`'s `linux-amd64.supervisor.runner01`. Both are independent and can be upgraded separately.
Over the long run hence, we may tend to see releases holding information such as following:
```
{
"json_version": 1,
"data" : {
"0": {
"0": {
"1": {
"1": {
"time": "12 Dec 20 15:06 +0530",
"description": "Beta test for 0.0.1-beta.1",
"bundles": {
"linux-amd64.supervisor": {
"runner": "linux-amd64.supervisor.runner01",
"data": ...
}
}
},
"2": {
"time": "13 Dec 20 15:06 +0530",
"description": "Beta test for 0.0.1-beta.2",
"bundles": {
"linux-amd64.supervisor": {
"runner": "linux-amd64.supervisor.runner02",
"data": ...
}
}
}
}
}
}
}
}
```
### marlinctl 2.0.0
marlinctl 2.0.0 (currently ctl2) marks the implementation of the above model. It consists of the following components:
1. marlinctl config: This describes the projects information along with global registries subsribed to. An example config is as follows:
```
config_version: 1
gateway_iris:
subscription:
- public
version: latest
storage: /root/.marlinctl/storage/projects/gateway_iris
runtime: linux-amd64.supervisor
forcedruntime: false
homedir: /root/.marlinctl/storage
marlinctl:
forcedruntime: false
runtime: linux-amd64
storage: /root/.marlinctl/storage/projects/marlinctl
subscription:
- public
version: latest
registries:
- branch: public
enabled: true
link: https://github.com/marlinprotocol/releases.git
local: /root/.marlinctl/registries/public
name: public
- branch: beta
enabled: true
link: https://github.com/marlinprotocol/releases.git
local: /root/.marlinctl/registries/beta
name: beta
- branch: alpha
enabled: false
link: https://github.com/marlinprotocol/releases.git
local: /root/.marlinctl/registries/alpha
name: alpha
- branch: dev
enabled: false
link: https://github.com/marlinprotocol/releases.git
local: /root/.marlinctl/registries/dev
name: dev
```
This is internally managed using `spf13/viper`. Project's channel subscription is supposed to be a subset of registries subscribed to (hence, project `gateway_iris` cannot subscribe to `alpha` since alpha registry is not enabled). For end users, by default `public` and `beta` registries are enabled globally and each project by default only subscribes to `public`
**Note**: While providing additional support to end users, we can ask them to explicitly enable beta channel for their project. This they can do using:
```
marlinctl gateway iris configure --enable-beta --runtime linux-amd64.supervisor --version 1.0.0-beta.3
```
Later, for coming back to release channel only, they can do
```
marlinctl gateway iris configure --runtime linux-amd64.supervisor --version latest
```
### Dynamic runtime-args
Every runner implements the runner interface:
```
type Runner interface {
PreRunSanity() error
Download() error
Prepare() error
Create(runtimeArgs map[string]string) error
Destroy() error
PostRun() error
Status() error
Logs() error
}
```
The method `Create(runtimeArgs map[string]string)` takes runtimeargs as map[string]string - offloading cli parsing at cobra level.
For example, when running
```
marlinctl gateway iris create --loglevel debug --skip-update-check
```
The output generated is as follows:
```
DEBUG Remote registeries pulled in 1.569560485s
DEBUG Project config found. Not creating defaults.
DEBUG Resolving "latest" project version to: 0.0.1-beta.1
INFO Successully verified gateway's integrity
INFO Successully verified bridge's integrity
INFO Keyfile information
Key Value
NodeId 5311a871331d37fe8cec9ed4e90a40afa4a568ac
INFO Running configuration
Key Value
GatewayProgram irisgateway
GatewayUser root
GatewayRunDir /
GatewayExecutablePath /root/.marlinctl/storage/projects/gateway_iris/0.0.1-beta.1/iris_gateway_linux-amd64
GatewayKeyfile /root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json
GatewayListenPortPeer 21900
GatewayMarlinIp 127.0.0.1
GatewayMarlinPort 21901
BridgeProgram irisbridge
BridgeUser root
BridgeRunDir /
BridgeExecutablePath /root/.marlinctl/storage/projects/gateway_iris/0.0.1-beta.1/iris_bridge_linux-amd64
BridgeBootstrapAddr 127.0.0.1:8002
INFO Trigerred bridge run
INFO Trigerred gateway run
INFO Waiting 10 seconds to poll for status
INFO Process status
Key Value
irisbridge RUNNING pid 2049759, uptime 0:00:10
irisgateway RUNNING pid 2049760, uptime 0:00:10
```
Any of the above running configurations can be changed using runtime-arguments as follows:
```
marlinctl gateway iris create --loglevel debug --skip-update-check --runtime-arguments "GatewayMarlinIp=89.91.22.214,BridgeRunDir=/root/.marlinctl,BridgeBootstrapAddr=127.0.0.1:9001"
```
to get
```
...
INFO Running configuration
Key Value
GatewayProgram irisgateway
GatewayUser root
GatewayRunDir /
GatewayExecutablePath /root/.marlinctl/storage/projects/gateway_iris/0.0.1-beta.1/iris_gateway_linux-amd64
GatewayKeyfile /root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json
GatewayListenPortPeer 21900
GatewayMarlinIp 89.91.22.214
GatewayMarlinPort 21901
BridgeProgram irisbridge
BridgeUser root
BridgeRunDir /root/.marlinctl
BridgeExecutablePath /root/.marlinctl/storage/projects/gateway_iris/0.0.1-beta.1/iris_bridge_linux-amd64
BridgeBootstrapAddr 127.0.0.1:9001
...
```
This offloading of runtime-arguments to runners is the right way to offload cli parsing burden across runners.
## Advanced usage
**Multi App support**:
Users may want to run multiple instances of the same project on their machines. This they can now do by providing `instance-id` flags. For example:
```
marlinctl gateway iris create --instance-id 001 --loglevel debug
```
This gives:
```
DEBUG Remote registeries pulled in 1.672679544s
DEBUG Latest marlinctl described upstream is current marlinctl's version. No updates to do.
DEBUG Project config found. Not creating defaults.
DEBUG Successully verified gateway's integrity
DEBUG Successully verified bridge's integrity
INFO Keyfile information
Key Value
NodeId 162ba52fd948c14cb2bf803178aaa85cbda93d8c
INFO Running configuration
Key Value
Runner linux-amd64.supervisor.runner01
Version 1.0.2
StartTime 01 Jan 21 10:06 +0530
GatewayProgram gatewayiris001
GatewayUser root
GatewayRunDir /
GatewayExecutablePath /root/.marlinctl/storage/projects/gateway_iris/1.0.2/gateway_iris_linux-amd64
GatewayKeyfile /root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json
GatewayListenPortPeer 21900
GatewayMarlinIp 127.0.0.1
GatewayMarlinPort 21901
BridgeProgram bridgeiris001
BridgeUser root
BridgeRunDir /
BridgeExecutablePath /root/.marlinctl/storage/projects/gateway_iris/1.0.2/bridge_iris_linux-amd64
BridgeBootstrapAddr 127.0.0.1:8002
DEBUG Trigerred bridge run
DEBUG Trigerred gateway run
INFO Waiting 10 seconds to poll for status
INFO Process status
Key Value
bridgeiris001 RUNNING pid 281776, uptime 0:00:11
gatewayiris001 RUNNING pid 281777, uptime 0:00:11
```
**Resource files**
Every instance run of a project has a resource file which keeps information of the runtime invocation in a json format. Here is the resource file for above run:
```
cat /root/.marlinctl/storage/projects/gateway_iris/common/project_gateway_iris_instance001.resource
```
This gives:
```
{
"Runner": "linux-amd64.supervisor.runner01",
"Version": "1.0.2",
"StartTime": "01 Jan 21 10:06 +0530",
"GatewayProgram": "gatewayiris001",
"GatewayUser": "root",
"GatewayRunDir": "/",
"GatewayExecutablePath": "/root/.marlinctl/storage/projects/gateway_iris/1.0.2/gateway_iris_linux-amd64",
"GatewayKeyfile": "/root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json",
"GatewayListenPortPeer": "21900",
"GatewayMarlinIp": "127.0.0.1",
"GatewayMarlinPort": "21901",
"BridgeProgram": "bridgeiris001",
"BridgeUser": "root",
"BridgeRunDir": "/",
"BridgeExecutablePath": "/root/.marlinctl/storage/projects/gateway_iris/1.0.2/bridge_iris_linux-amd64",
"BridgeBootstrapAddr": "127.0.0.1:8002"
}
```
Status command also shows information from resource file
```
marlinctl gateway iris status --instance-id 001 --loglevel debug
```
```
DEBUG Remote registeries pulled in 1.468580033s
DEBUG Latest marlinctl described upstream is current marlinctl's version. No updates to do.
DEBUG Project config found. Not creating defaults.
DEBUG Resource metadata: {linux-amd64.supervisor.runner01 1.0.2}
INFO Project configuration
Key Value
Subscription [public]
UpdatePolicy minor
CurrentVersion 1.0.2
Storage /root/.marlinctl/storage/projects/gateway_iris
Runtime linux-amd64.supervisor
ForcedRuntime false
AdditionalInfo map[]
INFO Keyfile information
Key Value
NodeId 162ba52fd948c14cb2bf803178aaa85cbda93d8c
INFO Resource information
Key Value
Runner linux-amd64.supervisor.runner01
Version 1.0.2
StartTime 01 Jan 21 10:06 +0530
GatewayProgram gatewayiris001
GatewayUser root
GatewayRunDir /
GatewayExecutablePath /root/.marlinctl/storage/projects/gateway_iris/1.0.2/gateway_iris_linux-amd64
GatewayKeyfile /root/.marlinctl/storage/projects/gateway_iris/common/keyfile.json
GatewayListenPortPeer 21900
GatewayMarlinIp 127.0.0.1
GatewayMarlinPort 21901
BridgeProgram bridgeiris001
BridgeUser root
BridgeRunDir /
BridgeExecutablePath /root/.marlinctl/storage/projects/gateway_iris/1.0.2/bridge_iris_linux-amd64
BridgeBootstrapAddr 127.0.0.1:8002
INFO Process status
Key Value
gatewayiris001 RUNNING pid 281777, uptime 0:03:50
bridgeiris001 RUNNING pid 281776, uptime 0:03:50
```
**UpdatePolicies, 0.0.0 and defualt runtime and updatepolicies**
Update policies have been enforced for semver in projects. Updates follows update policies which can be one of the four:
- major
- minor
- patch
- frozen
By default it will be set to `minor` for all projects. Marlinctl project defines the information now for default setting:
```
marlinctl:
additionalinfo:
defaultprojectruntime: linux-amd64.supervisor
defaultprojectupdatepolicy: minor
currentversion: 2.0.0
forcedruntime: false
runtime: linux-amd64
storage: /root/.marlinctl/storage/projects/marlinctl
subscription:
- public
updatepolicy: minor
```
Each project now has config as follows:
```
gateway_iris:
subscription:
- public
updatepolicy: minor
currentversion: 1.0.2
storage: /root/.marlinctl/storage/projects/gateway_iris
runtime: linux-amd64.supervisor
forcedruntime: false
additionalinfo: {}
```
Defaults come from additional info fields of marlinctl project
Explicit major upgrade: TODO
Sign in with Wallet
Connect another wallet