# Integrating a CI system using Radicle Webhooks
In this document we present all the required steps to integrate a 3rd party CI system with Radicle, using the provided outgoing webhooks.
Radicle is a sovereign peer-to-peer network for code collaboration, built on top of Git. In order to add the desired webhook functionality we have to run the Radicle CI Broker alongside the radicle node. Radicle CI Broker is responsible for grabbing any events that the radicle node generates and for each event it invokes any registered adapter. In our case the adapter will be the Radicle Webhooks Adapter that will invoke any registered outgoing webooks.
The second radicle node is required so that the first one will receive any updates from the network and then generate events so the broker will spawn any registered adapter.
> Making changes to a node wont generate any events to the same node.
## Requirements
The setup requires a radicle node that will be running alongside the radicle CI Broker and the Radicle CI Broker that will invoke the webhooks adapter.
Another radicle node is required with network access to the previous one (through private, public IP - or even in the same host) that will make changes to a repository. This way the first node will generate events about the changes that the broker will grab and invoke the webhook adapter.
### Radicle
Radicle must be set up following one of the official installation methodologies from: https://radicle.xyz/#try. After the radicle installation the `rad auth` command must be executed. More details about rad auth can be found [here](https://app.radicle.xyz/nodes/seed.radicle.xyz/rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5/tree/radicle-cli/examples/rad-auth.md).
After this you will be able to start you node by running
`rad node start`. At this point we all set!
You can run `radicle-httpd` and `rad web` so that you can access your radicle node through the web interface. `radicle-httpd` provides an HTTP API for managing the radicle node. `rad web` authenticates the radicle node using the `radicle-httpd` towards https://app.radicle.xyz/ so that users can manage their node through the web client.
> Web client and API offers a subset of functionality compared to the rad cli.
You are able to setup and run more radicle nodes in the same host by altering the radicle's home path through the `RAD_HOME` Environment Variable. So you can set up a new node using the `RAD_HOME=~/.new_radicle rad auth`. For the rad web you can procide the radicle-httpd URL to connect to `rad web https://app.radicle.xyz --connect 127.0.0.1:8080`.
So in order to setup two nodes in our host we have to run:
```
# Primary Node
rad auth
rad node start
radicle-httpd
rad web #optionally appent with: https://app.radicle.xyz --connect 127.0.0.1:8080
```
For the second node use
```
# Secondary Node
RAD_HOME=~/.secondary_radicle rad auth
RAD_HOME=~/.secondary_radicle rad node start
```
### Radicle Webhooks Adapter
As mentioned eralier, the Radicle Radicle CI Broker is responsible for grabbing the radicle node events and then invoke any registered adapter for any event. An adapter is responsible for conducting any tasks.
In our case, the Radicle Webhooks Adapter is responsible for invoking any registered webhooks when an event occurs.
> The Radicle Webhooks Adapter binary does not run as a stand alone application. Radicle CI broker is responsible for spawning it with the appropriate input data.
Radicle Webhooks Adapter read the webhook configuration from the repository that the radicle node event occurred.
The Radicle Webhooks Adapter binary must have appropriate execute permissions.
The configuration that the adapter requires is:
| EnvVar | Description | Default Value |
|------------------------|---------------------------------------------------|-------------------------|
| `WEBHOOKS_RETRIES` | Total attempts until get a successful response. | 3 |
| `WEBHOOKS_TIMOUT_SECS` | Timeout for webhook request. | 30 |
| `RAD_PUBLIC_URL` | Public URL of node. Used to generate repos' URL. | "http://127.0.0.1:8080" |
| `RAD_HOME` | Path for radicle home directory. | "~/.radicle" |
| `RAD_PASSPHRASE` | Passphrase for the radicle key. | "" |
More details about the radicle wbhooks adapter can be found at its [repository](https://app.radicle.xyz/nodes/radicle.yorgos.net.gr/rad:z2hCUNw2T1qU31LyGy7VPEiS7BkxW).
### Radicle CI Broker
The Radicle CI Broker is responsible for reading events from the Radicle node and invoke any registered adapter. Radicle CI Broker must be initiated and run at the background at all times in order to grab and process any new event from the radicle node. In our setup we should connect Radicle CI broker to our primary node.
The Configuration of the broker should be stored in a .yaml file as follows:
```
$ cat webhooks.yaml
default_adapter: webhooks
adapters:
webhooks:
command: radicle-webhooks-adapter
env:
filters:
```
`/tmp/bin/radicle-webhooks-adapter` should be replace with the full path of the Radicle webhooks adapter binary path.
Then the broker can be initiated through the command:
```
RUST_LOG=debug ./ci-broker /path/to/webhooks.yaml
```
`webhooks.yaml` is the path of the yaml file that contains the broker's configuration.
More details about the radicle CI broker can be found at its [repository](https://app.radicle.xyz/nodes/radicle.liw.fi/rad:zwTxygwuz5LDGBq255RA2CbNGrz8).
## Repository setup
Now that everything is in place we have to properly setup our repository. We can choose to set it up to any of the available radicle nodes but we should seed it through the node that runs the broker.
Now it's time to initialize a git repository. This will be done in our secondary node. We do it as a normal git repository through `git init` and then we should initialize it as a rad repository through `RAD_HOME=~/.secondary_radicle rad init`.
For more details about how to initialize a repository you can check this [example](https://app.radicle.xyz/nodes/seed.radicle.xyz/rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5/tree/radicle-cli/examples/rad-init.md).
Repository should contain the configuration for the webhooks. Webhooks' and should be stored under the `.radicle/webhooks/` directory in our repo. Each configuration file should be a `.yaml` (or `.yml`) that will contain all the details for each webhook. The schema of the file's content is like this:
```yaml=
---
outgoing_webhook_name_1:
payload_url: <some_url_1>
content_type: application/json
shared_secret: <some_secret_1>
```
* `outgoing_webhook_name_1` is the friendly name of the webhook
* `payload_url` is the URL that will be invoked when an event occurs
* `content_type` is the content type of the payload (currently only `application/json` is supported)
* `shared_secret` should be a high entropy secret key used to generate the SHA-256 hash signature using HMAC hex digest from the provided webhook's secret token alongside each payload. This result will be within the headers of the webhook request so that the recipient will be able to validate the request.
Multiple webhook configuration files are supported within the repository using multiple files or separating the entries using the three dashes `---`.
As a user stores all the webhooks' details within the repository these details must be somehow hidden to the rest of the users.So, these configuration files should be encrypted in order to ensure that the configuration data are not exposed.
In order to encrypt these files we should use the [age file encryption tool](https://github.com/FiloSottile/age). This tool permits the encryption of a file using multiple public keys and the decryption of each file using just one of the private keys. After the age tool is installed you can encrypt each file using the following command:
```
age -encrypt -R radicle1.pub -R radicle2.pub -R radicle3.pub -o destination_for_encrupted_file.yaml file_to_encrypt
```
The `-R radicleX.pub` is added for every public key we want to encrypt the file with. It is completely fine to use one or more keys. The keys that should e used are the radicle public/private keys located at `$RAD_HOME/keys/` directory. We should encrypt the webhooks settings using the keys of the users that we permit to be able to view and propose changes to these settings. We should definitely use the key that the webhooks adapter will use in order to decrypt the webhooks' settings files. This key is located under the $RAD_HOME and it's passphrase should be at $RAD_PASSPHRASE.
The `destination_for_encrupted_file.yaml` is the file that should be committed to the repo. It's critical here to use the public key of the radicle node where the radicle CI broker runs in order to allow the adapter to decrypt it later.
For more details and examples for this process you can refer to the [radicle-webhooks-adapter documentation](https://app.radicle.xyz/nodes/radicle.yorgos.net.gr/rad:z2hCUNw2T1qU31LyGy7VPEiS7BkxW/tree/docs/webhook_settings_persistence.md).
> For this setup there is no reason to follow the steps under the `Automated encryption & decryption` section but feel free to do so!
Now that we have our repository ready we have to ensure that the radicle node that runs alongside the broker (the primary node) is seeding the repository. Seeding a repository means that the Radicle node will seed that repo in the Radicle p2p network (as we seed in torrents). This can be done either by cloning the repo through the `rad clode RAD_ID` command or the `rad seed RAD_ID` command at the primary node (where the broker runs). `RAD_ID` is the repo's radicle ID. This can be retrieved through the `rad ls` command or by running the `rad .` command within the repository's folder.
> By seeding a repository we ensure that the a node receives any events for the specific repo.
## Testing
We can now check our setup if everything works fine. From the main node we should start the radicle node and the broker. The broker will report some logs:
```
Running `target/debug/ci-broker webhooks`
INFO ci_broker > Radicle CI broker starts
DEBUG ci_broker > loaded configuration: Config {
default_adapter: "webhooks",
adapters: {
"webhooks": Adapter {
command: "/tmp/bin/radicle-webhooks-adapter",
env: {},
},
},
filters: [],
}
INFO radicle_ci_broker::event > subscribing to local node events
```
> You can use a webhook listener service (such as the https://webhook.site/) in order to register the provided URL and check the requested payload alongside any headers.
>
### Push event
From the secondary node we should push some changes to the repo. Pushing will automatically sync with other nodes and will print something like in the broker's log output:
```
$ git push
✓ Patch b1d3420 updated to 54884569f96ac1db8ea1fb19004a536bb6e0ab83
✓ Synced with 2 node(s)
To rad://z32iyJDyFLqvPFzwHm8YadK4HQ2EY/z6MksMpnzPF48pk4XAnqVotKmfs2SE3bxA57UA8KL9DnWnY3
a680e4d..6a76ee5 ha -> patches/b1d3420337dc4817cb9f2c2a0320e60b11530a25
```
If the sync fails we can retry it through the `RAD_HOME=~/.secondary_radicle rad sync` command. A successful sync will generate some events to the broker.
If the webhook configuration is properly set up we can see in the logs the webhook URL printed out. We can see that we will receive two events from the broker which will make our broker invoke the webhooks for these two events. One event will be about the update of the `sigrefs` and the other one is for the actual push event on the repo.
### Patch event
Patch creation and updates generate new node events which then will invoke the webhooks. Each patch generates three events on the broker which results into three webhook invocations. One event will be about the update of the `sigrefs` and the other one is for the actual push event on the repo and the third one is about the patch creation/update.
In order to create a patch we can follow the following steps:
1. Checkout to a new branch and commit changes:
```
$ git checkout -b a-new-branch
$ touch some-file
$ git add some-file
$ git commit -v -m "Added some-file"
```
Open the patch from this branch :
```
$ git push rad -o patch.message="Added some-file" -o patch.message="Will use it later." HEAD:refs/patches
✓ Patch 0f3cd0b3a69c8f70bfa2d3366122c07704e5bb5f opened
To rad://z42hL2jL4XNk6K8oHQaSWfMgCL7ji/z6MknSLrJoTcukLrE435hVNQT4JUhbvWLX4kUzqkEStBU8Vi
* [new reference] HEAD -> refs/patches
```
If we want to make an update to a patch we can commit the changes and then push the changes using:
```
$ git push rad HEAD:refs/heads/a-new-branch
```
In order to view open patches we can run from the repo's directory this command:
```
$ rad patch list
```
In order to view details about each patch we can run from the repo's directory this command:
```
$ rad patch show PATCH_ID
```
More details about these can be found at the heartwood CLI examples pages:
https://app.radicle.xyz/nodes/seed.radicle.xyz/rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5/tree/radicle-cli/examples/rad-patch.md
https://app.radicle.xyz/nodes/seed.radicle.xyz/rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5/tree/radicle-cli/examples/rad-patch-update.md
## Patch Comment
There are cases where we want to report back to a patch the result of a pipeline. A patch is the term used by radicle for the Pull Request or the Merge Request used by other forges. In order to do so the 3rd party CI system can use an HTTP API reuuest to add a patch comment to the afforementioned patch in the webhook payload.
The API that should be invoked is
```
PATCH :radicle_httpd_url/api/v1/projects/:project_id/patches/:patch_id
--- payload ---
{
"type": "revision.comment",
"body": ":body"
"revision": ":revision_id"
}
```
Where
:radicle_httpd_url: The URL of the radicle HTTPD
:project_id: The project id (e.g. rad:z2hCUNw2T1qU31LyGy7VPEiS7BkxW)
:patch_id: The patch ID - can be found in the webhook payload
:body: The comment's body message
:revision_id: The patch revision ID - can be found in the webhook payload
In order to grant authorazation against the radicle-http an authorization header must be added to the request:
```
"content-type" : "application/json"
"Authorization" : "Bearer :token"
```
:token can be generated through httpd but it expires after some time if no refresh action is invoked. So we will use the `rad-web` which periodically refreshes the generated token. When we run `rad-web` we will see at the `radicle-httpd` logs a line that displays our token:
```
INFO request{id=80}: radicle_httpd: 127.0.0.1:55234 "GET /api/v1/sessions/XzOxhprQRUHKqIFzEKMasGTeb6WOJAcW HTTP/1.1" 200 OK 110.334µs 200
```
The `XzOxhprQRUHKqIFzEKMasGTeb6WOJAcW` is the token we are looking for. As long as the web client runs, it will update this token. If we restart it a new token will be generated and we should use that.
> There is an ongoing work to generate long-lived access tokens and this approach is a workaound.
It is clear that `radicle-httpd` should be accessible from the CI system so that it can invoke it to add the patch comment.
Sign in with Wallet
Connect another wallet