# Ackal Tutorial
## Table of Contents
+ [Status](#Status)
+ [Known Issues](#Known-Issues)
+ [Installation](#Installation)
+ [Console](#Console)
+ [Credentials](#Credentials)
+ [Domains](#Domains)
+ [Healthchecks](#Healthchecks)
+ [Metrics](#Metrics)
+ [`akalctl-http-proxy`](#ackalctl-http-proxy)
+ [Prometheus](#Prometheus)
+ [Grafana](#Grafana)
+ [Development](#Development)
+ [FAQ](#Frequently-Asked-Questions-FAQ)
+ [Contraints and Limitations](#Constraints-And-Limitations)
## Status (<span style="color:red">UNAVAILABLE</span>)
Ackal Service Health is [here](https://ackal-status.fly.dev).
You may check this site to determine whether the Ackal system is currently deployed.
<!--## <span style="color:red">Known Issues</span>
There's an issue with the renewal of credentials when using the public (i.e. `renew.ack.al:443`) endpoint of the token renewal service. The interim fix is to use the endpoint shown below. The endpoint changes daily. Please check here for today's URL:
|Date|URL|
|----|---|
|`2022-05-30`|`renew-mevgshqlpa-uw.a.run.app:443`|
-->
## Installation
Ackal's command-line tool, `ackalctl`, enables you to run commands against Ackal. The commands include `create`'ing, `list`'ing, and `delete`'ing Healthchecks against gRPC services as well as running a Prometheus Service Discovery proxy (`ackalctl-file-proxy` or `ackalctl-http-proxy`) that uses your credentials to authenticate Prometheus so that it may scrape your Healthchecks' metrics.
Please run `ackalctl` in a container:
```bash=
ACKALCTL_CONFIG="${HOME}/.config/ackalctl"
[ -d "${ACKALCTL_CONFIG}" ] || mkdir ${ACKALCTL_CONFIG}
# Replace VALUE with your Customer ID
CUSTOMER_ID="VALUE"
# Create config.json
echo "
{
\"customer\": {
\"id\": \"{CUSTOMER ID}\"
},
\"services\":{
\"ackal\": \"service.ack.al:443\",
\"renew\": \"renew.ack.al:443\"
}
}" > ${ACKALCTL_CONFIG_HOME}/config.json
ACKALCTL="ghcr.io/brabantcourt/ackalctl:v0.0.30"
# See `NOTE` below
podman run \
--interactive --tty --rm \
--volume=${ACKALCTL_CONFIG}:/root/.config/ackalctl \
${ACKALCTL}
```
> **NOTE**
> 1. In this example [`podman`](https://podman.io) is used but you may use [`docker`](https://docker.io) or any other OCI-compliant container runtime.
> 2. `ackalctl` stores your credentials (`credentials.json`) and application configuration (`configuration`) in your home (`${HOME}/.config/ackactl`) directory. This directory must exist before running the container to avoid mount errors.
> 3. In the remainder of this tutorial `ackalctl` will be used as shorthand for the `podman run ... ${ACKALCTL}` command shown above.
## Console
Ackal provides a web-based [Console](https://console.ack.al) to provide administrative functionality including logging in and subscribing to Ackal.
Ackal uses [Firebase Authentication](https://firebase.google.com/products/auth). This service supports various OAuth providers and enables you to choose your preferred provider(s) to login to Ackal. Once you are signed in, you can download your credentials as a JSON file to configure `ackalctl`.
Ackal use [Stripe](https://stripe.com). This service is a trusted payments service and enables you to subscribe to Ackal. Even if signed in, you will be able to use Ackal until you purchase a subscription.
Console displays user information and access to additional functionality:
+ [Login](#Login)
+ [Download](#Download)
+ [Subscribe](#Subscribe)
+ [Billing](#Billing)
### Login
Ackal supports several federated identity providers.
> **NOTE** Contact us if you want to associate different identities with a single Customer ID.
### Download
Clicking `DOWNLOAD` will download a Credentials file (`credentials.json`) to your machine.
`ackalctl` looks for this file in `${HOME}/.config/ackalctl/credential.json`.
### Subscribe
Clicking `SUBSCRIBE` will redirect you to Stripe and to the subscription signup page.
You may enter your billing details on this page to subscribe to Ackal.
### Billing
Clicking `BILLING` will redirect to Stripe and to your subscription details page.
You may view details of your subcription and cancel your subscription on this page.
## Credentials
Ackal's [Console](#console) provides a way for you to download your credentials as a JSON file (typically `credentials.json`).
`actlctl` stores user credentials in `${HOME}/.config/ackalctl/credentials.json`.
You will need a Customer ID. A Customer ID represents one (individual) or many (organizational) users.
After subscribing to Ackal, you will need to request a Customer ID before you may begin creating Healthchecks against your gRPC services.
> **NOTE** Once created in [Console](#console), your Credentials will expire within 60 minutes. Once you being using them with `ackalctl`, it will renew them for you indefinitely. If you need to refresh your Credentials in Console, log out and back in.
## Domains
You are permitted to create Healthchecks against servers on:
1. [public domains](#Public-Domains) (e.g. [Google Cloud Run](https://cloud.google.com/run))
2. domains that you have verified that you own/control
Before using domains that you own/control, you will need to generate a verification code and then add a DNS record to your domain to verify ownership to Ackal:
```bash=
DOMAIN="[YOUR DOMAIN]"
# Create the domain's record
ackalctl \
--domain_name=${DOMAIN} \
create domain
```
Either:
```bash=
ackalctl \
--domain_name=${DOMAIN} \
get domain
```
> **NOTE** The result includes a field `verified` that will be `false` until the domain has been verified
Or:
```bash=
DOMAIN="[YOUR DOMAIN]"
VERIFICATION=$(\
ackalctl \
--domain_name=${DOMAIN} \
--format=json \
get domain) && echo ${VERIFICATION}
```
And then ensure your `DOMAIN` DNS records include:
|Name|Type|Data|
|----|----|----|
||`TXT`|``"ackal-verification={VERIFICATION}"``|
> **NOTE** The requirement for `NAME` varies by DNS provider. Some permit the `NAME` field to be left blank, others require e.g. `@` as a placeholder. Please consult your documentation.
You may use Linux tools (e.g. `dig` or `nslookup`) or Google's Admin Toolbox [Dig](https://toolbox.googleapps.com/apps/dig/#TXT/) to verify that the `TXT` record has been created correctly.
Once you're confident the record has propagated, you can instruct Ackal to verify the domain:
```bash=
DOMAIN="[YOUR DOMAIN]"
ackalctl \
--domain_name=${DOMAIN} \
verify domain
```
You can confirm that the domain has been verified successfully:
```bash=
ackalctl \
--domain_name=${DOMAIN} \
get domain
```
> **NOTE** The result includes a field `verified` that will be `true` if Ackal was able to verify the domain.
## Healthchecks
Once you have `ackalctl` installed, you've logged in and downloaded your `credentials.json` file, you've acquired a Customer ID, you're ready to begin creating Healthchecks against gRPC services.
But you'll need gRPC services that implement the [gRPC Healthcheck Protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md). If you do not have your own services to test, there are 2 example services for you to use:
+ `healthcheck-server.brabantcourt.com:443`
+ `healthcheck-server.fly.dev:443`
You can healthcheck these services using a general-purpose tool such as [gRPCurl](https://github.com/fullstorydev/grpcurl) to confirm that they're working correctly:
```bash=
ENDPOINT="healthcheck-server.brabantcourt.com:443"
grpcurl \
${ENDPOINT} \
grpc.health.v1.Health/Check
```
> **NOTE** When you reference these gRPC services (using `--endpoint`) with `ackalctl`, it is important that you **not** prefix the `host:port` value with a scheme (i.e. no 'https') and you must include the port (i.e. `443`). Web browsers generally assume `443` when you're using `https` but this assumption is not valid for gRPC.
To test that `ackalctl` is working correctly:
```bash=
ackalctl list locations
```
Yields:
```console
ID Name
las-vegas Las Vegas
london London
los-angeles Los Angeles
salt-lake-city Salt Lake City
the-dalles The Dalles
tokyo Tokyo
```
> **NOTE** `ackalctl` supports JSON and YAML output formatting. Simple append `--format=json` or `--format=yaml` to any command, e.g. `ackalctl --format=json list locations`
>
> ```json
> {
> "locations":[
> {"id":"las-vegas","location":{"name":"Las Vegas"}},
> {"id":"london","location":{"name":"London"}},
> {"id":"los-angeles","location":{"name":"Los Angeles"}},
> {"id":"salt-lake-city","location":{"name":"Salt Lake City"}},
> {"id":"the-dalles","location":{"name":"The Dalles"}},
> {"id":"tokyo","location":{"name":"Tokyo"}}
> ]
> }
> ```
```bash=
ENDPOINT="healthcheck-server.brabantcourt.com:443"
PERIOD="120s"
LOCATION_IDS="las-vegas,the-dalles"
ackalctl \
--endpoint=${ENDPOINT} \
--period=${PERIOD} \
--location_ids=${LOCATION_IDS}
create check
```
> **NOTE** Healthchecks are uniquely identified by the service endpoint and the period.
Returns:
```console
id: {id}
```
Then:
```bash=
ackalctl \
--endpoint=${ENDPOINT} \
--period=${PERIOD} \
get check
```
Returns:
```console
get check
endpoint: ${ENDPOINT}
period: ${PERIOD}
locationIDs: ${LOCATION_IDS}
enabled: false
```
Or:
```bash=
ackalctl \
list checks
```
Returns:
```console
Endpoint Period Location(s)
${ENDPOINT} ${PERIOD} ${LOCATION_IDS}
```
You may delete Healthchecks too:
```bash=
ackalctl \
--endpoint=${ENDPOINT} \
--period=${PERIOD} \
delete check
```
> **NOTE** Because Healthchecks are uniquely identified by the service endpoint and the period, both values are required to delete a Healthcheck.
## Metrics
Ackal publishes Prometheus metrics for the Healthchecks that you create. The simplest way to interact with your Healthchecks' metrics is to use Prometheus (and possibly AlertManager and Grafana) to record, alert and graph them. Ackal provides 2 proxies (`ackal-http-proxy` and `ackal-file-proxy`) that perform service discovery for Prometheus. Running either (or both) of these proxies locally, they will leverage your credentials to securely access your Healthcheck so that their metrics may be scraped by Prometheus.
You will need to have at least one Healthcheck:
```bash=
ackalctl \
--endpoint=${ENDPOINT} \
--period=${PERIOD} \
--location_ids=${LOCATION_IDS}
get check
```
Returns:
```console
get check
endpoint: ${ENDPOINT}
period: ${PERIOD}
locationIDs: ${LOCATION_IDS}
enabled: false
```
> **OPTIONAL**
> Ackal's Prometheus proxies discover your Healthcheck endpoints using an Ackal service called `listr`. You may use `listr` directly using a tool like `curl`. The following commands use [`jq`](https://stedolan.github.io/jq/) for processing the JSON output of the `curl` commands.
> ```bash=
> CUSTOMER="[Your Customer ID]"
> TOKEN=$(more ${HOME}/.config/ackalctl/credentials.json | jq -r '.jwt')
>
> curl \
> --silent \
> --request GET \
> --header "Ackal-Customer: ${CUSTOMER}" \
> --header "Authorization: Bearer ${TOKEN}" \
> https://listr.ack.al:443/?format=prometheus \
> | jq -r .
> ```
> This will list your Ackal Healthcheck service endpoints in a Prometheus-friendly format:
> ```JSON
> [
> {
> "targets": [
> "{url}"
> ],
> "labels": {
> "endpoint": "{endpoint}",
> "location": "{location}",
> "period": "{period}"
> }
> }
> ]
> ```
> If you omit the query string (`?format=prometheus`), the resulting JSON will be in Ackal's native format.
>
> You can then use e.g. `curl` again to query one of the returned endpoints for its metrics:
> ```bash=
> curl \
> --silent \
> --request GET \
> --header "Ackal-Customer: ${CUSTOMER}" \
> --header "Authorization: Bearer ${TOKEN}" \
> ${endpoint}/metrics \
> | awk '\^ackal_exporter_check {print}
> ```
> Ackal's Checks' metrics are prefixed `ackal_exporter_check` so the results are filtered for these.
It is more convenient to configure a Prometheus server to discover your services for you. To do this, you will need to:
1. Run [`ackalctl-http-proxy`](#ackalctl-http-proxy)
1. Run [Prometheus](#Prometheus)
1. (Optional) Run [Grafana](#Grafana)
## `ackalctl-http-proxy`
The tutorial uses [Podman](https://podman.io) but you may use [Docker](https://docker.io) or any OCI-compliant container runtime.
`ackalctl-http-proxy` run on your host machine using whichever port (`PORT`) you prefer. The proxy is accessed by Prometheus (configured next). Prometheus uses the proxy to authenticate requests using your Ackal credentials (`${HOME}/.config/ackalctl/credentials.json`). When authorized, Prometheus uses the proxy to access Ackal's `listr` service to discover a list of your Healthchecks. Prometheus then uses the proxy to scrape metrics from each of your Healthchecks that it discovers.
```bash=
PORT="7777"
ENDPOINT="0.0.0.0:${PORT}"
ACKALCTL_HTTP_PROXY="ghcr.io/brabantcourt/ackalctl-http-proxy:220511"
podman run \
--detach --tty --rm \
--env=ACKAL_CUSTOMER_ID=${CUSTOMER} \
--volume=${HOME}/.config/ackalctl:/root/.config/ackalctl \
--publish=${PORT}:${PORT} \
${ACKALCTL_HTTP_PROXY} \
--endpoint=${ENDPOINT}
```
## Prometheus
Prometheus' configuration needs to include `ackalctl-http-proxy` as one of its scrape targets:
`prometheus.yml`:
```yaml=
global:
scrape_interval: 1m
evaluation_interval: 1m
scrape_configs:
- job_name: ackalctl-http-proxy
scheme: http
proxy_url: http://0.0.0.0:7777
http_sd_configs:
- refresh_interval: 1m
url: http://listr.ack.al:443/?format=prometheus
proxy_url: http://0.0.0.0:7777
```
> **NOTE**
> 1. If you used a different `ENDPOINT` to run `ackalctl-http-proxy`, then you will need to change both occurrences of `proxy_url` in this file.
> 2. You must use `http://listr.ack.al:443` as the URL for Ackal's `listr` service (see [Ackal Proxies and TLS](#Ackal-Proxies-and-TLS))
> 3. If you made the `curl` requests in the [Metrics](#Metrics) section, you can now `curl` `listr.ack.al` and your Healthcheck endpoints through the proxy **without** providing your Customer ID or Ackal token. This is because the proxy is providing this information on your behalf:
> ```bash
> curl \
> --silent \
> --request GET \
> --proxy http://0.0.0.0:7777 \
> http://listr.ack.al:443/?format=prometheus \
> | jq -r .
>
> curl \
> --silent \
> --request GET \
> --proxy http://0.0.0.0:7777 \
> ${endpoint}/metrics \
> | awk '\^ackal_exporter_check {print}
> ```
> **NOTE** You must replace `https://` with `http://` and add `:443` to the Healthcheck endpoint URLs (see [Ackal Proxies and TLS](#Ackal-Proxies-and-TLS))
Then launch Prometheus configured to use this file:
```bash=
podman run \
--interactive --tty --rm \
--net=host \
--volume=/path/to/prometheus.yml:/etc/prometheus/prometheus.yml:ro \
docker.io/prom/prometheus:v2.34.0 \
--config.file=/etc/prometheus/prometheus.yml \
--web.enable-lifecycle
```
> **NOTE** In order for Prometheus to access `ackalctl-http-proxy`, Prometheus must be bound to the host's network with `--net=host`
You should now be able to browse Prometheus (`http://localhost:9090`):
And see that your Healthchecks are being scraped (`http://localhost:9090/targets`):
You may interact with Ackal's Healthcheck metrics using PromQL:
`avg without(endpoint,status) (ackal_exporter_check_histogram_ms_bucket{job="ackalctl-http-proxy",location="{location}",endpoint="{endpoint}"})
`
> **NOTE** Replace `{location}` and `{endpoint}` in the above. Prometheus will prompt you with a list of available values as you type the query.
## Grafana
If you would prefer to interact with Ackal's Healthcheck metrics through Grafana:
```bash=
podman run \
--detach --rm --tty \
--net=host \
docker.io/grafana/grafana-oss:8.4.4
```
> **NOTE** In order for Grafana to access Prometheus, Grafana must (also) be bound to the host's network with `--net=host`.
You should now be able to browse Grafana (`http://localhost:3000`)
The default username|password are both `admin`.
You will want to add Prometheus as a datasource (`http://localhost:3000/datasources`). The URL will be the default (`http://localhost:9090`) but you must type this in. Then "Save & Test" to confirm that it works.
Then create a new Dashboard (`http://localhost:3000/dashboard`) and for the value of `Metrics Browser` you may use the same PromQL as above.
Alternatively, you may import an example [Ackal Check Dashboard](https://gist.github.com/DazWilkin/883c40755dea6b22eae885da5f7c150f) that includes dropdowns that enable you to choose which Check and Location you would like to view:
## Development
TBD
## Frequently Asked Questions (FAQ)
### Public Domains
Ackal supports the creation of Healthchecks against services running on public domains. These are domains -- often provided by cloud providers -- that are used to host services but which the developer does not own or control.
For this reason, Healthchecks created against these domains are permitted without the user first needing to verify ownership of the domain.
For public (non-authenticated) services on this domain -- like any other tool -- Ackal is able to access any service hosted on these domains.
Ackal supports the following public domains:
|Domain|Service|Provider|
|------|-------|--------|
|`appspot.com`|App Engine|Google|
|`cloudfunctions.net`|Cloud Functions|Google|
|`a.app.run`|Cloud Run|Google|
|`fly.dev`|fly.io|Fly|
### Ackal Proxies and TLS
### What is [gRPC Healthcheck Protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md)?
## Constraints and Limitations
### Healthcheck servers|services
Sign in with Wallet
Connect another wallet