# Jigasi Deployment Documentation

This documentation will start by the documentation of all the different custom docker images that we used for our deployment: in a first part, the ones related to transcription and then the ones related to translation.

The next part of the documentation is about the deployment on Kubernetes and the auto-scaling which was implemented.

At the end of this documentation, I also added the steps to follow in order to manually enable Jigasi as a Hidden Participant.

## Jigasi Hidden Participant

### Nota Bene
While looking through jitsi community threads, I noticed [this thread](https://community.jitsi.org/t/jigasi-transcriber-needs-to-be-hidden-participant/117649/4), which brings up the fact that wanting to use Jigasi as a Hidden Participant is incompatible with using Jigasi for SIP clients (who are calling in from their phones).

### Introduction

In order to enable transcription and translation via Jigasi, we needed Jigasi to join the meetings as a Hidden Participant.
This allows participants other than the moderator to seee the close-caption button and enable their subtitles.
This issue was brought up in [this community thread](https://community.jitsi.org/t/jigasi-transcriber-needs-to-be-hidden-participant/117649)

In order to do so, ideally a PR should be opened on [docker-jitsi-meet](https://github.com/jitsi/docker-jitsi-meet), with changes for the configurations of Jigasi, Prosody and Web.

The idea is to add an environment variable, `HIDDEN_JIGASI_USER`, which would be used by the three components as follows:

### Jigasi

The hidden domain used by jigasi can be different from the `XMPP_RECORDER_DOMAIN`, but we used the `XMPP_RECORDER_DOMAIN` during the project, since it was already a hidden domain in the web component of jitsi (because our recording via Jibri is also activated).

In case we want to use a different domain, another environment variable should be defined: `XMPP_HIDDEN_DOMAIN`, which contains said domain.

In the following parts of the documentation, `XMPP_RECORDER_DOMAIN` has been used, but could be changed with `XMPP_HIDDEN_DOMAIN`.

For jigasi, the changes required are as follows:

In `rootfs/defaults/sip-communicator.properties`, in the following part:

{{ if .Env.ENABLE_AUTH | default "0" | toBool }}
{{ if .Env.ENABLE_GUESTS | default "0" | toBool }}
org.jitsi.jigasi.xmpp.acc.USER_ID={{ $JIGASI_XMPP_USER }}@{{ $XMPP_GUEST_DOMAIN }}
org.jitsi.jigasi.xmpp.acc.ANONYMOUS_AUTH=true
{{ else }}
org.jitsi.jigasi.xmpp.acc.USER_ID={{ $JIGASI_XMPP_USER }}@{{ $XMPP_AUTH_DOMAIN }}
org.jitsi.jigasi.xmpp.acc.ANONYMOUS_AUTH=false
{{ end }}
org.jitsi.jigasi.xmpp.acc.PASS={{ .Env.JIGASI_XMPP_PASSWORD }}
org.jitsi.jigasi.xmpp.acc.ALLOW_NON_SECURE=true
{{ end }}

A third case should be added, depending on the value of `HIDDEN_JIGASI_USER`, which gives the following configuration values:

{{ if .Env.HIDDEN_JIGASI_USER | default "0" | toBool }}
et donc il faut l'importer aussi au debut du fichier.
org.jitsi.jigasi.xmpp.acc.USER_ID={{ $JIGASI_XMPP_USER }}@{{ $XMPP_RECORDER_DOMAIN }}
org.jitsi.jigasi.xmpp.acc.ANONYMOUS_AUTH=false
org.jitsi.jigasi.xmpp.acc.PASS={{ .Env.JIGASI_XMPP_PASSWORD }}
org.jitsi.jigasi.xmpp.acc.ALLOW_NON_SECURE=true
{{ end }}

### Prosody
In prosody, we need to register the jigasi user, with the hidden domain it will use, at the initialization of prosody.
This is done in `rootfs/cont-init.d/10-config`. Another block should be added, just like the one already used for the normal jigasi user:

if [[ ! -z $JIGASI_XMPP_PASSWORD ]]; then
    OLD_JIGASI_XMPP_PASSWORD=passw0rd
    if [[ "$JIGASI_XMPP_PASSWORD" == "$OLD_JIGASI_XMPP_PASSWORD" ]]; then
        echo 'FATAL ERROR: Jigasi auth password must be changed, check the README'
        exit 1
    fi
    prosodyctl --config $PROSODY_CFG register $JIGASI_XMPP_USER $XMPP_AUTH_DOMAIN $JIGASI_XMPP_PASSWORD
fi

But it should be dependant on `JIGASI_HIDDEN_USER` or `JIGASI_HIDDEN_XMPP_PASSWORD`:

if [[ ! -z $JIGASI_HIDDEN_XMPP_PASSWORD ]]; then
    OLD_JIGASI_XMPP_PASSWORD=passw0rd
    if [[ "$JIGASI_HIDDEN_XMPP_PASSWORD" == "$OLD_JIGASI_XMPP_PASSWORD" ]]; then
        echo 'FATAL ERROR: Jigasi hidden auth password must be changed, check the README'
        exit 1
    fi
    prosodyctl --config $PROSODY_CFG register $JIGASI_XMPP_USER $XMPP_HIDDEN_DOMAIN $JIGASI_HIDDEN_XMPP_PASSWORD
fi

In this case, we should also use new environment variables for the jigasi hidden domain, the password associated to jigasi when it is a Hidden Participant, and maybe also a differnet `JIGASI_HIDDEN_XMPP_USER`.

### Web

In the front-end, we should enable the hidden domain just like it is already done for Jibri. For Jibri, this is done in `rootfs/defaults:settings-config.js` in the following way:

{{ if $ENABLE_RECORDING  -}}

config.hiddenDomain = '{{ $XMPP_RECORDER_DOMAIN }}';

if (!config.hasOwnProperty('recordingService')) config.recordingService = {};

// Whether to enable file recording or not using the "service" defined by the finalizer in Jibri
config.recordingService.enabled = {{ $ENABLE_SERVICE_RECORDING }};

// Whether to enable live streaming or not.
config.liveStreamingEnabled = {{ $ENABLE_LIVESTREAMING }};

{{ if .Env.DROPBOX_APPKEY -}}
// Enable the dropbox integration.
if (!config.hasOwnProperty('dropbox')) config.dropbox = {};
config.dropbox.appKey = '{{ .Env.DROPBOX_APPKEY }}';
{{ if .Env.DROPBOX_REDIRECT_URI -}}
// A URL to redirect the user to, after authenticating
// by default uses:
// 'https://jitsi-meet.example.com/static/oauth.html'
config.dropbox.redirectURI = '{{ .Env.DROPBOX_REDIRECT_URI }}';
{{ end -}}
{{ end -}}

// Whether to show the possibility to share file recording with other people
// (e.g. meeting participants), based on the actual implementation
// on the backend.
config.recordingService.sharingEnabled = {{ $ENABLE_FILE_RECORDING_SHARING }};
{{ end -}}

Another condition should be added in this if block (or `HIDDEN_JIGASI_USER`), or in case the used domain should be different (general case), a whole different block should be added to render `XMPP_HIDDEN_DOMAIN` hidden.

## Transcription
The jigasi image used in this case is a custom image that we built ourselves, because the changes done in the PRs already merged to [jitsi/jigasi](https://github.com/jitsi/jigasi) haven't seen a release to [jitsi/docker-jitsi-meet](https://github.com/jitsi/docker-jitsi-meet) yet.

For the deployments that we launched, we added the needed changes in `rootfs/defaults/sip-communicator.properties`, and we also modified the `Dockerfile` used to build the image, in order to copy a `jigasi.jar` (which contained all the new custom transcription service classes) that was built using [jitsi/jigasi](https://github.com/jitsi/jigasi), by following the README.

This step is only needed right now because there hasn't been a new jigasi release to `docker-jitsi-meet` in quite a while, even if the PRs corresponding to the changes implementing the transcription were already merged to `jitsi/jigasi`.
We won't need these steps anymore, once we get the next jigasi release on `docker-jitsi-meet`.

### Changes in sip-communicator.properties

In the part of this file that configures transcriptions:

{{ if .Env.ENABLE_TRANSCRIPTIONS | default "0" | toBool }}
# Transcription config
org.jitsi.jigasi.ENABLE_TRANSCRIPTION=true
org.jitsi.jigasi.transcription.ENABLE_TRANSLATION=false
{{ if .Env.JIGASI_CUSTOM_TRANSCRIPTION_SERVICE }}
org.jitsi.jigasi.transcription.customService={{ .Env.JIGASI_CUSTOM_TRANSCRIPTION_SERVICE }}
org.jitsi.jigasi.transcription.vosk.websocket_url={{ .Env.VOSK_WEBSOCKET_URL | default "ws://vosk-en.jitsi.svc:2700" }}
{{end}}
org.jitsi.jigasi.transcription.DIRECTORY=/tmp/transcripts
org.jitsi.jigasi.transcription.BASE_URL={{ .Env.PUBLIC_URL }}/transcripts
org.jitsi.jigasi.transcription.jetty.port=-1
org.jitsi.jigasi.transcription.ADVERTISE_URL={{ .Env.JIGASI_TRANSCRIBER_ADVERTISE_URL | default "false"}}
org.jitsi.jigasi.transcription.SAVE_JSON=false
org.jitsi.jigasi.transcription.SEND_JSON=true
org.jitsi.jigasi.transcription.SAVE_TXT=true
org.jitsi.jigasi.transcription.SEND_TXT={{ .Env.JIGASI_TRANSCRIBER_SEND_TXT | default "false"}}
org.jitsi.jigasi.transcription.RECORD_AUDIO={{ .Env.JIGASI_TRANSCRIBER_RECORD_AUDIO | default "false"}}
org.jitsi.jigasi.transcription.RECORD_AUDIO_FORMAT=wav
{{end}}

We have added the case in which we use a custom transcription service, and we configure this service as well as it's websocket if a custom service is used.

### Dockerfile

We have only added a line which copies the `jigasi.jar` file.

ARG JITSI_REPO=jitsi
ARG BASE_TAG=latest
FROM ${JITSI_REPO}/base-java:${BASE_TAG}

LABEL org.opencontainers.image.title="Jitsi Gateway to SIP (jigasi)"
LABEL org.opencontainers.image.description="Server-side application that allows regular SIP clients to join conferences."
LABEL org.opencontainers.image.url="https://github.com/jitsi/jigasi"
LABEL org.opencontainers.image.source="https://github.com/jitsi/docker-jitsi-meet"
LABEL org.opencontainers.image.documentation="https://jitsi.github.io/handbook/"

RUN apt-dpkg-wrap apt-get update && \
    apt-dpkg-wrap apt-get install -y jigasi jq curl && \
    apt-cleanup

COPY rootfs/ /
COPY jigasi.jar /usr/share/jigasi/jigasi.jar
VOLUME ["/config", "/tmp/transcripts"]

### Available image

`annagrigoriu/testing:jigasi-transcription-final`

## Translation

In this case, we still run into the problem of needing jigasi to join as a hidden participant.

The steps to take are almost the same as the ones previously described for the transcription (use of a `jigasi.jar` built from the official repository [jitsi/jigasi](https://github.com/jitsi/jigasi), which contains the classes that implement LibreTranslate translation), changes to the `Dockerfile` so that it copies the `jigasi.jar` over (so the same `Dockerfile` as for the Transcription) and changes to `sip-communicator.properties`.

I am only going to go into further details about the changes in `sip-communicator.properties` in this case.

### sip-communicator.properties

We are going to add another if block in the part of the file that configures transcription, in the case where a custom translation service is used:

{{ if .Env.ENABLE_TRANSCRIPTIONS | default "0" | toBool }}
# Transcription config
 
org.jitsi.jigasi.ENABLE_TRANSCRIPTION=true
org.jitsi.jigasi.transcription.ENABLE_TRANSLATION=true
org.jitsi.jigasi.transcription.customService={{ .Env.JIGASI_CUSTOM_TRANSCRIPTION_SERVICE | default "" }}
org.jitsi.jigasi.transcription.vosk.websocket_url={{ .Env.VOSK_WEBSOCKET_URL | default "ws://vosk-en.jitsi.svc.cluster.local:2700" }}
org.jitsi.jigasi.transcription.DIRECTORY=/tmp/transcripts
org.jitsi.jigasi.transcription.BASE_URL={{ .Env.PUBLIC_URL }}/transcripts
org.jitsi.jigasi.transcription.jetty.port=-1
org.jitsi.jigasi.transcription.ADVERTISE_URL={{ .Env.JIGASI_TRANSCRIBER_ADVERTISE_URL | default "false"}}
org.jitsi.jigasi.transcription.SAVE_JSON=false
org.jitsi.jigasi.transcription.SEND_JSON=true
org.jitsi.jigasi.transcription.SAVE_TXT=true
org.jitsi.jigasi.transcription.SEND_TXT={{ .Env.JIGASI_TRANSCRIBER_SEND_TXT | default "false"}}
org.jitsi.jigasi.transcription.RECORD_AUDIO={{ .Env.JIGASI_TRANSCRIBER_RECORD_AUDIO | default "false"}}
org.jitsi.jigasi.transcription.RECORD_AUDIO_FORMAT=wav
{{end}}

{{ if .Env.JIGASI_CUSTOM_TRANSLATION_SERVICE }}
org.jitsi.jigasi.transcription.translationService={{ .Env.JIGASI_CUSTOM_TRANSLATION_SERVICE | default "org.jitsi.jigasi.transcription.LibreTranslateTranslationService" }}
org.jitsi.jigasi.transcription.libreTranslate.api_url=http://libretranslate.jitsi.svc.cluster.local:5000/translate
{{ end }}

These changes add the LibreTranslate custom translation service, as well as it's API.

### Available Image
`annagrigoriu/testing:jigasi-translation-8`

## Jigasi Auto-scaling on Kubernetes

Jigasi's auto-scaling is heavily based on jibri's auto-scaling, which was already implemented.

The principle is the same: we have a Horizontal Pod Autoscaler, a jigasi Service Account, and a python script which runs in a sidecar container in the jigasi pod. I will go into further detail about these three components.
Jigasi pods can have a `status` label which is either unknown, idle, busy or shutdown.
The general guideline in our auto-scaling was to have one jigasi pod per meeting.

To start, we had to change the Terraform deployment in order to add another pool, "jigasi". This is configured the same exact way as for the Jibri pool.
In `terraform/kubernetes.tf`:

resource "scaleway_k8s_pool" "jigasi" {
   autohealing         = lookup(var.k8s_nodepool_autohealing, terraform.workspace, true)
   autoscaling         = lookup(var.k8s_jigasi_nodepool_autoscale, terraform.workspace, true)
   cluster_id          = scaleway_k8s_cluster.kube_cluster.id
   container_runtime   = lookup(var.k8s_nodepool_container_runtime, terraform.workspace, "containerd")
   max_size            = lookup(var.k8s_jigasi_nodepool_max_nodes, terraform.workspace, 5)
   min_size            = lookup(var.k8s_jigasi_nodepool_min_nodes, terraform.workspace, 1)
   name                = "jigasi"
   node_type           = lookup(var.k8s_jigasi_nodepool_flavor, terraform.workspace, "GP1-S")
   size                = lookup(var.k8s_jigasi_nodepool_size, terraform.workspace, 1)
   wait_for_pool_ready = false

   # We wait for default pool to be ready before creating the jigasi pool,
   # otherwise some kube-system pods created by scaleway might be scheduled
   # on the jigasi pool at cluster initialization
   depends_on = [ scaleway_k8s_pool.default ]
}

### Jigasi Deployment
There's not much special about the deployment here. It is done the same way as Jibri, with a ReplicaSet which only accepts `status` labels with values of unknown or idle. This makes it so that busy pods are not in the ReplicaSet.

We also decided to deploy VOSK in the same pod as Jigasi.

In one Jigasi pod we are going to have:
- a jigasi container
- a vosk container
- a sidecar container which contains the python script

### Python Script: jigasi-metadata-updater.py
The idea behind this script, as well as its structure is the same as the metadata-updater used for jibti.
The status of a jigasi instance is determined by the number of meetings which are currently running with that jigasi instance. We get that number by a simple GET request to the Stats API in jigasi `http://127.0.0.1:8788/about/stats`.
As soon as the number of meetings goes to 1, jigasi's status will be updated to busy.

The main loop of this script is as follows:

while True:
    try:
        new_jigasi_status = get_jigasi_status()
    except (URLError, HTTPError):
        logging.exception("Unable to get the Jigasi status")
        update_pod_metadata(0, "shutdown")
        logging.info("Pod is shutting down, conference ended")
        break

    if new_jigasi_status != jigasi_status:
        logging.info("Jigasi's status changed to : %s", new_jigasi_status)
        deletion_cost = get_pod_deletion_cost(new_jigasi_status)
        try:
            if new_jigasi_status == "IDLE" and jigasi_status == "BUSY":
                new_jigasi_status == "BUSY"
            status_label = new_jigasi_status.lower()
            update_pod_metadata(deletion_cost, status_label)
            logging.info("pod-deletion-cost annotation updated to %s", deletion_cost)
            logging.info("status label updated to %s", status_label)
            jigasi_status = new_jigasi_status
        except (FileNotFoundError, HTTPError, URLError):
            logging.exception("Unable to update pod metadata")
        if new_jigasi_status == "BUSY" and not_shutdown:
            logging.info("Initiating graceful shutdown")
            asyncio.run(initiate_graceful_shutdown())
            not_shutdown = False
    time.sleep(update_period_seconds)

Every `update_period_seconds`, we get jigasi's new status.
If the status changed, we calculate the new deletion cost, and then if jigasi's status goes back o idle after having been busy (which means the conference ended), we force it to stay busy (this makes sure that the pod doesn't rejoin the jigasi ReplicaSet). We then update the pod metadata.

I added a new variable in the script, which is called `not_shutdown` and is initialized at `True`.
The first time that jigasi's status has a value of `busy`, the last if loop in the while loop will be executed, and it will run graceful_shutdown on the jigasi container. Graceful shutdown waits for all the meetings that are currenly using this jigasi instance to end, before killing the jigasi process, but it also makes sure that new connections aren't made to this jigasi instance. This is a way to make sure that only one single meeting has used this specific jigasi pod.

***Important Mention***:
In order to allow running graceful_shutdown.sh in jigasi, in the way that we're doing it, we need to add a ling in `sip-communicator.properties`, in jigasi:

# Enable this property to be able to shutdown gracefully jigasi using
# a rest command
org.jitsi.jigasi.ENABLE_REST_SHUTDOWN=true

The function which runs graceful shutdown:

async def initiate_graceful_shutdown():
    """
    Call Kubernetes API to execute the graceful shutdown command in the Jigasi container,
    and stop further incoming calls from connecting, while waiting for all current connections 
    to end before shutting the process down.
    """
    url = f"{k8s_ws_api}/api/v1/namespaces/{namespace}/pods/{pod_name}/exec?container=jigasi&command=/usr/share/jigasi/graceful_shutdown.sh&command=-p&command=1&stdin=true&stderr=true&stdout=true&tty=true"
    headers = {
        "Authorization": f"Bearer {bearer}",
        "Accept": "*/*",
    }
    ssl_context = create_default_context()
    ssl_context.load_verify_locations(cacert)
    try:
        async with websockets.connect(url, extra_headers=headers, ssl=ssl_context) as websocket:
            logging.info("Graceful shutdown initiated")
    except Exception as e:
        logging.info("Graceful shutdown initiated")

The exception in this function happens whether or not graceful shutdown was executed, because we never receive a response in the websocket. But even if the exception is raised, the graceful shutdown is initiated.

I would also like to mention that we need a websocket here, a simple GET or POST request via HTTP/HTTPS isn't enough to execute commands via Kubernetes API.

The fact that you can `exec` into pods via Kubernetes API isn't documented whatsoever in the official Kubernetes documentation. I did however find [this article](https://cloud.redhat.com/blog/executing-commands-in-pods-using-k8s-api), which helped me implement this solution, as well as a couple of issues on git which helped me determine which rules should be applied to the jigasi Service Account in order to allow it to `exec` into pods.

I have one last remark to make about the script, which is the fact that at the very beginning of the `while True` loop, when we're requesting the new jigasi status:

    try:
        new_jigasi_status = get_jigasi_status()
    except (URLError, HTTPError):
        logging.exception("Unable to get the Jigasi status")
        update_pod_metadata(0, "shutdown")
        logging.info("Pod is shutting down, conference ended")
        break

In the case where the request to jigasi's Stats API raises an error (so, when it isn't available), we change jigasi's status label to `shutdown` with a deletion cost of 0, and we `break` the infinite loop (which allows Kubernetes to start terminating this orphaned jigasi pod).

I decided to finish the script (break the loop) this way because once there isn't any meetings left using this instance of jgiasi, the `graceful_shutdown.sh` script effectively starts the shutdown, and the API becomes unavailable.
Since the API becomes unavailable, there is no other way for the metadata-updater to know that it can stop running, and allow the pod to be terminated.

***Important remark:***
When I implemented jigasi's auto-scaling for the first time, ina way that was completely equivalent to Jibri's auto-scaling (other than the fact that it used jigasi's stats API instead of Jibri's Health API), I noticed that the pod which was just used by a meeting would go back to having an `idle` status (since it wasn't in use anymore), and it rejoined the jigasi Replicaset, which would then have 3 available pods.
Since the minimum in our scaling is 2 pods, the "youngest" pod would be terminated (and not the one that was just used by a meeting!).
This isn't the behaviour we wanted from the scaling.

In order to solve this problem, the pod needs to be left orphaned from the ReplicaSet (with a status label other than idle or unknown), and we need to `break` the infinite while loop in our metadata-updater, which allows the pod to be terminated.

Jibri's auto-scaling should maybe be changed in order to solve this problem as well (since we want one meeting per pod, and we want to kill the pod which was just used).

### Jigasi Service Account
Jigasi needs to have a service account in order to access the Kubernetes API and:
- update the pod's metadata (label and deletion cost)
- exec into the jigasi container from the metadata-updater sidecar container and run jigasi's graceful shutdown script

The rules needed to be able to exec via Kubernetes API are the following:
- get
- post
- create

### jigasi Horizontal Pod Autoscaler
The horizontal pod autoscaler is configured in the same way as Jibri's horizontal pod autoscaler.

## How to configure Jigasi as a Hidden Participant manually

### Jigasi
You need the following lines in jigasi's `sip-communicator.properties` (the same ones that I put in an if int he first part of this documentation):

org.jitsi.jigasi.xmpp.acc.USER_ID={{ $JIGASI_XMPP_USER }}@recorder.{{ $XMPP_DOMAIN }}
org.jitsi.jigasi.xmpp.acc.ANONYMOUS_AUTH=false
org.jitsi.jigasi.xmpp.acc.PASS={{ .Env.JIGASI_XMPP_PASSWORD }}
org.jitsi.jigasi.xmpp.acc.ALLOW_NON_SECURE=true

### Web
On our jitsi deployment, since recording is activated, `XMPP_RECORDER_DOMAIN` is already configured as a hidden domain.
We used this domain for our hidden participant jigasi in the rest of the configuration.

### Prosody
You need to register jigasi's hidden participant user, with the hidden domain that you want to use (`XMPP_RECORDER_DOMAIN` in our case).
You need to exec into the prosody pod and execute the following command:

`prosodyctl --config /config/prosody.cfg.lua register jigasi recorder.<domain name> <password>`

With the same password as the one that you used in `sip-communicator.properties`, in `org.jitsi.jigasi.xmpp.acc.PASS` in your Jigasi configuration.

We were unable to execute this command in the kubernetes manifests of the prosody deployment (whether it be as a simple command, or under lifecycle postStart), because the configuration file `config/prosody.cfg.lua` wasn't in the container by the time these commands were executed.