---
# System prepended metadata

title: Configure NGINX with LetsEncrypt in front of OpenNMS

---

Configure NGINX with LetsEncrypt in front of OpenNMS
====

The following explains how to perform TLS termination using [NGINX](https://www.nginx.com/) and [LetsEncrypt](https://letsencrypt.org/) for OpenNMS and Grafana; all running on the same server.

Even if you can manage TLS within the embedded Jetty available in OpenNMS, it is better and faster doing it outside of it using an external proxy. There are multiple ones out there, and NGINX is one of them. You could also do this using [Apache HTTPD](http://httpd.apache.org/), [Envoy](https://www.envoyproxy.io/), or any other.

The following procedure has been designed and tested on RHEL/CentOS 8.

## Requirements

* Have an [Azure Subscription](https://azure.microsoft.com/en-us/free/) ready.
* Install [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
* Install [Multipass](https://multipass.run/)

The scripts used through this tutorial use [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html), make sure to have it installed.

:::info
Make sure to log into Azure using `az login` prior creating the Azure resources.
:::

Keep in mind that nothing prevents you from using another cloud provider. Avoiding a local VM is important because LetsEncrypt must reach the website to issue and validate the certificate.

## Create common Environment Variables

```bash=
# Main
export RG_NAME="OpenNMS" # Change it to use a shared one
export LOCATION="eastus" # Azure Region
export VM_NAME="$USER-onms01"
export FQDN="$VM_NAME.$LOCATION.cloudapp.azure.com"
export EMAIL="agalue@opennms.org" # Please use your own email
export ONMS_VM_SIZE="Standard_D2s_v3" # 2 VCPU, 8 GB of RAM
export ONMS_HEAP="4096" # Expressed in MB and must fit ONMS_VM_SIZE
export MINION_LOCATION="Apex"
export MINION_ID="minion01"
export MINION_HEAP="1g" # Java Heap Size
```

:::success
Feel free to change the content and keep in mind that `$USER` is what we will use throughout this tutorial to identify all the resources we will create in Azure uniquely.
:::

:::warning
Do not confuse the Azure Location or Region with the Minion Location; they are both unrelated things.
:::

To simplify the deployment, I'm not going to use a custom DNS name for the server. Instead, I'm going to use the public FQDN assigned by Azure. Of course, as we don't own that domain, Let's Encrypt will use HTTP for validation purposes.

In Azure, the default public DNS follow the same pattern:

```
<name>.<location>.cloudapp.azure.com
```

:::warning
The name must be unique within the `location` (or Azure region), so make sure to guarantee that. I'm going to use `agalue-onms01`.
:::

That means, the FQDN of the Azure VM would be:

```
agalue-onms01.eastus.cloudapp.azure.com
```

The above is what we can use to access the VM via SSH and to configure Minions, besides being able to create a valid certificate for it.

## Create a cloud-init template for the applications.

The following [cloud-init](https://cloudinit.readthedocs.io/en/latest/) template contains all the details to install PostgreSQL, OpenJDK 11, OpenNMS Horizon 28+, Grafana 7, Helm 7, Nginx, and [Certbot](https://certbot.eff.org/) on CentOS/RHEL 8.


Create a file called `/tmp/opennms-template.yaml` with the following content:

```yaml=
#cloud-config
package_upgrade: false

write_files:

- owner: root:root
  path: /opt/opennms/etc/opennms.properties.d/webui.properties
  content: |
    org.opennms.netmgt.jetty.host = 127.0.0.1
    opennms.web.base-url = https://%x%c/

- owner: root:root
  path: /opt/opennms/etc/opennms.properties.d/rrd.properties
  content: |
    org.opennms.rrd.storeByGroup=true
    org.opennms.rrd.storeByForeignSource=true
    org.opennms.rrd.strategyClass=org.opennms.netmgt.rrd.rrdtool.MultithreadedJniRrdStrategy
    org.opennms.rrd.interfaceJar=/usr/share/java/jrrd2.jar
    opennms.library.jrrd2=/usr/lib64/libjrrd2.so

- owner: root:root
  path: /opt/opennms/bin/configure-amq.sh
  permissions: '0750'
  content: |
    #/bin/bash
    PASSWD="0p3nNM5"
    TEMP_P12="/tmp/ssl.p12.$(date +%s)"
    TEMP_KEYSTORE="/tmp/ssl.keystore.$(date +%s)"
    TARGET_KEYSTORE="/opt/opennms/etc/opennms.letsencrypt.jks"
    openssl pkcs12 -export \
      -in /etc/letsencrypt/live/$FQDN/fullchain.pem \
      -inkey /etc/letsencrypt/live/$FQDN/privkey.pem \
      -out $TEMP_P12 -name opennms -password pass:$PASSWD
    keytool -importkeystore \
      -deststorepass $PASSWD -destkeypass $PASSWD \
      -destkeystore $TEMP_KEYSTORE \
      -srcstorepass $PASSWD -srckeystore $TEMP_P12 \
      -srcstoretype PKCS12  -alias opennms
    cp $TEMP_KEYSTORE $TARGET_KEYSTORE
    chmod 440 $TARGET_KEYSTORE
    AMQ_FILE=/opt/opennms/etc/opennms-activemq.xml
    sed -r -i '/0.0.0.0:61616/s/([<][!]--|--[>])//g' $AMQ_FILE
    sed -r -i '/0.0.0.0:61616/s/tcp/ssl/' $AMQ_FILE
    read -r -d '' CTX <<EOF
    <sslContext>
      <sslContext keyStore="$TARGET_KEYSTORE" keyStorePassword="$PASSWD"/>
    </sslContext>
    EOF
    TEMP_AMQ="/tmp/amq.$(date +%s)"
    perl -plne "print '$CTX' if /<destinationPolicy>/;" $AMQ_FILE > $TEMP_AMQ
    mv -f $TEMP_AMQ $AMQ_FILE
    rm -f $TEMP_P12 $TEMP_KEYSTORE

- owner: root:root
  path: /etc/nginx/default.d/opennms.conf
  content: |
    server_name $FQDN;
    # maintain the .well-known directory alias for LetsEncrypt renewals
    location /.well-known {
        alias /var/www/$FQDN/.well-known;
    }
    location /hawtio/ {
        proxy_pass http://localhost:8980/hawtio/;
    }
    location /grafana/ {
        proxy_pass http://localhost:3000/;
    }
    location /opennms/ {
        proxy_set_header    Host $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_set_header    Upgrade $http_upgrade;
        proxy_set_header    Connection "Upgrade";
        proxy_pass          http://localhost:8980/opennms/;
        proxy_redirect      default;
        proxy_read_timeout  90;
    }

runcmd:

# Install and configure NGINX
- dnf install -y epel-release
- dnf install -y nginx
- mkdir -p /var/www/$FQDN/.well-known
- chown nginx:nginx /var/www
- setsebool -P httpd_can_network_connect 1
- systemctl --now enable nginx

# Install and start Haveged
- dnf install -y haveged
- systemctl --now enable haveged

# Install Dependencies (OpenJDK 11, PostgreSQL 10)
- dnf install -y postgresql-server java-11-openjdk-devel epel-release

# OpenNMS Horizon, Grafana and Helm
- dnf install -y https://yum.opennms.org/repofiles/opennms-repo-stable-rhel8.noarch.rpm
- dnf install -y jrrd2 opennms-core opennms-webapp-jetty opennms-webapp-hawtio opennms-helm 

# Install and configure certbot
- dnf install -y certbot python3-certbot-nginx
- certbot --nginx -d $FQDN --non-interactive --agree-tos -m $EMAIL

# Configure and start PostgreSQL
- /usr/bin/postgresql-setup --initdb --unit postgresql
- sed -r -i "/^(local|host)/s/(peer|ident)/trust/g" /var/lib/pgsql/data/pg_hba.conf
- systemctl --now enable postgresql

# Configure and start OpenNMS
- sed -r -i '/enabled="false"/{$!{N;s/ enabled="false"[>]\n(.*OpenNMS:Name=Syslogd.*)/>\n\1/}}' /opt/opennms/etc/service-configuration.xml
- /opt/opennms/bin/configure-amq.sh
- echo 'JAVA_HEAP_SIZE=$ONMS_HEAP' > /opt/opennms/etc/opennms.conf
- /opt/opennms/bin/runjava -s
- RUNAS=opennms /opt/opennms/bin/fix-permissions
- /opt/opennms/bin/install -dis
- systemctl --now enable opennms

# Configure and start Grafana
- sed -i -r "s|^;domain =.*|domain = $FQDN|" /etc/grafana/grafana.ini
- sed -i -r "s|^;root_url =.*|root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/|" /etc/grafana/grafana.ini
- systemctl --now enable grafana-server
```

:::warning
The environment variables `$FQDN`, `$EMAIL` and `$ONMS_HEAP` must be substituted prior using the template, and we're going to use `envsubst` for this purpose.
:::

The above expects Horizon 28 or newer. If you're planing to use Horizon 27, make sure to add the following to the template, right before starting OpenNMS:

```yaml=
# Patch OpenNMS due to NMS-13111 to use TLS with AMQ
- rm -f /opt/opennms/lib/bcprov-jdk15on*
- wget -O/opt/opennms/lib/bcprov-jdk15on-169b07.jar https://downloads.bouncycastle.org/betas/bcprov-jdk15on-169b07.jar
```

## Create VM for the applications

If you're not planning to use a shared resource group in Azure, go ahead and create it:

```bash=
az group create -n $RG_NAME -l $LOCATION --tags Owner=$USER
```

Create a CentOS 8 VM using the above template:

```bash=
envsubst '$FQDN $EMAIL $ONMS_HEAP' < opennms-template.yaml > opennms.yaml

az vm create --resource-group $RG_NAME --name $VM_NAME \
   --size $ONMS_VM_SIZE \
   --image OpenLogic:CentOS:8_4:latest \
   --ssh-key-values ~/.ssh/id_rsa.pub \
   --public-ip-sku Standard \
   --public-ip-address-dns-name $VM_NAME \
   --custom-data /tmp/opennms.yaml \
   --tags Owner=$USER \
   --output table

az vm open-port -g $RG_NAME -n $VM_NAME --port 61616 --priority 100 -o table
az vm open-port -g $RG_NAME -n $VM_NAME --port 443 --priority 110 -o table
az vm open-port -g $RG_NAME -n $VM_NAME --port 80 --priority 120 -o table
```

Note that for the public FQDN, you only need to specify the name portion. I'm opening port 80, as `Certbot` requires it for validating the certificate using HTTP. As we're going to be accessing the applications via HTTPS, we need access to port 443. Finally, for testing purposes (if you're interested), I'm opening the ActiveMQ port (port 61616).

:::warning
In case there is a problem, SSH into the VM using the public IP and the provided credentials and check /var/log/cloud-init-output.log to verify the progress and the status of the cloud-init execution.
:::

If everything goes well, you can access the applications through the following URLs:

OpenNMS: https://agalue-onms01.eastus.cloudapp.azure.com/opennms/

HawtIO: https://agalue-onms01.eastus.cloudapp.azure.com/hawtio/

Grafana: https://agalue-onms01.eastus.cloudapp.azure.com/grafana/

ActiveMQ: `failover://ssl://agalue-onms01.eastus.cloudapp.azure.com:61616`

You can verify in your browser that the certificate is valid and issues by LetsEncrypt.

In production, it is advised to configure a cron job to renew the certificate at least once monthly.

:::success
I recommend checking `/etc/nginx/nginx.conf`. The `Certbot` utility modifies that file to add all the SSL/TLS settings required to use the LetsEncrypt certificate with Nginx. This can be beneficial if you want to use a self-signed certificate or a third-party CA to understand how to configure Nginx.
:::

Here are the generated TLS/SSL settings added to `nginx.conf`:

```
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/agalue-onms01.eastus.cloudapp.azure.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/agalue-onms01.eastus.cloudapp.azure.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
```

## Create Minion VMs using `multipass`

After verifying that OpenNMS is up and running, you can proceed to create the Minions.

Create a [cloud-init](https://cloudinit.readthedocs.io/en/latest/) script to deploy Minion in Ubuntu and save it at `/tmp/minion-template.yaml`:

```yaml=
#cloud-config
package_upgrade: true

write_files:
  - owner: root:root
    path: /tmp/org.opennms.minion.controller.cfg
    content: |
      location=$MINION_LOCATION
      id=$MINION_ID
      http-url=https://$FQDN/opennms
      broker-url=failover://ssl://$FQDN:61616

apt:
  preserve_sources_list: true
  sources:
    opennms:
      source: deb https://debian.opennms.org stable main

packages:
  - opennms-minion

bootcmd:
  - curl -s https://debian.opennms.org/OPENNMS-GPG-KEY | apt-key add -

runcmd:
  - mv -f /tmp/org.opennms.minion.controller.cfg /etc/minion/
  - sed  -r 's/# export (JAVA_(MIN|MAX)_MEM)=.*/export \1="$MINION_HEAP"/' /etc/default/minion
  - /usr/share/minion/bin/scvcli set opennms.http admin admin
  - /usr/share/minion/bin/scvcli set opennms.broker admin admin
  - systemctl --now enable minion
```

:::info
Note the usage of environment variables within the YAML template. We will substitute them before creating the VM.
:::

Then, start the new Minion via `multipass` with one core and 2GB of RAM:

```bash=
envsubst < /tmp/minion-template.yaml > /tmp/$MINION_ID.yaml

multipass launch -c 1 -m 2G -n minion --cloud-init /tmp/$MINION_ID.yaml
```

## Clean Up

To remove all the cloud resources, execute the following:

```bash=
az group delete -g $RG_NAME -y
```

If you cannot remove the resource group, as all the resources created will be prefixed with the VM name, the following should work:

```bash=
IDS=($(az resource list \
  --resource-group $RG_NAME \
  --query "[?contains(name,'$VM_NAME') && type!='Microsoft.Compute/disks']".id \
  --output tsv | tr '\n' ' '))

for id in "${IDS[@]}"; do
  echo "Removing $id"
  az resource delete --ids "$id" --verbose
done

DISKS=($(az resource list \
  --resource-group $RG_NAME \
  --query "[?contains(name,'$VM_NAME') && type=='Microsoft.Compute/disks']".id \
  --output tsv | tr '\n' ' '))

for id in "${DISKS[@]}"; do
  echo "Removing $id"
  az resource delete --ids "$id" --verbose
done
```

Then clean the local resources:

```bash=
multipass delete $MINION_ID
multipass purge
