Try   HackMD

Configure NGINX with LetsEncrypt in front of OpenNMS

The following explains how to perform TLS termination using NGINX and LetsEncrypt for OpenNMS and Grafana; all running on the same server.

Even if you can manage TLS within the embedded Jetty available in OpenNMS, it is better and faster doing it outside of it using an external proxy. There are multiple ones out there, and NGINX is one of them. You could also do this using Apache HTTPD, Envoy, or any other.

The following procedure has been designed and tested on RHEL/CentOS 8.

Requirements

The scripts used through this tutorial use envsubst, make sure to have it installed.

Make sure to log into Azure using az login prior creating the Azure resources.

Keep in mind that nothing prevents you from using another cloud provider. Avoiding a local VM is important because LetsEncrypt must reach the website to issue and validate the certificate.

Create common Environment Variables












# Main
export RG_NAME="OpenNMS" # Change it to use a shared one
export LOCATION="eastus" # Azure Region
export VM_NAME="$USER-onms01"
export FQDN="$VM_NAME.$LOCATION.cloudapp.azure.com"
export EMAIL="agalue@opennms.org" # Please use your own email
export ONMS_VM_SIZE="Standard_D2s_v3" # 2 VCPU, 8 GB of RAM
export ONMS_HEAP="4096" # Expressed in MB and must fit ONMS_VM_SIZE
export MINION_LOCATION="Apex"
export MINION_ID="minion01"
export MINION_HEAP="1g" # Java Heap Size

Feel free to change the content and keep in mind that $USER is what we will use throughout this tutorial to identify all the resources we will create in Azure uniquely.

Do not confuse the Azure Location or Region with the Minion Location; they are both unrelated things.

To simplify the deployment, I'm not going to use a custom DNS name for the server. Instead, I'm going to use the public FQDN assigned by Azure. Of course, as we don't own that domain, Let's Encrypt will use HTTP for validation purposes.

In Azure, the default public DNS follow the same pattern:

<name>.<location>.cloudapp.azure.com

The name must be unique within the location (or Azure region), so make sure to guarantee that. I'm going to use agalue-onms01.

That means, the FQDN of the Azure VM would be:

agalue-onms01.eastus.cloudapp.azure.com

The above is what we can use to access the VM via SSH and to configure Minions, besides being able to create a valid certificate for it.

Create a cloud-init template for the applications.

The following cloud-init template contains all the details to install PostgreSQL, OpenJDK 11, OpenNMS Horizon 28+, Grafana 7, Helm 7, Nginx, and Certbot on CentOS/RHEL 8.

Create a file called /tmp/opennms-template.yaml with the following content:



























































































































#cloud-config
package_upgrade: false

write_files:

- owner: root:root
  path: /opt/opennms/etc/opennms.properties.d/webui.properties
  content: |
    org.opennms.netmgt.jetty.host = 127.0.0.1
    opennms.web.base-url = https://%x%c/

- owner: root:root
  path: /opt/opennms/etc/opennms.properties.d/rrd.properties
  content: |
    org.opennms.rrd.storeByGroup=true
    org.opennms.rrd.storeByForeignSource=true
    org.opennms.rrd.strategyClass=org.opennms.netmgt.rrd.rrdtool.MultithreadedJniRrdStrategy
    org.opennms.rrd.interfaceJar=/usr/share/java/jrrd2.jar
    opennms.library.jrrd2=/usr/lib64/libjrrd2.so

- owner: root:root
  path: /opt/opennms/bin/configure-amq.sh
  permissions: '0750'
  content: |
    #/bin/bash
    PASSWD="0p3nNM5"
    TEMP_P12="/tmp/ssl.p12.$(date +%s)"
    TEMP_KEYSTORE="/tmp/ssl.keystore.$(date +%s)"
    TARGET_KEYSTORE="/opt/opennms/etc/opennms.letsencrypt.jks"
    openssl pkcs12 -export \
      -in /etc/letsencrypt/live/$FQDN/fullchain.pem \
      -inkey /etc/letsencrypt/live/$FQDN/privkey.pem \
      -out $TEMP_P12 -name opennms -password pass:$PASSWD
    keytool -importkeystore \
      -deststorepass $PASSWD -destkeypass $PASSWD \
      -destkeystore $TEMP_KEYSTORE \
      -srcstorepass $PASSWD -srckeystore $TEMP_P12 \
      -srcstoretype PKCS12  -alias opennms
    cp $TEMP_KEYSTORE $TARGET_KEYSTORE
    chmod 440 $TARGET_KEYSTORE
    AMQ_FILE=/opt/opennms/etc/opennms-activemq.xml
    sed -r -i '/0.0.0.0:61616/s/([<][!]--|--[>])//g' $AMQ_FILE
    sed -r -i '/0.0.0.0:61616/s/tcp/ssl/' $AMQ_FILE
    read -r -d '' CTX <<EOF
    <sslContext>
      <sslContext keyStore="$TARGET_KEYSTORE" keyStorePassword="$PASSWD"/>
    </sslContext>
    EOF
    TEMP_AMQ="/tmp/amq.$(date +%s)"
    perl -plne "print '$CTX' if /<destinationPolicy>/;" $AMQ_FILE > $TEMP_AMQ
    mv -f $TEMP_AMQ $AMQ_FILE
    rm -f $TEMP_P12 $TEMP_KEYSTORE

- owner: root:root
  path: /etc/nginx/default.d/opennms.conf
  content: |
    server_name $FQDN;
    # maintain the .well-known directory alias for LetsEncrypt renewals
    location /.well-known {
        alias /var/www/$FQDN/.well-known;
    }
    location /hawtio/ {
        proxy_pass http://localhost:8980/hawtio/;
    }
    location /grafana/ {
        proxy_pass http://localhost:3000/;
    }
    location /opennms/ {
        proxy_set_header    Host $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_set_header    Upgrade $http_upgrade;
        proxy_set_header    Connection "Upgrade";
        proxy_pass          http://localhost:8980/opennms/;
        proxy_redirect      default;
        proxy_read_timeout  90;
    }

runcmd:

# Install and configure NGINX
- dnf install -y epel-release
- dnf install -y nginx
- mkdir -p /var/www/$FQDN/.well-known
- chown nginx:nginx /var/www
- setsebool -P httpd_can_network_connect 1
- systemctl --now enable nginx

# Install and start Haveged
- dnf install -y haveged
- systemctl --now enable haveged

# Install Dependencies (OpenJDK 11, PostgreSQL 10)
- dnf install -y postgresql-server java-11-openjdk-devel epel-release

# OpenNMS Horizon, Grafana and Helm
- dnf install -y https://yum.opennms.org/repofiles/opennms-repo-stable-rhel8.noarch.rpm
- dnf install -y jrrd2 opennms-core opennms-webapp-jetty opennms-webapp-hawtio opennms-helm 

# Install and configure certbot
- dnf install -y certbot python3-certbot-nginx
- certbot --nginx -d $FQDN --non-interactive --agree-tos -m $EMAIL

# Configure and start PostgreSQL
- /usr/bin/postgresql-setup --initdb --unit postgresql
- sed -r -i "/^(local|host)/s/(peer|ident)/trust/g" /var/lib/pgsql/data/pg_hba.conf
- systemctl --now enable postgresql

# Configure and start OpenNMS
- sed -r -i '/enabled="false"/{$!{N;s/ enabled="false"[>]\n(.*OpenNMS:Name=Syslogd.*)/>\n\1/}}' /opt/opennms/etc/service-configuration.xml
- /opt/opennms/bin/configure-amq.sh
- echo 'JAVA_HEAP_SIZE=$ONMS_HEAP' > /opt/opennms/etc/opennms.conf
- /opt/opennms/bin/runjava -s
- RUNAS=opennms /opt/opennms/bin/fix-permissions
- /opt/opennms/bin/install -dis
- systemctl --now enable opennms

# Configure and start Grafana
- sed -i -r "s|^;domain =.*|domain = $FQDN|" /etc/grafana/grafana.ini
- sed -i -r "s|^;root_url =.*|root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/|" /etc/grafana/grafana.ini
- systemctl --now enable grafana-server

The environment variables $FQDN, $EMAIL and $ONMS_HEAP must be substituted prior using the template, and we're going to use envsubst for this purpose.

The above expects Horizon 28 or newer. If you're planing to use Horizon 27, make sure to add the following to the template, right before starting OpenNMS:




# Patch OpenNMS due to NMS-13111 to use TLS with AMQ
- rm -f /opt/opennms/lib/bcprov-jdk15on*
- wget -O/opt/opennms/lib/bcprov-jdk15on-169b07.jar https://downloads.bouncycastle.org/betas/bcprov-jdk15on-169b07.jar

Create VM for the applications

If you're not planning to use a shared resource group in Azure, go ahead and create it:

az group create -n $RG_NAME -l $LOCATION --tags Owner=$USER

Create a CentOS 8 VM using the above template:
















envsubst '$FQDN $EMAIL $ONMS_HEAP' < opennms-template.yaml > opennms.yaml

az vm create --resource-group $RG_NAME --name $VM_NAME \
   --size $ONMS_VM_SIZE \
   --image OpenLogic:CentOS:8_4:latest \
   --ssh-key-values ~/.ssh/id_rsa.pub \
   --public-ip-sku Standard \
   --public-ip-address-dns-name $VM_NAME \
   --custom-data /tmp/opennms.yaml \
   --tags Owner=$USER \
   --output table

az vm open-port -g $RG_NAME -n $VM_NAME --port 61616 --priority 100 -o table
az vm open-port -g $RG_NAME -n $VM_NAME --port 443 --priority 110 -o table
az vm open-port -g $RG_NAME -n $VM_NAME --port 80 --priority 120 -o table

Note that for the public FQDN, you only need to specify the name portion. I'm opening port 80, as Certbot requires it for validating the certificate using HTTP. As we're going to be accessing the applications via HTTPS, we need access to port 443. Finally, for testing purposes (if you're interested), I'm opening the ActiveMQ port (port 61616).

In case there is a problem, SSH into the VM using the public IP and the provided credentials and check /var/log/cloud-init-output.log to verify the progress and the status of the cloud-init execution.

If everything goes well, you can access the applications through the following URLs:

OpenNMS: https://agalue-onms01.eastus.cloudapp.azure.com/opennms/

HawtIO: https://agalue-onms01.eastus.cloudapp.azure.com/hawtio/

Grafana: https://agalue-onms01.eastus.cloudapp.azure.com/grafana/

ActiveMQ: failover://ssl://agalue-onms01.eastus.cloudapp.azure.com:61616

You can verify in your browser that the certificate is valid and issues by LetsEncrypt.

In production, it is advised to configure a cron job to renew the certificate at least once monthly.

I recommend checking /etc/nginx/nginx.conf. The Certbot utility modifies that file to add all the SSL/TLS settings required to use the LetsEncrypt certificate with Nginx. This can be beneficial if you want to use a self-signed certificate or a third-party CA to understand how to configure Nginx.

Here are the generated TLS/SSL settings added to nginx.conf:

listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/agalue-onms01.eastus.cloudapp.azure.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/agalue-onms01.eastus.cloudapp.azure.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

Create Minion VMs using multipass

After verifying that OpenNMS is up and running, you can proceed to create the Minions.

Create a cloud-init script to deploy Minion in Ubuntu and save it at /tmp/minion-template.yaml:































#cloud-config
package_upgrade: true

write_files:
  - owner: root:root
    path: /tmp/org.opennms.minion.controller.cfg
    content: |
      location=$MINION_LOCATION
      id=$MINION_ID
      http-url=https://$FQDN/opennms
      broker-url=failover://ssl://$FQDN:61616

apt:
  preserve_sources_list: true
  sources:
    opennms:
      source: deb https://debian.opennms.org stable main

packages:
  - opennms-minion

bootcmd:
  - curl -s https://debian.opennms.org/OPENNMS-GPG-KEY | apt-key add -

runcmd:
  - mv -f /tmp/org.opennms.minion.controller.cfg /etc/minion/
  - sed  -r 's/# export (JAVA_(MIN|MAX)_MEM)=.*/export \1="$MINION_HEAP"/' /etc/default/minion
  - /usr/share/minion/bin/scvcli set opennms.http admin admin
  - /usr/share/minion/bin/scvcli set opennms.broker admin admin
  - systemctl --now enable minion

Note the usage of environment variables within the YAML template. We will substitute them before creating the VM.

Then, start the new Minion via multipass with one core and 2GB of RAM:




envsubst < /tmp/minion-template.yaml > /tmp/$MINION_ID.yaml

multipass launch -c 1 -m 2G -n minion --cloud-init /tmp/$MINION_ID.yaml

Clean Up

To remove all the cloud resources, execute the following:

az group delete -g $RG_NAME -y

If you cannot remove the resource group, as all the resources created will be prefixed with the VM name, the following should work:




















IDS=($(az resource list \
  --resource-group $RG_NAME \
  --query "[?contains(name,'$VM_NAME') && type!='Microsoft.Compute/disks']".id \
  --output tsv | tr '\n' ' '))

for id in "${IDS[@]}"; do
  echo "Removing $id"
  az resource delete --ids "$id" --verbose
done

DISKS=($(az resource list \
  --resource-group $RG_NAME \
  --query "[?contains(name,'$VM_NAME') && type=='Microsoft.Compute/disks']".id \
  --output tsv | tr '\n' ' '))

for id in "${DISKS[@]}"; do
  echo "Removing $id"
  az resource delete --ids "$id" --verbose
done

Then clean the local resources:



multipass delete $MINION_ID
multipass purge
Last changed by 
 
Alejandro Galue
0
751

Read more

Sizing Cassandra

The use case for having Cassandra (or ScyllaDB) as the backend for the performance metrics in OpenNMS is for storing a huge amount of non-aggregated data, which is not possible with RRDtool. RRDtool is very good for those installations with a finite and predictable amount of metrics where the size and the I/O requirements are feasible for modern SSD disks. This is important as RRDtool only scales vertically, meaning that when the current disks&apos; limits are reached (mostly due to speed, not space), a faster disk is required. This is when Cassandra or ScyllaDB can help. Although there is a high learning curve, using any of these applications requires a commitment to have qualified personnel to manage this database. ScyllaDB is binary compatible with Cassandra (even at the SSTable level), but they are implemented very differently. Cassandra is implemented in Java, meaning that all the JVM tuning is required and other internal Cassandra tunings. On the other hand, ScyllaDB is implemented in modern C++, taking advantage of the CPU where it is running. That means ScyllaDB can manage huge machines as nodes, whereas Cassandra would require multiple instances on huge machines. It is feasible to have faster results with ScyllaDB compared with its Java sibling in terms of performance. Configuring and managing both applications is different, even if they will provide the same operational result with OpenNMS, so the decision should be carefully analyzed, especially by the team that will support this database.

May 4, 2022
Simple OpenNMS Environment using ScyllaDB

This lab starts an OpenNMS instance, a 3-node ScyllaDB cluster, and optionally an instance for the Scylla Monitoring Stack in the cloud, for learning purposes. To monitor a network, it is advised to enable ActiveMQ or Kafka and use Minions. For simplicity, the embedded ActiveMQ will be enabled. :::success Follow this guide for general information about how to configure Scylla/Cassandra for OpenNMS/Newts. Most of that knowledge is transferable to ScyllaDB. ::: Requirements

Nov 19, 2021
Simple OpenNMS/Minion Environment using Kafka in Azure

This lab starts an OpenNMS instance and a 3 node ZK/Kafka cluster in the cloud and two Minions on your machine, using Kafka for communication through Multipass and Azure, for learning purposes. :::warning The lab doesn&apos;t cover security by default (user authentication and encryption), which is crucial if we ever want to expose the Kafka cluster to the Internet. A separate section covers the required changes for this. ::: :::success Keep in mind that nothing prevents us from skipping using the cloud provider and do everything with Multipass (or VirtualBox, or Hyper-V, or VMWare). The reason for using a cloud provider is to prove that OpenNMS can monitor unreachable devices via Minion. Similarly, we could use any other cloud provider instead of Azure. However I won&apos;t explain how to port the solution here. :::

Nov 19, 2021
Simple OpenNMS/Minion Environment using the embedded ActiveMQ in Azure

This lab starts an OpenNMS instance in the cloud and two Minions on your machine, using ActiveMQ for communication through Multipass and Azure, for learning purposes. :::danger The lab doesn&apos;t cover security (in terms of encryption), which is crucial if you ever want to expose AMQ to the Internet. Follow this guide to learn how to do it with LetsEncrypt. ::: :::success Keep in mind that nothing prevents you from skipping using the cloud provider and do everything with Multipass (or VirtualBox, or Hyper-V, or VMWare). The reason for using a cloud provider is to prove that OpenNMS can monitor unreachable devices via Minion. Similarly, you could use any other cloud provider instead of Azure. However I won&apos;t explain how to port the solution here. :::

Nov 19, 2021
Read more from Alejandro Galue
Published on HackMD