Simple OpenNMS/Minion Environment using the embedded ActiveMQ in Azure

This lab starts an OpenNMS instance in the cloud and two Minions on your machine, using ActiveMQ for communication through Multipass and Azure, for learning purposes.

The lab doesn't cover security (in terms of encryption), which is crucial if you ever want to expose AMQ to the Internet. Follow this guide to learn how to do it with LetsEncrypt.

Keep in mind that nothing prevents you from skipping using the cloud provider and do everything with Multipass (or VirtualBox, or Hyper-V, or VMWare). The reason for using a cloud provider is to prove that OpenNMS can monitor unreachable devices via Minion. Similarly, you could use any other cloud provider instead of Azure. However I won't explain how to port the solution here.

Requirements

• Have an Azure Subscription ready.
• Install Azure CLI
• Install Multipass

The scripts used through this tutorial use envsubst, make sure to have it installed.

Make sure to log into Azure using az login prior creating the VM.

If you have a restricted account in Azure, make sure you have the Network Contributor role and the Virtual Machine Contributor role associated with your Azure AD account for the resource group on which you would like to create the VM. Of course, Owner or Contributor at resource group level are welcome.

Tune the VMs image size accordingly.

Create common Environment Variables

export RG_NAME="OpenNMS" # Change it if you're using a shared one
export LOCATION="eastus" # Azure Region
export DOMAIN="$LOCATION.cloudapp.azure.com" # Public Azure DNS Domain
export ONMS_VM_NAME="$USER-onms01"
export ONMS_VM_SIZE="Standard_D2s_v3" # 2 VCPU, 8 GB of RAM
export ONMS_FQDN="$ONMS_VM_NAME.$DOMAIN"
export ONMS_HEAP_SIZE="4096" # Expressed in MB and must fit ONMS_VM_SIZE
export MINION_LOCATION="Durham"
export MINION_HEAP_SIZE="1G" # Must fit VM RAM
export MINION_ID1="minion01"
export MINION_ID2="minion02"

Feel free to change the content if needed.

Do not confuse the Azure Location or Region with the Minion Location; they are both unrelated things.

We're going to leverage the Azure DNS services to avoid the need to remember and using Public IP addresses, which helps if you're interested in having HTTPS with valid certificates as explained here.

In Azure, the default public DNS follow the same pattern:

<vm-name>.<location>.cloudapp.azure.com

To make the VM for OpenNMS unique, I added the username to the VM name, as you can see in the ONMS_FQDN environment variable, which translates to:

agalue-onms01.eastus.cloudapp.azure.com

The above is what we can use to access the VM via SSH and to configure Minions.

Create the Azure Resource Group

This is a necessary step, as every resource in Azure must belong to a resource group and a location.

However, you can omit the following command and use an existing one if you prefer. In that case, make sure to adjust the environment variable RG_NAME so the subsequent commands will target the correct group.

az group create -n $RG_NAME -l $LOCATION --tags Owner=$USER

Create an Azure VM for OpenNMS

Create a cloud-init script to deploy OpenNMS in Ubuntu and save it at /tmp/opennms-template.yaml:

#cloud-config
package_upgrade: true

write_files:
  - owner: root:root
    path: /etc/opennms-overlay/opennms.properties.d/rrd.properties
    content: |
      org.opennms.rrd.storeByGroup=true
      org.opennms.rrd.storeByForeignSource=true
      org.opennms.rrd.strategyClass=org.opennms.netmgt.rrd.rrdtool.MultithreadedJniRrdStrategy
      org.opennms.rrd.interfaceJar=/usr/share/java/jrrd2.jar
      opennms.library.jrrd2=/usr/lib/jni/libjrrd2.so

  - owner: root:root
    path: /etc/opennms-overlay/opennms.properties.d/amq.properties
    # ActiveMQ tuning example
    content: |
      org.opennms.jms.timeout=120000
      org.opennms.activemq.client.max-connections=8
      org.opennms.activemq.client.concurrent-consumers=16
      org.opennms.ipc.rpc.threads=1
      org.opennms.ipc.rpc.queue.max=50000
      org.opennms.ipc.rpc.threads.max=100

  - owner: root:root
    permissions: '0400'
    path: /etc/snmp/snmpd.conf
    content: |
      rocommunity public default
      syslocation Azure - $LOCATION
      syscontact $USER
      dontLogTCPWrappersConnects yes
      disk /

apt:
  preserve_sources_list: true
  sources:
    opennms:
      source: deb https://debian.opennms.org stable main

packages:
  - snmp
  - snmpd
  - jrrd2
  - opennms
  - opennms-webapp-hawtio

bootcmd:
  - curl -s https://debian.opennms.org/OPENNMS-GPG-KEY | apt-key add -

runcmd:
  # Configure PostgreSQL
  - systemctl --now enable postgresql
  - sudo -u postgres createuser opennms
  - sudo -u postgres psql -c "ALTER USER postgres WITH PASSWORD 'postgres';"
  - sudo -u postgres psql -c "ALTER USER opennms WITH PASSWORD 'opennms';"
  - sed -r -i 's/password=""/password="postgres"/' /etc/opennms/opennms-datasources.xml
  # Enable and tune ActiveMQ
  - sed -r -i '/0.0.0.0:61616/s/([<][!]--|--[>])//g' /etc/opennms/opennms-activemq.xml
  - sed -r -i '/memoryLimit/s/1mb/10mb/' /etc/opennms/opennms-activemq.xml
  - sed -r -i '/memoryUsage/s/20 mb/512 mb/' /etc/opennms/opennms-activemq.xml
  - sed -r -i '/storeUsage/s/1 gb/4 gb/' /etc/opennms/opennms-activemq.xml
  - sed -r -i '/tempUsage/s/100 mb/1 gb/' /etc/opennms/opennms-activemq.xml
  # Enable Syslogd
  - sed -r -i '/enabled="false"/{$!{N;s/ enabled="false"[>]\n(.*OpenNMS:Name=Syslogd.*)/>\n\1/}}' /etc/opennms/service-configuration.xml
  # Initialize and start OpenNMS
  - rsync -avr /etc/opennms-overlay/ /etc/opennms/
  - echo 'JAVA_HEAP_SIZE=$ONMS_HEAP_SIZE' > /etc/opennms/opennms.conf
  - /usr/share/opennms/bin/runjava -s
  - /usr/share/opennms/bin/fix-permissions
  - /usr/share/opennms/bin/install -dis
  - systemctl --now enable opennms

Note the usage of environment variables within the YAML template. We will substitute them before creating the VM.

The above installs the latest OpenJDK 11, the latest PostgreSQL, and the latest OpenNMS Horizon. I added the most basic configuration for PostgreSQL to work with authentication. The embedded ActiveMQ is enabled, as well as Syslogd.

Create an Ubuntu VM with 2 Cores an 8GB of RAM for OpenNMS (that's what you'd get with Standard_D2s_v3):

envsubst < /tmp/opennms-template.yaml > /tmp/opennms.yaml

az vm create --resource-group $RG_NAME --name $ONMS_VM_NAME \
 --size $ONMS_VM_SIZE \
 --image UbuntuLTS \
 --admin-username $USER \
 --ssh-key-values ~/.ssh/id_rsa.pub \
 --public-ip-address-dns-name $ONMS_VM_NAME \
 --custom-data /tmp/opennms.yaml \
 --tags Owner=$USER \
 --output table

az vm open-port -g $RG_NAME -n $ONMS_VM_NAME \
  --port 61616 --priority 100 \
  --output table

az vm open-port -g $RG_NAME -n $ONMS_VM_NAME \
  --port 8980 --priority 200 \
  --output table

Note that I'm assuming the usage of SSH Keys for password-less access. Make sure to have a public key located at ~/.ssh/id_rsa.pub, or update the az vm create command.

By default, the above creates a VNet, a default subnet within it, and it associates a public IP address and a public DNS name to the NIC that Azure will create for the VM.

I've chosen password-based access for simplicity but feel free to use SSH-Keys if needed.

Keep in mind that the cloud-init process starts after the VM is running, meaning you should wait about 5 minutes after the az vm create is finished to see OpenNMS up and running.

The last two commands expose the WebUI and AMQ ports to the Internet via the auto-generated Azure Security Group for the VM associated with its NIC.

In case there is a problem, SSH into the VM using the public IP and the provided credentials and check /var/log/cloud-init-output.log to verify the progress and the status of the cloud-init execution.

Create Minion VMs using multipass

After verifying that OpenNMS is up and running, you can proceed to create the Minions.

Create a cloud-init script to deploy Minion in Ubuntu and save it at /tmp/minion-template.yaml:

#cloud-config
package_upgrade: true

write_files:
  - owner: root:root
    path: /tmp/org.opennms.minion.controller.cfg
    content: |
      location=$MINION_LOCATION
      id=$MINION_ID
      http-url=http://$ONMS_FQDN:8980/opennms
      broker-url=failover:tcp://$ONMS_FQDN:61616

apt:
  preserve_sources_list: true
  sources:
    opennms:
      source: deb https://debian.opennms.org stable main

packages:
  - opennms-minion

bootcmd:
  - curl -s https://debian.opennms.org/OPENNMS-GPG-KEY | apt-key add -

runcmd:
  - mv -f /tmp/org.opennms.minion.controller.cfg /etc/minion/
  - sed -i -r 's/# export JAVA_MIN_MEM=.*/export JAVA_MIN_MEM="$MINION_HEAP_SIZE"/' /etc/default/minion
  - sed -i -r 's/# export JAVA_MAX_MEM=.*/export JAVA_MAX_MEM="$MINION_HEAP_SIZE"/' /etc/default/minion
  - /usr/share/minion/bin/scvcli set opennms.http admin admin
  - /usr/share/minion/bin/scvcli set opennms.broker admin admin
  - systemctl --now enable minion

Note the usage of environment variables within the YAML template. We will substitute them before creating the VM.

Then, create the runtime template:

export MINION_ID=$MINION_ID1

envsubst < /tmp/minion-template.yaml > /tmp/$MINION_ID1.yaml

Then, start the new Minion via multipass with one core and 2GB of RAM:

multipass launch -c 1 -m 2G -n $MINION_ID1 --cloud-init /tmp/$MINION_ID1.yaml

Optionally, create a second Minion in the same location:

export MINION_ID=$MINION_ID2

envsubst < /tmp/minion-template.yaml > /tmp/$MINION_ID2.yaml

multipass launch -c 1 -m 2G -n $MINION_ID2 --cloud-init /tmp/$MINION_ID2.yaml

In case there is a problem, access the VM (e.x., multipass shell $MINION_ID1) and check /var/log/cloud-init-output.log to verify the progress and the status of the cloud-init execution.

Test

As you can see, the location name is Durham (a.k.a. $MINION_LOCATION), and you should see the Minions on that location registered in OpenNMS.

SSH into the OpenNMS server and create a requisition with a node in the same network as the Minion VMs, and make sure to associate it with the appropriate location. For instance,

/usr/share/opennms/bin/provision.pl requisition add Test
/usr/share/opennms/bin/provision.pl node add Test srv01 srv01
/usr/share/opennms/bin/provision.pl node set Test srv01 location Durham
/usr/share/opennms/bin/provision.pl interface add Test srv01 192.168.0.40
/usr/share/opennms/bin/provision.pl interface set Test srv01 192.168.0.40 snmp-primary P
/usr/share/opennms/bin/provision.pl requisition import Test

Make sure to replace 192.168.0.40 with the IP of a working server in your network (reachable from the Minion VM), and do not forget to use the same location as defined in $MINION_LOCATION.

Please keep in mind that Minions are VMs on your machine. 192.168.0.40 is the IP of my machine which is why Minions can reach it (and vice versa), to access an external machine on your network, make sure to define static routes on that machine so it can reach the Minions through your machine (assuming you're running Linux or macOS).

OpenNMS which runs in Azure, and have no access to 192.168.0.40 directly, should be able to collect data and monitor that node through any of the Minions. In fact, you can stop one of them, and OpenNMS would continue monitoring it.

To test asynchronous messages, you can send SNMP traps or Syslog messages to one of the Minions. Usually, you could put a Load Balancer in front of the Minions and use its IP when sending messages from the monitored devices. Alternatively, you could use udpgen for this purpose.

The machine that will be running udpgen must be part of the OpenNMS inventory. Then, find the IP of the Minion using multipass list, then execute the following from the machine added as a node above (the examples assumes the IP of the Minion is 192.168.75.16):

To send SNMP Traps:

udpgen -h 192.168.75.16 -x snmp -r 1 -p 1162

To send Syslog Messages:

udpgen -h 192.168.75.16 -x syslog -r 1 -p 1514

The C++ version of udpgen only works on Linux. If you're on MacOS or Windows, you can use the Go version of it.

Note that an event definition is required when using udpgen to send traps. Here is what you'd need for Eventd:

<events xmlns="http://xmlns.opennms.org/xsd/eventconf">
   <event>
      <mask>
         <maskelement>
            <mename>id</mename>
            <mevalue>.1.3.6.1.1.6.3.1.1.5</mevalue>
         </maskelement>
         <maskelement>
            <mename>generic</mename>
            <mevalue>6</mevalue>
         </maskelement>
         <maskelement>
            <mename>specific</mename>
            <mevalue>1</mevalue>
         </maskelement>
      </mask>
      <uei>uei.opennms.org/udpgen/testTrap</uei>
      <event-label>udpgen test trap</event-label>
      <descr>Sample Event %parm[all]%</descr>
      <logmsg dest="logndisplay">Sample Event %parm[all]%</logmsg>
      <severity>Warning</severity>
   </event>
</events>

The Hawtio UI in OpenNMS can help to visualize the Camel and ActiveMQ internals, to understand what's circulating between OpenNMS and the Minions.

For OpenNMS, Hawtio is available through http://$ONMS_IP:8980/hawtio (use the ActiveMQ Tab) if the package opennms-webapp-hawtio was installed (which is the case with the cloud-init template used). Alternatively, you could use the opennms-activemq:stats command from the Karaf Shell.

For Minions, Hawtio is available through http://$MINION_IP1:8181/hawtio and http://$MINION_IP2:8181/hawtio respectively (use the Camel Tab).

Add a Load Balancer in front of the Minions (Optional)

In production, when having multiple Minions per location, it is a good practice to put a Load Balancer in front of them so that the devices can use a single destination for SNMP Traps, Syslog, and Flows.

The following creates a cloud-init template for Ubuntu to start a basic LB using nginx through multipass for SNMP Traps (with a listener on port 162) and Syslog Messages (with a listener on port 514). Save the template at /tmp/nginx-template.yaml:

#cloud-config
package_upgrade: true

packages:
  - nginx

write_files:
  - owner: root:root
    path: /etc/nginx/nginx.conf
    content: |
      user www-data;
      worker_processes auto;
      pid /run/nginx.pid;
      include /etc/nginx/modules-enabled/*.conf;
      events {
        worker_connections 768;
      }
      stream {
        upstream syslog_udp  {
          server $MINION_IP1:1514;
          server $MINION_IP2:1514;
        }
        upstream trap_udp  {
          server $MINION_IP1:1162;
          server $MINION_IP2:1162;
        }
        server {
          listen 514 udp;
          proxy_pass syslog_udp;
          proxy_responses 0;
        }
        server {
          listen 162 udp;
          proxy_pass trap_udp;
          proxy_responses 0;
        }
      }

runcmd:
  - systemctl restart nginx

Note the usage of environment variables within the YAML template. We will substitute them before creating the VM.

Then, update the template and create the LB:

export MINION_IP1=$(multipass info $MINION_ID1 | grep IPv4 | awk '{print $2}')
export MINION_IP2=$(multipass info $MINION_ID2 | grep IPv4 | awk '{print $2}')

envsubst < /tmp/nginx-template.yaml > /tmp/nginx.yaml

multipass launch -n nginx --cloud-init /tmp/nginx.yaml
echo "Load Balancer $(multipass info nginx | grep IPv4)"

Flows are outside the scope of this test as that requires more configuration on Minions and OpenNMS besides having an Elasticsearch cluster up and running with the required plugin in place.

Clean Up

When you're done, make sure to delete the cloud resources.

If you created the resource group for this exercise, you could remove all the resources with the following command:

az group delete -g $RG_NAME

If you're using an existing resource group that you cannot remove, make sure only to remove all the resources created in this tutorial. All of them should be easily identified as they will contain the username and the VM name as part of the resource name. The easiest way is to use the Azure Portal for this operation. Alternatively,

IDS=($(az resource list \
  --resource-group $RG_NAME \
  --query "[?contains(name,'$USER-') && type!='Microsoft.Compute/disks']".id \
  --output tsv | tr '\n' ' '))

for id in "${IDS[@]}"; do
  echo "Removing $id"
  az resource delete --ids "$id" --verbose
done

DISKS=($(az resource list \
  --resource-group $RG_NAME \
  --query "[?contains(name,'$USER-') && type=='Microsoft.Compute/disks']".id \
  --output tsv | tr '\n' ' '))

for id in "${DISKS[@]}"; do
  echo "Removing $id"
  az resource delete --ids "$id" --verbose
done

The reason to have two sets of deletion groups is that, by default, the list contains disks initially, which cannot be removed before the VMs. For this reason, we exclude the disks on the first set, and then we remove the disks.

Then clean the local resources:

multipass delete $MINION_ID1 $MINION_ID2
multipass purge

Remember to remove the nginx instance if you decided to use it.