# Simple OpenNMS/Minion Environment using the embedded ActiveMQ in Azure This lab starts an OpenNMS instance in the cloud and two Minions on your machine, using ActiveMQ for communication through `Multipass` and `Azure`, for learning purposes. :::danger The lab doesn't cover security (in terms of encryption), which is crucial if you ever want to expose AMQ to the Internet. Follow [this](https://hackmd.io/@agalue/HyGyD0diN) guide to learn how to do it with LetsEncrypt. ::: ![](https://i.imgur.com/Qe428AO.png) :::success Keep in mind that nothing prevents you from skipping using the cloud provider and do everything with `Multipass` (or `VirtualBox`, or `Hyper-V`, or `VMWare`). The reason for using a cloud provider is to prove that OpenNMS can monitor unreachable devices via Minion. Similarly, you could use any other cloud provider instead of Azure. However I won't explain how to port the solution here. ::: ## Requirements * Have an [Azure Subscription](https://azure.microsoft.com/en-us/free/) ready. * Install [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) * Install [Multipass](https://multipass.run/) The scripts used through this tutorial use [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html), make sure to have it installed. :::info Make sure to log into Azure using `az login` prior creating the VM. ::: :::danger If you have a restricted account in Azure, make sure you have the `Network Contributor` role and the `Virtual Machine Contributor` role associated with your Azure AD account for the resource group on which you would like to create the VM. Of course, `Owner` or `Contributor` at resource group level are welcome. Tune the VMs image size accordingly. ::: ## Create common Environment Variables ```bash= export RG_NAME="OpenNMS" # Change it if you're using a shared one export LOCATION="eastus" # Azure Region export DOMAIN="$LOCATION.cloudapp.azure.com" # Public Azure DNS Domain export ONMS_VM_NAME="$USER-onms01" export ONMS_VM_SIZE="Standard_D2s_v3" # 2 VCPU, 8 GB of RAM export ONMS_FQDN="$ONMS_VM_NAME.$DOMAIN" export ONMS_HEAP_SIZE="4096" # Expressed in MB and must fit ONMS_VM_SIZE export MINION_LOCATION="Durham" export MINION_HEAP_SIZE="1G" # Must fit VM RAM export MINION_ID1="minion01" export MINION_ID2="minion02" ``` :::info Feel free to change the content if needed. ::: :::danger Do not confuse the Azure Location or Region with the Minion Location; they are both unrelated things. ::: We're going to leverage the Azure DNS services to avoid the need to remember and using Public IP addresses, which helps if you're interested in having HTTPS with valid certificates as explained [here](https://hackmd.io/@agalue/HyGyD0diN). In Azure, the default public DNS follow the same pattern: ``` <vm-name>.<location>.cloudapp.azure.com ``` To make the VM for OpenNMS unique, I added the username to the VM name, as you can see in the `ONMS_FQDN` environment variable, which translates to: ``` agalue-onms01.eastus.cloudapp.azure.com ``` The above is what we can use to access the VM via SSH and to configure Minions. ## Create the Azure Resource Group This is a necessary step, as every resource in Azure must belong to a resource group and a location. However, you can omit the following command and use an existing one if you prefer. In that case, make sure to adjust the environment variable `RG_NAME` so the subsequent commands will target the correct group. ```bash= az group create -n $RG_NAME -l $LOCATION --tags Owner=$USER ``` ## Create an Azure VM for OpenNMS Create a [cloud-init](https://cloudinit.readthedocs.io/en/latest/) script to deploy OpenNMS in Ubuntu and save it at `/tmp/opennms-template.yaml`: ```yaml= #cloud-config package_upgrade: true write_files: - owner: root:root path: /etc/opennms-overlay/opennms.properties.d/rrd.properties content: | org.opennms.rrd.storeByGroup=true org.opennms.rrd.storeByForeignSource=true org.opennms.rrd.strategyClass=org.opennms.netmgt.rrd.rrdtool.MultithreadedJniRrdStrategy org.opennms.rrd.interfaceJar=/usr/share/java/jrrd2.jar opennms.library.jrrd2=/usr/lib/jni/libjrrd2.so - owner: root:root path: /etc/opennms-overlay/opennms.properties.d/amq.properties # ActiveMQ tuning example content: | org.opennms.jms.timeout=120000 org.opennms.activemq.client.max-connections=8 org.opennms.activemq.client.concurrent-consumers=16 org.opennms.ipc.rpc.threads=1 org.opennms.ipc.rpc.queue.max=50000 org.opennms.ipc.rpc.threads.max=100 - owner: root:root permissions: '0400' path: /etc/snmp/snmpd.conf content: | rocommunity public default syslocation Azure - $LOCATION syscontact $USER dontLogTCPWrappersConnects yes disk / apt: preserve_sources_list: true sources: opennms: source: deb https://debian.opennms.org stable main packages: - snmp - snmpd - jrrd2 - opennms - opennms-webapp-hawtio bootcmd: - curl -s https://debian.opennms.org/OPENNMS-GPG-KEY | apt-key add - runcmd: # Configure PostgreSQL - systemctl --now enable postgresql - sudo -u postgres createuser opennms - sudo -u postgres psql -c "ALTER USER postgres WITH PASSWORD 'postgres';" - sudo -u postgres psql -c "ALTER USER opennms WITH PASSWORD 'opennms';" - sed -r -i 's/password=""/password="postgres"/' /etc/opennms/opennms-datasources.xml # Enable and tune ActiveMQ - sed -r -i '/0.0.0.0:61616/s/([<][!]--|--[>])//g' /etc/opennms/opennms-activemq.xml - sed -r -i '/memoryLimit/s/1mb/10mb/' /etc/opennms/opennms-activemq.xml - sed -r -i '/memoryUsage/s/20 mb/512 mb/' /etc/opennms/opennms-activemq.xml - sed -r -i '/storeUsage/s/1 gb/4 gb/' /etc/opennms/opennms-activemq.xml - sed -r -i '/tempUsage/s/100 mb/1 gb/' /etc/opennms/opennms-activemq.xml # Enable Syslogd - sed -r -i '/enabled="false"/{$!{N;s/ enabled="false"[>]\n(.*OpenNMS:Name=Syslogd.*)/>\n\1/}}' /etc/opennms/service-configuration.xml # Initialize and start OpenNMS - rsync -avr /etc/opennms-overlay/ /etc/opennms/ - echo 'JAVA_HEAP_SIZE=$ONMS_HEAP_SIZE' > /etc/opennms/opennms.conf - /usr/share/opennms/bin/runjava -s - /usr/share/opennms/bin/fix-permissions - /usr/share/opennms/bin/install -dis - systemctl --now enable opennms ``` :::info Note the usage of environment variables within the YAML template. We will substitute them before creating the VM. ::: The above installs the latest OpenJDK 11, the latest PostgreSQL, and the latest OpenNMS Horizon. I added the most basic configuration for PostgreSQL to work with authentication. The embedded ActiveMQ is enabled, as well as `Syslogd`. Create an Ubuntu VM with 2 Cores an 8GB of RAM for OpenNMS (that's what you'd get with [Standard_D2s_v3](https://docs.microsoft.com/en-us/azure/virtual-machines/dv3-dsv3-series)): ```bash= envsubst < /tmp/opennms-template.yaml > /tmp/opennms.yaml az vm create --resource-group $RG_NAME --name $ONMS_VM_NAME \ --size $ONMS_VM_SIZE \ --image UbuntuLTS \ --admin-username $USER \ --ssh-key-values ~/.ssh/id_rsa.pub \ --public-ip-address-dns-name $ONMS_VM_NAME \ --custom-data /tmp/opennms.yaml \ --tags Owner=$USER \ --output table az vm open-port -g $RG_NAME -n $ONMS_VM_NAME \ --port 61616 --priority 100 \ --output table az vm open-port -g $RG_NAME -n $ONMS_VM_NAME \ --port 8980 --priority 200 \ --output table ``` :::info Note that I'm assuming the usage of SSH Keys for password-less access. Make sure to have a public key located at `~/.ssh/id_rsa.pub`, or update the `az vm create` command. ::: By default, the above creates a VNet, a default subnet within it, and it associates a public IP address and a public DNS name to the NIC that Azure will create for the VM. I've chosen password-based access for simplicity but feel free to use SSH-Keys if needed. Keep in mind that the `cloud-init` process starts after the VM is running, meaning you should wait about 5 minutes after the `az vm create` is finished to see OpenNMS up and running. The last two commands expose the WebUI and AMQ ports to the Internet via the auto-generated Azure Security Group for the VM associated with its NIC. :::warning In case there is a problem, SSH into the VM using the public IP and the provided credentials and check `/var/log/cloud-init-output.log` to verify the progress and the status of the cloud-init execution. ::: ## Create Minion VMs using `multipass` After verifying that OpenNMS is up and running, you can proceed to create the Minions. Create a [cloud-init](https://cloudinit.readthedocs.io/en/latest/) script to deploy Minion in Ubuntu and save it at `/tmp/minion-template.yaml`: ```yaml= #cloud-config package_upgrade: true write_files: - owner: root:root path: /tmp/org.opennms.minion.controller.cfg content: | location=$MINION_LOCATION id=$MINION_ID http-url=http://$ONMS_FQDN:8980/opennms broker-url=failover:tcp://$ONMS_FQDN:61616 apt: preserve_sources_list: true sources: opennms: source: deb https://debian.opennms.org stable main packages: - opennms-minion bootcmd: - curl -s https://debian.opennms.org/OPENNMS-GPG-KEY | apt-key add - runcmd: - mv -f /tmp/org.opennms.minion.controller.cfg /etc/minion/ - sed -i -r 's/# export JAVA_MIN_MEM=.*/export JAVA_MIN_MEM="$MINION_HEAP_SIZE"/' /etc/default/minion - sed -i -r 's/# export JAVA_MAX_MEM=.*/export JAVA_MAX_MEM="$MINION_HEAP_SIZE"/' /etc/default/minion - /usr/share/minion/bin/scvcli set opennms.http admin admin - /usr/share/minion/bin/scvcli set opennms.broker admin admin - systemctl --now enable minion ``` :::info Note the usage of environment variables within the YAML template. We will substitute them before creating the VM. ::: Then, create the runtime template: ```bash= export MINION_ID=$MINION_ID1 envsubst < /tmp/minion-template.yaml > /tmp/$MINION_ID1.yaml ``` Then, start the new Minion via `multipass` with one core and 2GB of RAM: ```bash= multipass launch -c 1 -m 2G -n $MINION_ID1 --cloud-init /tmp/$MINION_ID1.yaml ``` Optionally, create a second Minion in the same location: ```bash= export MINION_ID=$MINION_ID2 envsubst < /tmp/minion-template.yaml > /tmp/$MINION_ID2.yaml multipass launch -c 1 -m 2G -n $MINION_ID2 --cloud-init /tmp/$MINION_ID2.yaml ``` :::warning In case there is a problem, access the VM (e.x., `multipass shell $MINION_ID1`) and check `/var/log/cloud-init-output.log` to verify the progress and the status of the cloud-init execution. ::: ## Test As you can see, the location name is `Durham` (a.k.a. `$MINION_LOCATION`), and you should see the Minions on that location registered in OpenNMS. SSH into the OpenNMS server and create a requisition with a node in the same network as the Minion VMs, and make sure to associate it with the appropriate location. For instance, ```bash= /usr/share/opennms/bin/provision.pl requisition add Test /usr/share/opennms/bin/provision.pl node add Test srv01 srv01 /usr/share/opennms/bin/provision.pl node set Test srv01 location Durham /usr/share/opennms/bin/provision.pl interface add Test srv01 192.168.0.40 /usr/share/opennms/bin/provision.pl interface set Test srv01 192.168.0.40 snmp-primary P /usr/share/opennms/bin/provision.pl requisition import Test ``` :::warning Make sure to replace `192.168.0.40` with the IP of a working server in your network (reachable from the Minion VM), and do not forget to use the same location as defined in `$MINION_LOCATION`. ::: :::danger Please keep in mind that Minions are VMs on your machine. `192.168.0.40` is the IP of my machine which is why Minions can reach it (and vice versa), to access an external machine on your network, make sure to define static routes on that machine so it can reach the Minions through your machine (assuming you're running Linux or macOS). ::: OpenNMS which runs in Azure, and have no access to `192.168.0.40` directly, should be able to collect data and monitor that node through any of the Minions. In fact, you can stop one of them, and OpenNMS would continue monitoring it. To test asynchronous messages, you can send SNMP traps or Syslog messages to one of the Minions. Usually, you could put a Load Balancer in front of the Minions and use its IP when sending messages from the monitored devices. Alternatively, you could use [udpgen](https://github.com/OpenNMS/udpgen) for this purpose. The machine that will be running `udpgen` must be part of the OpenNMS inventory. Then, find the IP of the Minion using `multipass list`, then execute the following from the machine added as a node above (the examples assumes the IP of the Minion is `192.168.75.16`): To send SNMP Traps: ```bash= udpgen -h 192.168.75.16 -x snmp -r 1 -p 1162 ``` To send Syslog Messages: ```bash= udpgen -h 192.168.75.16 -x syslog -r 1 -p 1514 ``` :::success The C++ version of `udpgen` only works on Linux. If you're on MacOS or Windows, you can use the [Go](https://github.com/agalue/udpgen) version of it. ::: Note that an event definition is required when using `udpgen` to send traps. Here is what you'd need for `Eventd`: ```xml= <events xmlns="http://xmlns.opennms.org/xsd/eventconf"> <event> <mask> <maskelement> <mename>id</mename> <mevalue>.1.3.6.1.1.6.3.1.1.5</mevalue> </maskelement> <maskelement> <mename>generic</mename> <mevalue>6</mevalue> </maskelement> <maskelement> <mename>specific</mename> <mevalue>1</mevalue> </maskelement> </mask> <uei>uei.opennms.org/udpgen/testTrap</uei> <event-label>udpgen test trap</event-label> <descr>Sample Event %parm[all]%</descr> <logmsg dest="logndisplay">Sample Event %parm[all]%</logmsg> <severity>Warning</severity> </event> </events> ``` The [Hawtio](https://hawt.io/) UI in OpenNMS can help to visualize the Camel and ActiveMQ internals, to understand what's circulating between OpenNMS and the Minions. For OpenNMS, Hawtio is available through `http://$ONMS_IP:8980/hawtio` (use the ActiveMQ Tab) if the package `opennms-webapp-hawtio` was installed (which is the case with the `cloud-init` template used). Alternatively, you could use the `opennms-activemq:stats` command from the Karaf Shell. For Minions, Hawtio is available through `http://$MINION_IP1:8181/hawtio` and `http://$MINION_IP2:8181/hawtio` respectively (use the Camel Tab). ## Add a Load Balancer in front of the Minions (Optional) In production, when having multiple Minions per location, it is a good practice to put a Load Balancer in front of them so that the devices can use a single destination for SNMP Traps, Syslog, and Flows. The following creates a [cloud-init](https://cloudinit.readthedocs.io/en/latest/) template for Ubuntu to start a basic LB using `nginx` through `multipass` for SNMP Traps (with a listener on port 162) and Syslog Messages (with a listener on port 514). Save the template at /tmp/nginx-template.yaml: ```yaml= #cloud-config package_upgrade: true packages: - nginx write_files: - owner: root:root path: /etc/nginx/nginx.conf content: | user www-data; worker_processes auto; pid /run/nginx.pid; include /etc/nginx/modules-enabled/*.conf; events { worker_connections 768; } stream { upstream syslog_udp { server $MINION_IP1:1514; server $MINION_IP2:1514; } upstream trap_udp { server $MINION_IP1:1162; server $MINION_IP2:1162; } server { listen 514 udp; proxy_pass syslog_udp; proxy_responses 0; } server { listen 162 udp; proxy_pass trap_udp; proxy_responses 0; } } runcmd: - systemctl restart nginx ``` :::info Note the usage of environment variables within the YAML template. We will substitute them before creating the VM. ::: Then, update the template and create the LB: ```bash= export MINION_IP1=$(multipass info $MINION_ID1 | grep IPv4 | awk '{print $2}') export MINION_IP2=$(multipass info $MINION_ID2 | grep IPv4 | awk '{print $2}') envsubst < /tmp/nginx-template.yaml > /tmp/nginx.yaml multipass launch -n nginx --cloud-init /tmp/nginx.yaml echo "Load Balancer $(multipass info nginx | grep IPv4)" ``` :::warning Flows are outside the scope of this test as that requires more configuration on Minions and OpenNMS besides having an Elasticsearch cluster up and running with the required plugin in place. ::: ## Clean Up When you're done, make sure to delete the cloud resources. If you created the resource group for this exercise, you could remove all the resources with the following command: ```bash= az group delete -g $RG_NAME ``` If you're using an existing resource group that you cannot remove, make sure only to remove all the resources created in this tutorial. All of them should be easily identified as they will contain the username and the VM name as part of the resource name. The easiest way is to use the Azure Portal for this operation. Alternatively, ```bash= IDS=($(az resource list \ --resource-group $RG_NAME \ --query "[?contains(name,'$USER-') && type!='Microsoft.Compute/disks']".id \ --output tsv | tr '\n' ' ')) for id in "${IDS[@]}"; do echo "Removing $id" az resource delete --ids "$id" --verbose done DISKS=($(az resource list \ --resource-group $RG_NAME \ --query "[?contains(name,'$USER-') && type=='Microsoft.Compute/disks']".id \ --output tsv | tr '\n' ' ')) for id in "${DISKS[@]}"; do echo "Removing $id" az resource delete --ids "$id" --verbose done ``` The reason to have two sets of deletion groups is that, by default, the list contains disks initially, which cannot be removed before the VMs. For this reason, we exclude the disks on the first set, and then we remove the disks. Then clean the local resources: ```bash= multipass delete $MINION_ID1 $MINION_ID2 multipass purge ``` :::warning Remember to remove the `nginx` instance if you decided to use it. :::