Try   HackMD

Simple OpenNMS Environment using Cassandra

This lab starts an OpenNMS instance, a 3-node Cassandra cluster and optionally an instance for Cassandra Reaper in the cloud, for learning purposes.

To monitor a network, it is advised to enable ActiveMQ or Kafka and use Minions. For simplicity, the embedded ActiveMQ will be enabled.

Follow this guide for general information about how to configure Cassandra for OpenNMS/Newts.

Requirements

The scripts used through this tutorial use envsubst, make sure to have it installed.

Make sure to log into Azure using az login prior creating the VM.

If you have a restricted account in Azure, make sure you have the Network Contributor role and the Virtual Machine Contributor role associated with your Azure AD account for the resource group where you want to create the VM. Of course, either Owner or Contributor at the resource group level are welcome.

There are problems with cqlsh in 3.11.11 (see CASSANDRA-16822 for more details). Use this repository to install an older version, by adjusting the baseurl under yum_repos for the cloud-init template for Cassandra.

Create common Environment Variables























# Main
export RG_NAME="OpenNMS" # Change it to use a shared one
export LOCATION="eastus" # Azure Region
export VNET_CIDR="13.0.0.0/16"
export VNET_SUBNET="13.0.1.0/24"
export VNET_NAME="$USER-cassandra-vnet"
export VNET_SUBNET_NAME="subnet1"
export CASSANDRA_VERSION="40x" # Either 311x or 40x
export CASSANDRA_VM_SIZE="Standard_D2s_v3" # 2 VCPU, 8 GB of RAM
export CASSANDRA_DISK_SIZE="512" # Disk size in GB
export CASSANDRA_HEAP="4g" # Must fit CASSANDRA_VM_SIZE
export CASSANDRA_CLUSTER_NAME="OpenNMS-Cluster"
export CASSANDRA_DATACENTER="DC1"
export CASSANDRA_REPLICATION_FACTOR="2"
export CASSANDRA_SEED="$USER-cassandra1"
export NEWTS_KEYSPACE_NAME="newts"
export ONMS_HEAP="4096" # Expressed in MB and must fit ONMS_VM_SIZE
export ONMS_VM_SIZE="Standard_D2s_v3" # 2 VCPU, 8 GB of RAM
export ONMS_VM_NAME="$USER-cassandra-onms"
export REAPER_VM_NAME="$USER-cassandra-reaper"
# Generated
export SUBNET_BASE=${VNET_SUBNET/\.0\/24/}

Feel free to change the content and keep in mind that $USER is what we will use throughout this tutorial to identify all the resources we will create in Azure uniquely.

Do not confuse the Azure Location or Region with the Minion Location; they are both unrelated things.

We're going to leverage the Azure DNS services to avoid the need to remember and using Public IP addresses.

In Azure, the default public DNS follow the same pattern:

<vm-name>.<location>.cloudapp.azure.com

To make the VMs FQDN unique, we're going to add the username to the VM name. For instance, the OpenNMS FQDN would be:

agalue-onms01.eastus.cloudapp.azure.com

The above is what we can use to access the VM via SSH and to configure Minions.

Create the Azure Resource Group

This is a necessary step, as every resource in Azure must belong to a resource group and a location.

However, you can omit the following command and use an existing one if you prefer. In that case, make sure to adjust the environment variable RG_NAME so the subsequent commands will target the correct group.

az group create -n $RG_NAME -l $LOCATION

Create the Virtual Network








az network vnet create -g $RG_NAME \
  --name $VNET_NAME \
  --address-prefix $VNET_CIDR \
  --subnet-name $VNET_SUBNET_NAME \
  --subnet-prefix $VNET_SUBNET \
  --tags Owner=$USER \
  --output table

The reason for creating the VNET is that we need static IP addresses for each Cassandra instance.

Create VMs for the Cassandra cluster

We will name each VM like follows, based on the chosen username and CIDR for the VNET and the subnet within it:

  • agalue-cassandra1 (13.0.1.11)
  • agalue-cassandra2 (13.0.1.12)
  • agalue-cassandra3 (13.0.1.13)

Note the hostnames include the chosen username to make them unique, which is mandatory for shared resource groups and the default Azure DNS public domain on the chosen region.

Remember that each VM in Azure is reachable within the same VNet from any other VM through its hostname, and the following will take advantage of this fact when configuring seed nodes and Newts access from OpenNMS.

It is assumed that each VM would have at least 8GB of RAM, due to the chosen amount of HEAP that will be reserved (a Standard_D2s_v3 instance should be sufficient).

It is also assumed that agalue-cassandra1 will be the seed node.

Network Topology will be used as that's a typical scenario in production. For this, we're going to use rack-awareness, but a single-dc. The DC name will be DC1, and for the rack, we're going to use the hostname of the Cassandra instance.

Create the cloud-init YAML file as /tmp/cassandra-template.yaml with the following content for Cassandra 3.11.x using OpenJDK 8 or Cassandra 4.x using OpenJDK 11, expecting to be installed in RHEL/CentOS 8, with a dedicated disk for data:














































































































































































































































































#cloud-config
package_upgrade: false

write_files:
  - owner: root:root
    path: /etc/sysctl.d/99-cassandra.conf
    content: |
      net.ipv4.tcp_keepalive_time=60
      net.ipv4.tcp_keepalive_probes=3
      net.ipv4.tcp_keepalive_intvl=10
      net.core.rmem_max=16777216
      net.core.wmem_max=16777216
      net.core.rmem_default=16777216
      net.core.wmem_default=16777216
      net.core.optmem_max=40960
      net.ipv4.tcp_rmem=4096 87380 16777216
      net.ipv4.tcp_wmem=4096 65536 16777216
      net.ipv4.tcp_window_scaling=1
      net.core.netdev_max_backlog=2500
      net.core.somaxconn=65000
      vm.swappiness=1
      vm.zone_reclaim_mode=0
      vm.max_map_count=1048575

  - owner: root:root
    path: /etc/cassandra/fix-schema.cql
    content: |
      ALTER KEYSPACE system_auth WITH REPLICATION = {
        'class' : 'NetworkTopologyStrategy',
        '$CASSANDRA_DATACENTER' : 3
      };
      ALTER KEYSPACE system_distributed WITH REPLICATION = {
        'class' : 'NetworkTopologyStrategy',
        '$CASSANDRA_DATACENTER' : 3
      };
      ALTER KEYSPACE system_traces WITH REPLICATION = {
        'class' : 'NetworkTopologyStrategy',
        '$CASSANDRA_DATACENTER' : 2
      };

  - owner: root:root
    path: /etc/cassandra/reaper.cql
    content: |
      CREATE KEYSPACE IF NOT EXISTS reaper_db WITH replication = {
        'class' : 'NetworkTopologyStrategy',
        '$CASSANDRA_DATACENTER' : $CASSANDRA_REPLICATION_FACTOR
      };

  - owner: root:root
    path: /etc/cassandra/newts.cql
    content: |
      CREATE KEYSPACE IF NOT EXISTS $NEWTS_KEYSPACE_NAME WITH replication = {
        'class' : 'NetworkTopologyStrategy',
        '$CASSANDRA_DATACENTER' : $CASSANDRA_REPLICATION_FACTOR
      };
      CREATE TABLE IF NOT EXISTS $NEWTS_KEYSPACE_NAME.samples (
        context text,
        partition int,
        resource text,
        collected_at timestamp,
        metric_name text,
        value blob,
        attributes map<text, text>,
        PRIMARY KEY((context, partition, resource), collected_at, metric_name)
      ) WITH compaction = {
        'compaction_window_size': '7',
        'compaction_window_unit': 'DAYS',
        'expired_sstable_check_frequency_seconds': '86400',
        'class':  'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy'
      } AND gc_grace_seconds = 604800
        AND read_repair_chance = 0;
      CREATE TABLE IF NOT EXISTS $NEWTS_KEYSPACE_NAME.terms (
        context text,
        field text,
        value text,
        resource text,
        PRIMARY KEY((context, field, value), resource)
      );
      CREATE TABLE IF NOT EXISTS $NEWTS_KEYSPACE_NAME.resource_attributes (
        context text,
        resource text,
        attribute text,
        value text,
        PRIMARY KEY((context, resource), attribute)
      );
      CREATE TABLE IF NOT EXISTS $NEWTS_KEYSPACE_NAME.resource_metrics (
        context text,
        resource text,
        metric_name text,
        PRIMARY KEY((context, resource), metric_name)
      );

  - owner: root:root
    permissions: '0750'
    path: /etc/cassandra/bootstrap.sh
    content: |
      #!/bin/bash
      function wait_for_seed {
        until echo -n > /dev/tcp/$CASSANDRA_SEED/9042; do
          echo "### seed unavailable - sleeping"
          sleep 5
        done
      }
      function start_cassandra {
        echo "Starting Cassandra..."
        systemctl enable cassandra
        systemctl start cassandra
      }
      if [ ! -f "/etc/cassandra/.configured" ]; then
        echo "Cassandra is not configured."
        exit
      fi
      index=$(hostname | awk '{ print substr($0,length,1) }')
      echo "Bootstrapping instance $index..."
      if [[ "$index" == "1" ]]; then
        start_cassandra
        wait_for_seed
        echo "Configuring keyspaces..."
        cqlsh -f /etc/cassandra/fix-schema.cql $(hostname)
        cqlsh -f /etc/cassandra/newts.cql $(hostname)
        cqlsh -f /etc/cassandra/reaper.cql $(hostname)
      else
        wait_for_seed
        start_delay=$((60*($index-1)))
        sleep $start_delay
        start_cassandra
      fi

  - owner: root:root
    permissions: '0750'
    path: /etc/cassandra/configure.sh
    content: |
      #!/bin/bash
      if [ "$(id -u -n)" != "root" ]; then
        echo "Error: you must run this script as root" >&2
        exit 4  # According to LSB: 4 - user had insufficient privileges
      fi
      if [ -f "/etc/cassandra/.configured" ]; then
        echo "Cassandra node already configured."
        exit
      fi
      if [ "$CASSANDRA_VERSION" == "40x" ]; then
        dnf install -y java-11-openjdk
        echo 2 | alternatives --config java
        echo 2 | alternatives --config python
        sed -i -r "s/AND read_repair_chance = 0//" /etc/cassandra/newts.cql
      else
        dnf install -y python2
        echo 3 | alternatives --config python
      fi
      intf="eth0"
      # Basic Configuration
      cfg="/etc/cassandra/conf/cassandra.yaml"
      sed -r -i "/cluster_name:/s/Test Cluster/$CASSANDRA_CLUSTER_NAME/" $cfg
      sed -r -i "/seeds:/s/127.0.0.1/$CASSANDRA_SEED/" $cfg
      sed -r -i "s/^listen_address/#listen_address/" $cfg
      sed -r -i "s/^rpc_address/#rpc_address/" $cfg
      sed -r -i "s/^# listen_interface: .*/listen_interface: $intf/" $cfg
      sed -r -i "s/^# rpc_interface: .*/rpc_interface: $intf/" $cfg
      sed -r -i "/^endpoint_snitch:/s/SimpleSnitch/GossipingPropertyFileSnitch/" $cfg
      sed -r -i "/^endpoint_snitch:/a dynamic_snitch: false" $cfg
      # Performance Tuning
      cores=$(cat /proc/cpuinfo | grep "^processor" | wc -l)
      sed -r -i "/num_tokens:/s/256/16/" $cfg
      sed -r -i "/enable_materialized_views:/s/true/false/" $cfg
      sed -r -i "s/#concurrent_compactors: .*/concurrent_compactors: $cores/" $cfg
      # Network Topology
      cfg="/etc/cassandra/conf/cassandra-rackdc.properties"
      index=$(hostname | awk '{ print substr($0,length,1) }')
      sed -r -i "/^dc/s/=.*/=$CASSANDRA_DATACENTER/" $cfg
      sed -r -i "/^rack/s/=.*/=Rack$index/" $cfg
      rm -f /etc/cassandra/conf/cassandra-topology.properties
      # Enable JMX Access
      cfg="/etc/cassandra/conf/cassandra-env.sh"
      sed -r -i "/rmi.server.hostname/s/.public name./$(hostname)/" $cfg
      sed -r -i "/rmi.server.hostname/s/^#//" $cfg
      sed -r -i "/jmxremote.access/s/#//" $cfg
      sed -r -i "/LOCAL_JMX=/s/yes/no/" $cfg
      # Configure Heap (make sure it is consistent with the available RAM)
      cfg="/etc/cassandra/conf/jvm.options"
      if [ "$CASSANDRA_VERSION" == "40x" ]; then
        cfg="/etc/cassandra/conf/jvm-server.options"
      fi
      sed -r -i "s/#-Xms4G/-Xms$CASSANDRA_HEAP/" $cfg
      sed -r -i "s/#-Xmx4G/-Xmx$CASSANDRA_HEAP/" $cfg
      # Disable CMSGC and enable G1GC
      if [ "$CASSANDRA_VERSION" == "40x" ]; then
        cfg="/etc/cassandra/conf/jvm11-server.options"
      fi
      ToDisable=(UseParNewGC UseConcMarkSweepGC CMSParallelRemarkEnabled SurvivorRatio MaxTenuringThreshold CMSInitiatingOccupancyFraction UseCMSInitiatingOccupancyOnly CMSWaitDuration CMSParallelInitialMarkEnabled CMSEdenChunksRecordAlways CMSClassUnloadingEnabled)
      for entry in "${ToDisable[@]}"; do
        sed -r -i "/$entry/s/-XX/#-XX/" $cfg
      done
      ToEnable=(UseG1GC G1RSetUpdatingPauseTimePercent MaxGCPauseMillis InitiatingHeapOccupancyPercent ParallelGCThreads)
      for entry in "${ToEnable[@]}"; do
        sed -r -i "/$entry/s/#-XX/-XX/" $cfg
      done
      # Configure Data Disk
      disk=/dev/disk/azure/scsi1/lun0
      dev=$disk-part1
      label=LUN0
      echo "Waiting on $disk"
      while [ ! -e $(readlink -f $disk) ]; do
        printf '.'
        sleep 10
      done
      echo ';' | sfdisk $disk
      echo "Waiting on $dev"
      while [ ! -e $(readlink -f $dev) ]; do
        printf '.'
        sleep 1
      done
      mkfs -t xfs -L $label $dev
      mkdir /tmp/__cassandra
      mv /var/lib/cassandra/* /tmp/__cassandra/
      mount -L $label /var/lib/cassandra
      mv /tmp/__cassandra/* /var/lib/cassandra/
      echo "LABEL=$label /var/lib/cassandra auto defaults,noatime 0 0" >> /etc/fstab
      # Fix Permissions
      chown cassandra:cassandra /etc/cassandra/jmxremote.*
      touch /etc/cassandra/.configured

  - owner: root:root
    permissions: '0400'
    path: /etc/cassandra/jmxremote.password
    content: |
      monitorRole QED
      controlRole R&D
      cassandra cassandra

  - owner: root:root
    permissions: '0400'
    path: /etc/cassandra/jmxremote.access
    content: |
      monitorRole readonly
      cassandra   readwrite
      controlRole readwrite \
                  create javax.management.monitor.*,javax.management.timer.* \
                  unregister

  - owner: root:root
    permissions: '0400'
    path: /etc/snmp/snmpd.conf
    content: |
      rocommunity public default
      syslocation Azure
      syscontact IT
      dontLogTCPWrappersConnects yes
      disk /var/lib/cassandra

yum_repos:
  cassandra:
    name: Apache cassandra
    baseurl: https://downloads.apache.org/cassandra/redhat/$CASSANDRA_VERSION/
    gpgkey: https://downloads.apache.org/cassandra/KEYS
    enabled: true
    gpgcheck: true

packages:
  - net-snmp
  - net-snmp-utils
  - cassandra
  - cassandra-tools

runcmd:
  - sysctl --system
  - systemctl enable --now snmpd
  - /etc/cassandra/configure.sh
  - /etc/cassandra/bootstrap.sh

Even if Cassandra 4 supports OpenJDK 11, it is considered an experimental feature as mentioned here. However, some people report great results with OpenJDK 16 and ZGC instead G1GC.

For simplicity, I'm extracting the last digit from the hostname and use it as part of the rack name. You can apply other rules if needed; for instance, if you're deploying multiple instances of Cassandra in the same physical machine, you could use the hostname as rack or whatever makes sense to you. Please keep in mind this affects how data replicates across the cluster.

Create the Cassandra cluster on CentOS 8 VMs:
























envsubst "$(env | cut -d= -f1 | sed -e 's/^/$/')" < /tmp/cassandra-template.yaml > /tmp/cassandra.yaml

for i in {1..3}; do
  VM_NAME="$USER-cassandra$i"
  VM_IP="$SUBNET_BASE.1$i"

  echo "Creating VM $VM_NAME ($VM_IP)..."

  az vm create --resource-group $RG_NAME --name $VM_NAME \
    --size $CASSANDRA_VM_SIZE \
    --image OpenLogic:CentOS:8_4:latest \
    --admin-username $USER \
    --ssh-key-values ~/.ssh/id_rsa.pub \
    --vnet-name $VNET_NAME \
    --subnet $VNET_SUBNET_NAME \
    --private-ip-address $VM_IP \
    --public-ip-address-dns-name $VM_NAME \
    --public-ip-sku Standard \
    --data-disk-sizes-gb $CASSANDRA_DISK_SIZE \
    --custom-data /tmp/cassandra.yaml \
    --tags Owner=$USER \
    --no-wait
done

There is no need to open ports, as any VM can reach any other VM through any port by default, and the cluster won't be exposed to the internet (except for SSH via FQDN).

Also note that static IPs will be used because that's a Cassandra requirement.

All the VMs will be created simultaneously; although, the bootstrap script takes care about having only one joining the cluster at a time.

Create a VM for OpenNMS

Create a cloud-init script to deploy OpenNMS and PostgreSQL in Ubuntu with the following content and save it at /tmp/opennms-template.yaml:

























































































#cloud-config
package_upgrade: true

write_files:
  - owner: root:root
    path: /etc/opennms-overlay/default-foreign-source.xml
    content: |
      <foreign-source xmlns="http://xmlns.opennms.org/xsd/config/foreign-source" name="default" date-stamp="2021-03-31T00:00:00.000Z">
        <scan-interval>1d</scan-interval>
        <detectors>
          <detector name="ICMP" class="org.opennms.netmgt.provision.detector.icmp.IcmpDetector"/>
          <detector name="SNMP" class="org.opennms.netmgt.provision.detector.snmp.SnmpDetector"/>
        </detectors>
        <policies/>
      </foreign-source>

  - owner: root:root
    path: /etc/opennms-overlay/opennms.properties.d/newts.properties
    content: |
      org.opennms.timeseries.strategy=newts
      org.opennms.newts.config.hostname=$CASSANDRA_SEED
      org.opennms.newts.config.keyspace=$NEWTS_KEYSPACE_NAME
      org.opennms.newts.config.port=9042
      org.opennms.newts.config.read_consistency=ONE
      org.opennms.newts.config.write_consistency=ANY
      org.opennms.newts.config.resource_shard=604800
      org.opennms.newts.config.ttl=31540000
      org.opennms.newts.config.cache.priming.enable=true
      org.opennms.newts.config.cache.priming.block_ms=60000
      # The following settings most be tuned in production
      org.opennms.newts.config.writer_threads=2
      org.opennms.newts.config.ring_buffer_size=131072
      org.opennms.newts.config.cache.max_entries=131072

  - owner: root:root
    permissions: '0750'
    path: /etc/opennms-overlay/bootstrap.sh
    content: |
      #!/bin/bash
      systemctl --now enable postgresql
      sudo -u postgres createuser opennms
      sudo -u postgres psql -c "ALTER USER postgres WITH PASSWORD 'postgres';"
      sudo -u postgres psql -c "ALTER USER opennms WITH PASSWORD 'opennms';"
      sed -r -i 's/password=""/password="postgres"/' /etc/opennms/opennms-datasources.xml
      sed -r -i '/0.0.0.0:61616/s/([<][!]--|--[>])//g' /etc/opennms/opennms-activemq.xml
      sed -r -i '/enabled="false"/{$!{N;s/ enabled="false"[>]\n(.*OpenNMS:Name=Syslogd.*)/>\n\1/}}' /etc/opennms/service-configuration.xml
      rsync -avr /etc/opennms-overlay/ /etc/opennms/
      echo 'JAVA_HEAP_SIZE=$ONMS_HEAP' > /etc/opennms/opennms.conf
      sed -r -i "s/cassandra-username/cassandra/" /etc/opennms/poller-configuration.xml
      sed -r -i "s/cassandra-password/cassandra/" /etc/opennms/poller-configuration.xml
      sed -r -i "s/cassandra-username/cassandra/" /etc/opennms/collectd-configuration.xml
      sed -r -i "s/cassandra-password/cassandra/" /etc/opennms/collectd-configuration.xml
      /usr/share/opennms/bin/runjava -s
      /usr/share/opennms/bin/fix-permissions
      /usr/share/opennms/bin/install -dis
      until echo -n > /dev/tcp/$CASSANDRA_SEED/9042; do
        echo "### Cassandra seed unavailable - sleeping"
        sleep 5
      done
      systemctl --now enable opennms

  - owner: root:root
    permissions: '0400'
    path: /etc/snmp/snmpd.conf
    content: |
      rocommunity public default
      syslocation Azure
      syscontact IT
      dontLogTCPWrappersConnects yes
      disk /

apt:
  preserve_sources_list: true
  sources:
    opennms:
      source: deb https://debian.opennms.org stable main

packages:
  - snmp
  - snmpd
  - opennms
  - opennms-webapp-hawtio

bootcmd:
  - curl -s https://debian.opennms.org/OPENNMS-GPG-KEY | apt-key add -

runcmd:
  - /etc/opennms-overlay/bootstrap.sh

The above installs the latest OpenJDK 11, the latest PostgreSQL, and the latest OpenNMS Horizon. I added the most basic configuration for PostgreSQL to work with authentication. The embedded ActiveMQ is enabled, as well as Syslogd.

Create an Ubuntu VM for OpenNMS:





















envsubst < /tmp/opennms-template.yaml > opennms.yaml

az vm create --resource-group $RG_NAME --name $ONMS_VM_NAME \
  --size $ONMS_VM_SIZE \
  --image UbuntuLTS \
  --admin-username $USER \
  --ssh-key-values ~/.ssh/id_rsa.pub \
  --vnet-name $VNET_NAME \
  --subnet $VNET_SUBNET_NAME \
  --private-ip-address "$SUBNET_BASE.100" \
  --public-ip-address-dns-name $ONMS_VM_NAME \
  --public-ip-sku Standard \
  --custom-data /tmp/opennms.yaml \
  --tags Owner=$USER \
  --output table

az vm open-port -g $RG_NAME -n $ONMS_VM_NAME \
  --port 61616 --priority 100 -o table
az vm open-port -g $RG_NAME -n $ONMS_VM_NAME \
  --port 8980 --priority 200 -o table

Keep in mind that the cloud-init process starts once the VM is running, meaning you should wait about 5 minutes after the az vm create is finished to see OpenNMS up and running.

In case there is a problem, SSH into the VM using the public IP and the provided credentials and check /var/log/cloud-init-output.log to verify the progress and the status of the cloud-init execution.

You can SSH the Cassandra VMs from the OpenNMS VM, as those don't have public IP addresses.

Create a VM for Cassandra Reaper

This is an optional step, not required to use Cassandra.

Create a cloud-init script to deploy Cassandra Reaper in Ubuntu with the following content and save it at /tmp/reaper-template.yaml:








































































































#cloud-config
package_upgrade: true

write_files:
  - owner: root:root
    permissions: '0400'
    path: /etc/snmp/snmpd.conf
    content: |
      rocommunity public default
      syslocation Azure
      syscontact IT
      dontLogTCPWrappersConnects yes
      disk /

  - owner: root:root
    permissions: '0644'
    path: /etc/cassandra-reaper/cassandra-reaper.yaml
    content: |
      segmentCountPerNode: 64
      repairParallelism: SEQUENTIAL
      repairIntensity: 0.9
      scheduleDaysBetween: 7
      repairRunThreadCount: 15
      hangingRepairTimeoutMins: 30
      storageType: cassandra
      enableCrossOrigin: true
      incrementalRepair: false
      blacklistTwcsTables: true
      enableDynamicSeedList: true
      repairManagerSchedulingIntervalSeconds: 10
      jmxConnectionTimeoutInSeconds: 5
      useAddressTranslator: false
      maxParallelRepairs: 10
      purgeRecordsAfterInDays: 30
      numberOfRunsToKeepPerUnit: 10
      datacenterAvailability: ALL
      jmxAuth:
        username: cassandra
        password: cassandra
      logging:
        level: INFO
        loggers:
          io.dropwizard: WARN
          org.eclipse.jetty: WARN
        appenders:
        - type: console
          logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"
          threshold: WARN
        - type: file
          logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"
          currentLogFilename: /var/log/cassandra-reaper/reaper.log
          archivedLogFilenamePattern: /var/log/cassandra-reaper/reaper-%d.log.gz
          archivedFileCount: 20
      server:
        type: default
        applicationConnectors:
        - type: http
          port: 8080
          bindHost: 0.0.0.0
        adminConnectors:
        - type: http
          port: 8081
          bindHost: 0.0.0.0
        requestLog:
          appenders: []
      cassandra:
        clusterName: "$CASSANDRA_CLUSTER_NAME"
        contactPoints: ["$CASSANDRA_SEED"]
        keyspace: reaper_db
      autoScheduling:
        enabled: false
        initialDelayPeriod: PT15S
        periodBetweenPolls: PT10M
        timeBeforeFirstSchedule: PT5M
        scheduleSpreadPeriod: PT6H
      metrics:
        frequency: 1 minute
        reporters:
          - type: log
            logger: metrics
      accessControl:
        sessionTimeout: PT10M
        shiro:
          iniConfigs: ["classpath:shiro.ini"]

apt:
  preserve_sources_list: true
  sources:
    cassandra-reaper:
      source: deb https://dl.bintray.com/thelastpickle/reaper-deb wheezy main

packages:
  - snmp
  - snmpd
  - openjdk-8-jdk
  - reaper

bootcmd:
  - apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 2895100917357435

runcmd:
  - systemctl enable cassandra-reaper
  - systemctl start cassandra-reaper

Then,


















envsubst < /tmp/reaper-template.yaml > /tmp/reaper.yaml

az vm create --resource-group $RG_NAME --name $REAPER_VM_NAME \
  --size Standard_D2s_v3 \
  --image UbuntuLTS \
  --admin-username $USER \
  --ssh-key-values ~/.ssh/id_rsa.pub \
  --vnet-name $VNET_NAME \
  --subnet $VNET_SUBNET_NAME \
  --private-ip-address "$SUBNET_BASE.101" \
  --public-ip-address-dns-name $REAPER_VM_NAME \
  --public-ip-sku Standard \
  --custom-data /tmp/reaper.yaml \
  --tags Owner=$USER \
  --output table

az vm open-port -g $RG_NAME -n $REAPER_VM_NAME --port 8080 --priority 200 -o table

Monitor the infrastructure

Wait until OpenNMS is up and running and then execute the following, to start monitoring all the Cassandra servers, the OpenNMS server and the Reaper server via SNMP and JMX.








































cat <<EOF >/tmp/OpenNMS.xml
<?xml version="1.0"?>
<model-import date-stamp="$(date +"%Y-%m-%dT%T.000Z")" foreign-source="OpenNMS">
  <node foreign-id="onms-server" node-label="onms-server">
    <interface ip-addr="$SUBNET_BASE.100" status="1" snmp-primary="P"/>
    <interface ip-addr="127.0.0.1" status="1" snmp-primary="N">
      <monitored-service service-name="OpenNMS-JVM"/>
      <monitored-service service-name="Postgres"/>
    </interface>
  </node>
  <node foreign-id="reaper" node-label="reaper">
    <interface ip-addr="$SUBNET_BASE.101" status="1" snmp-primary="P">
      <monitored-service service-name="HTTP-8080"/>
    </interface>
  </node>
EOF

for i in {1..3}; do
  cat <<EOF >>/tmp/OpenNMS.xml
  <node foreign-id="cassandra$i" node-label="cassandra$i">
    <interface ip-addr="$SUBNET_BASE.1$i" status="1" snmp-primary="P">
      <monitored-service service-name="JMX-Cassandra"/>
      <monitored-service service-name="JMX-Cassandra-Newts"/>
    </interface>
  </node>
EOF
done

cat <<EOF >>/tmp/OpenNMS.xml
</model-import>
EOF

ONMS_FQDN="$ONMS_VM_NAME.$LOCATION.cloudapp.azure.com"

curl -v -u admin:admin \
  -H 'Content-Type: application/xml' -d @/tmp/OpenNMS.xml \
  http://$ONMS_FQDN:8980/opennms/rest/requisitions
curl -v -u admin:admin -X PUT \
  http://$ONMS_FQDN:8980/opennms/rest/requisitions/OpenNMS/import

Create a Minion VM on your network

Create the following cloud-init template to create a Minion (assuming the embedded ActiveMQ within OpenNMS is in place) and save it as /tmp/minion-template.yaml:
































#cloud-config
package_upgrade: true

write_files:
  - owner: root:root
    path: /tmp/org.opennms.minion.controller.cfg
    content: |
      location=$MINION_LOCATION
      id=$MINION_ID
      http-url=http://$ONMS_FQDN:8980/opennms
      broker-url=failover:tcp://$ONMS_FQDN:61616

apt:
  preserve_sources_list: true
  sources:
    opennms:
      source: deb https://debian.opennms.org stable main

packages:
  - opennms-minion

bootcmd:
  - curl -s https://debian.opennms.org/OPENNMS-GPG-KEY | apt-key add -

runcmd:
  - mv -f /tmp/org.opennms.minion.controller.cfg /etc/minion/
  - sed -i -r 's/# export JAVA_MIN_MEM=.*/export JAVA_MIN_MEM="$MINION_HEAP_SIZE"/' /etc/default/minion
  - sed -i -r 's/# export JAVA_MAX_MEM=.*/export JAVA_MAX_MEM="$MINION_HEAP_SIZE"/' /etc/default/minion
  - /usr/share/minion/bin/scvcli set opennms.http admin admin
  - /usr/share/minion/bin/scvcli set opennms.broker admin admin
  - systemctl --now enable minion

Note the usage of environment variables within the YAML template. We will substitute them before creating the VM.

Then, create the runtime template:







export MINION_ID="minion01"
export MINION_LOCATION="Durham"
export MINION_HEAP_SIZE="1g"
export ONMS_FQDN="$ONMS_VM_NAME.$LOCATION.cloudapp.azure.com"

envsubst < /tmp/minion-template.yaml > /tmp/$MINION_ID.yaml

Then, start the new Minion via multipass with one core and 2GB of RAM:

multipass launch -c 1 -m 2G -n $MINION_ID --cloud-init /tmp/$MINION_ID.yaml

Clean Up

When you're done, make sure to delete the cloud resources.

If you created the resource group for this exercise, you could remove all the resources with the following command:

az group delete -g $RG_NAME

If you're using an existing resource group that you cannot remove, make sure only to remove all the resources created in this tutorial. All of them should be easily identified as they will contain the username and the VM name as part of the resource name. The easiest way is to use the Azure Portal for this operation. Alternatively,




















IDS=($(az resource list \
  --resource-group $RG_NAME \
  --query "[?contains(name,'$USER-') && type!='Microsoft.Compute/disks']".id \
  --output tsv | tr '\n' ' '))

for id in "${IDS[@]}"; do
  echo "Removing $id"
  az resource delete --ids "$id" --verbose
done

DISKS=($(az resource list \
  --resource-group $RG_NAME \
  --query "[?contains(name,'$USER-') && type=='Microsoft.Compute/disks']".id \
  --output tsv | tr '\n' ' '))

for id in "${DISKS[@]}"; do
  echo "Removing $id"
  az resource delete --ids "$id" --verbose
done

The reason to have two sets of deletion groups is that, by default, the list contains disks initially, which cannot be removed before the VMs. For this reason, we exclude the disks on the first set, and then we remove the disks.

Then clean the local resources:



multipass delete $MINION_ID
multipass purge
Last changed by 
 
Alejandro Galue
0
506

Read more

Sizing Cassandra

The use case for having Cassandra (or ScyllaDB) as the backend for the performance metrics in OpenNMS is for storing a huge amount of non-aggregated data, which is not possible with RRDtool. RRDtool is very good for those installations with a finite and predictable amount of metrics where the size and the I/O requirements are feasible for modern SSD disks. This is important as RRDtool only scales vertically, meaning that when the current disks&apos; limits are reached (mostly due to speed, not space), a faster disk is required. This is when Cassandra or ScyllaDB can help. Although there is a high learning curve, using any of these applications requires a commitment to have qualified personnel to manage this database. ScyllaDB is binary compatible with Cassandra (even at the SSTable level), but they are implemented very differently. Cassandra is implemented in Java, meaning that all the JVM tuning is required and other internal Cassandra tunings. On the other hand, ScyllaDB is implemented in modern C++, taking advantage of the CPU where it is running. That means ScyllaDB can manage huge machines as nodes, whereas Cassandra would require multiple instances on huge machines. It is feasible to have faster results with ScyllaDB compared with its Java sibling in terms of performance. Configuring and managing both applications is different, even if they will provide the same operational result with OpenNMS, so the decision should be carefully analyzed, especially by the team that will support this database.

May 4, 2022
Simple OpenNMS Environment using ScyllaDB

This lab starts an OpenNMS instance, a 3-node ScyllaDB cluster, and optionally an instance for the Scylla Monitoring Stack in the cloud, for learning purposes. To monitor a network, it is advised to enable ActiveMQ or Kafka and use Minions. For simplicity, the embedded ActiveMQ will be enabled. :::success Follow this guide for general information about how to configure Scylla/Cassandra for OpenNMS/Newts. Most of that knowledge is transferable to ScyllaDB. ::: Requirements

Nov 19, 2021
Simple OpenNMS/Minion Environment using Kafka in Azure

This lab starts an OpenNMS instance and a 3 node ZK/Kafka cluster in the cloud and two Minions on your machine, using Kafka for communication through Multipass and Azure, for learning purposes. :::warning The lab doesn&apos;t cover security by default (user authentication and encryption), which is crucial if we ever want to expose the Kafka cluster to the Internet. A separate section covers the required changes for this. ::: :::success Keep in mind that nothing prevents us from skipping using the cloud provider and do everything with Multipass (or VirtualBox, or Hyper-V, or VMWare). The reason for using a cloud provider is to prove that OpenNMS can monitor unreachable devices via Minion. Similarly, we could use any other cloud provider instead of Azure. However I won&apos;t explain how to port the solution here. :::

Nov 19, 2021
Simple OpenNMS/Minion Environment using the embedded ActiveMQ in Azure

This lab starts an OpenNMS instance in the cloud and two Minions on your machine, using ActiveMQ for communication through Multipass and Azure, for learning purposes. :::danger The lab doesn&apos;t cover security (in terms of encryption), which is crucial if you ever want to expose AMQ to the Internet. Follow this guide to learn how to do it with LetsEncrypt. ::: :::success Keep in mind that nothing prevents you from skipping using the cloud provider and do everything with Multipass (or VirtualBox, or Hyper-V, or VMWare). The reason for using a cloud provider is to prove that OpenNMS can monitor unreachable devices via Minion. Similarly, you could use any other cloud provider instead of Azure. However I won&apos;t explain how to port the solution here. :::

Nov 19, 2021
Read more from Alejandro Galue
Published on HackMD