Monitor Calix C7 Rings

A ring is a set of Calix C7 Devices that share the same IP Address. For this reason, you cannot apply the usual way to add a node to the OpenNMS inventory.

The idea would be to add an OpenNMS node for each C7 entity with a unique SNMP agent, regardless of its nature: a node, a shelve, a card, a location, etc.

Then, use the metadata feature to populate the specifics of the SNMP agent at the node level. This is what the SNMP Collector would use at run-time to know how to reach the SNMP Agent, as ALL the ring entities would have the same IP Address. In other words, multiple OpenNMS nodes will have the same IP address but a different port to reach their respective statistics.

There are 2 ways to populate the inventory:

1. Use requisitions via Provisiond.
2. Manual provision via ReST API (without using requisitions or Provisiond in general).

With Provisiond, the idea is to allow scan SNMP (using metadata on the detectors) to discover the physical interfaces (or SNMP Interfaces) and persist the required ones based on a given filter via SNMP Policies. No IP interface would be persisted this way.

Unfortunately, as reaching a C7 entity via SNMP has the same priority as business-related requests, it is important to minimize the SNMP requests sent to those devices. To achieve this, set the scan-interval to be one a week or even higher. The reason for this is, once the OpenNMS node is populated, chances are, there is no need to rescan, as we assume the ifIndex is fixed across all physical interfaces on a given SNMP agent. If this is not the case, the best approach would be using requisitions.

But, if that is not possible due to the size of the IP-MIB::ipAddressTable, IF-MIB::ifTable and IF-MIB::ifXTable, we have the second option, which involves using the ReST end-point for nodes, to populate the database (or the inventory) manually.

This document intends to collect SNMP Interfaces statistics only, in other words, statistics from SNMP tables indexed by the ifIndex. Other statistics like private MIBs are not suitable for this solution, as the SNMP Collector works in OpenNMS by retrieving the whole columns of a given table. That goes against the fact that making SNMP queries against a C7 agent is expensive. The only use case to retrieve a given set of rows is for tables indexed by the ifIndex.

Populate the Inventory Manually

The onmsctl tool is the easiest way to manipulate the inventory to avoid dealing with the OpeNNMS ReST API.

Version v1.0.0-beta7 or newer is required.

To populate the inventory for an OpenNMS node we need to know the following:

• Hostname/FQDN (for node-label and optionally for the hostname of the IP interface)
• System Enterprise OID (required by the SNMP Collector)
• IP address of the Ring
• SNMP Interface details for data collection purposes.
  • ifIndex
  • ifName
  • ifDescr
  • ifAlias
  • ifSpeed
• Node level metadata to reach the SNMP Agent
  • SNMP Port
  • SNMP Community
  • SNMP Version
  • SNMP Timeout
  • SNMP Retries

Add a node

onmsctl nodes add \
  --label "test-node-01" \
  --sysOID ".1.3.6.1.4.1.6321"

The output should be something like this:

2020/10/21 10:04:58 Adding node test-node-01
2020/10/21 10:04:58 Node added with ID 2

Take a note about the Node ID as this value will be used on subsequent requests.

Defining a node that way will be interpreted by Provisiond as an auto-discovered node, meaning Provisiond will attempt to scan the node regularly according to the default-foreign-source.xml. It is recommended to have a dedicated OpenNMS instance to monitor the C7 rings and make sure there are no detectors and policies on this file to avoid undesired side effects.

The sysObjectId is a mandatory field that the SNMP Collector will use to know which systemDef inside the chosen SNMP Collection within datacollection-config.xml will be used to determinate the set of metrics to be collected for the device.

Optionally, the following SNMP attributes can also be set, as those would be visible on the WebUI:

• sysName
• sysContact
• sysDescription
• sysLocation

Do not set the location element unless the node is going to be reachable via Minion.

Add an SNMP Interface

onmsctl nodes snmpInterfaces add \
  --ifIndex 100 \
  --ifOper 1 \
  --ifAdmin 1 \
  --ifName eth0 \
  --ifDescr Ethernet0 \
  --ifAlias "Important Interface" \
  --ifSpeed 100000000 \
  --ifType 6 \
  --collect \
  2

The line 11 contains the Node ID as an argument.

The output should be something like this:

2020/10/21 10:07:05 Adding SNMP Interface with index 100

Add an IP interface

onmsctl nodes ipInterfaces add \
 --ipAddr "10.0.0.1" \
 --hostname "test-node-01.local" \
 --isManaged M \
 --snmpPrimary P \
 --ifIndex 100 \
 2

The line 7 contains the Node ID as an argument.

The output should be something like this:

2020/10/21 10:11:05 Associating SNMP interface with ID 7 and ifIndex 100 to 10.0.0.1
2020/10/21 10:11:05 Adding IP Interface 10.0.0.1

To specify an ifIndex to associate that IP interface with an existing SNMP Interface, that physical interface must exist in the chosen node.

The snmpPrimary flag has to be P (i.e., Primary) to let the SNMP Collector know that this IP will be used to retrieve SNMP metrics.

Add a monitored service

onmsctl nodes ipInterfaces services add \
  2 \
  "10.0.0.1" \
  "SNMP-C7"

The line 2 contains the Node ID as an argument.
The line 3 contains the IP Address of the parent interface where the service would be added.
The line 4 contains the name of the service.

The command returns nothing on success.

Add metadata

The metadata can be added at the node level, interface level or service level. Assuming the idea is that each unique SNMP Agent on the C7 ring will have a node in OpenNMS, the node-level metadata would be enough for this solution.

onmsctl nodes metadata set \
  --context "C7" \
  --key "snmpPort" \
  --value 5005 \
  2

The line 5 contains the Node ID as an argument.

The command returns nothing on success.

The context is a way to group a set of metadata, and a metadata itself is a key-value pair.

When using requisitions, the context is always going to be requisitions and cannot be changed. This is why, for custom ones, we should use a unique name.

The idea is to add one metadata entry for each variable that makes the SNMP agent unique.

The context and the key must match the service configuration inside collectd-configuration.xml.

Create Nodes via YAML

Even if the above commands offer a way to build the inventory, there is a better approach, especially when there is a need to add multiple nodes with multiple IP and SNMP interfaces. This method involves creating a YAML file and then use onmsctl to send the request to OpenNMS.

The advantage is that it will take care about all the dependencies like the Node ID automatically for you.

The content of the YAML file should be something like this:

---
nodes:
- label: "node01"
  sysObjectId: "1.3.6.1.4.1.666.1.1"
  sysName: "node01"
  sysLocation: "Earth"
  sysContact: "agalue"
  sysDescription: "Sample Node"
  metadata:
  - context: "C7"
    key: "snmpPort"
    value: "5000"
  ipInterfaces:
  - ipAddress: "192.168.0.10"
    ifIndex: 100 # Must match one of the snmpInterfaces
    services:
    - serviceType:
        name: "SNMP-C7"
  snmpInterfaces:
  - ifIndex: 100
    ifName: "eth0"
    ifDescr: "eth0"
    ifAlias: "admin"
    ifSpeed: 1000000000
    ifType: 6

Assuming the content of the file exist in nodes.yaml, you can send it to OpenNMS like this:

onmsctl nodes apply -f nodes.yaml

The output would be something like this:

2020/10/21 10:25:28 Adding node node01
2020/10/21 10:25:28 Node added with ID 3
2020/10/21 10:25:28 Adding SNMP Interface with index 100
2020/10/21 10:25:28 Associating SNMP interface with ID 11 and ifIndex 100 to 192.168.0.10
2020/10/21 10:25:28 Adding IP Interface 192.168.0.10

You can specify multiple nodes, multiple IP or SNMP interfaces per node, multiple services per IP, and multiple metadata elements per node, IP interface or service.

Useful commands

With a simple command, it is possible to change the metadata, and the collector will use it on the next attempt.

For instance, to change the timeout for node with ID 7, the following command can do it:

onmsctl nodes metadata set -c C7 -k snmpTimeout -v 20000 2

Note that the last argument is the Node ID.

To list the current metadata:

onmsctl nodes metadata list 2

You'll get:

Context	Key		Value
C7	snmpPort	5005
C7	snmpTimeout	20000

To remove an existing metadata:

onmsctl nodes metadata delete -c C7 -k snmpTimeout 2

Note that the last argument is the Node ID.

Collector Configuration

We recommend having a dedicated package for C7, although, as the service name is unique, this is not strictly required.

There, we will use a special service. That is because there is a custom behavior associated with a monitored service when it is called SNMP, and we don't want that for this use case, so we decided to use a custom one called SNMP-C7:

<package name="C7" remote="false">
  <filter>IPADDR != '0.0.0.0'</filter>
  <include-range begin="1.1.1.1" end="254.254.254.254"/>
  <service name="SNMP-C7" interval="300000" user-defined="false" status="on">
    <parameter key="collection" value="C7"/>
    <parameter key="port" value="${C7:snmpPort|162}"/>
    <parameter key="version" value="${C7:snmpVersion|2c}"/>
    <parameter key="read-community" value="${C7:snmpCommunity|public}"/>
    <parameter key="thresholding-enabled" value="true"/>
  </service>
</package>

In case there is going to be a different set of metrics depending on the SNMP Agent, the collection can also be assigned via metadata.

The collection should match a valid snmp-collection inside the datacollection-config.xml. Inside of it, a datacollection-group must be specified, and inside of it, a systemDef referencing the same sysObjectId used when the node was defined.

<datacollection-config xmlns="http://xmlns.opennms.org/xsd/config/datacollection" rrdRepository="/opt/opennms/share/rrd/snmp/">
   <snmp-collection name="C7" snmpStorageFlag="select">
      <rrd step="300">
         <rra>RRA:AVERAGE:0.5:1:2016</rra>
         <rra>RRA:AVERAGE:0.5:12:1488</rra>
         <rra>RRA:AVERAGE:0.5:288:366</rra>
         <rra>RRA:MAX:0.5:288:366</rra>
         <rra>RRA:MIN:0.5:288:366</rra>
      </rrd>
      <include-collection dataCollectionGroup="C7-Metrics"/>
   </snmp-collection>
</datacollection-config>

Note that the name of the snmp-collection matches the collection parameters inside the Collection Package.

The rrd section is mandatory but only used when the RRDtool or JRobin backend is enabled. This is ignored when using Newts/Cassandra.

<datacollection-group xmlns="http://xmlns.opennms.org/xsd/config/datacollection" name="C7-Metrics">
...
   <systemDef name="C7-Devices">
      <sysoid>.1.3.6.1.4.1.6321</sysoid>
      <collect>
         <includeGroup>mib2-X-interfaces</includeGroup>
         ...
      </collect>
   </systemDef>
</datacollection-group>

The above XML is a sample of what the actual configuration should look like. Note that the name of the datacollection-group matches the include-collection entry from datacollectioon-config.xml. This XML is intended to be stored on a file inside the etc/datacollection directory.

It is crucial to set the following property inside /opt/opennms/etc/opennms.properties.d for Collectd, to avoid performing bulk requests against the whole ifTable or ifXTable, as this is forbidden on a C7:

org.opennms.netmgt.collectd.SnmpCollector.limitCollectionToInstances=true

That is another reason for having a dedicated OpenNMS for monitoring C7, as that is a global property that affects all the monitored nodes.