# Monitor Calix C7 Rings

A ring is a set of [Calix C7 Devices](https://www.calix.com/platforms/non-axos-access-systems/c-series.html) that share the same IP Address. For this reason, you cannot apply the usual way to add a node to the OpenNMS inventory.

The idea would be to add an OpenNMS node for each C7 entity with a unique SNMP agent, regardless of its nature: a node, a shelve, a card, a location, etc.

Then, use the metadata feature to populate the specifics of the SNMP agent at the node level. This is what the SNMP Collector would use at run-time to know how to reach the SNMP Agent, as *ALL* the ring entities would have the same IP Address. In other words, multiple OpenNMS nodes will have the same IP address but a different port to reach their respective statistics.

There are 2 ways to populate the inventory:

1) Use requisitions via Provisiond.
2) Manual provision via ReST API (without using requisitions or Provisiond in general).

With Provisiond, the idea is to allow scan SNMP (using metadata on the detectors) to discover the physical interfaces (or SNMP Interfaces) and persist the required ones based on a given filter via SNMP Policies. No IP interface would be persisted this way.

Unfortunately, as reaching a C7 entity via SNMP has the same priority as business-related requests, it is important to minimize the SNMP requests sent to those devices. To achieve this, set the `scan-interval` to be one a week or even higher. The reason for this is, once the OpenNMS node is populated, chances are, there is no need to rescan, as we assume the `ifIndex` is fixed across all physical interfaces on a given SNMP agent. If this is not the case, the best approach would be using requisitions.

But, if that is not possible due to the size of the `IP-MIB::ipAddressTable`, `IF-MIB::ifTable` and `IF-MIB::ifXTable`, we have the second option, which involves using the ReST end-point for nodes, to populate the database (or the inventory) manually.

:::danger
This document intends to collect SNMP Interfaces statistics only, in other words, statistics from SNMP tables indexed by the `ifIndex`. Other statistics like private MIBs are not suitable for this solution, as the SNMP Collector works in OpenNMS by retrieving the whole columns of a given table. That goes against the fact that making SNMP queries against a C7 agent is expensive. The only use case to retrieve a given set of rows is for tables indexed by the `ifIndex`.
:::

## Populate the Inventory Manually

The [onmsctl](https://github.com/OpenNMS/onmsctl) tool is the easiest way to manipulate the inventory to avoid dealing with the OpeNNMS ReST API.

:::danger
Version `v1.0.0-beta7` or newer is required.
:::

To populate the inventory for an OpenNMS node we need to know the following:

* Hostname/FQDN (for node-label and optionally for the hostname of the IP interface)
* System Enterprise OID (required by the SNMP Collector)
* IP address of the Ring
* SNMP Interface details for data collection purposes.
* ifIndex
* ifName
* ifDescr
* ifAlias
* ifSpeed
* Node level metadata to reach the SNMP Agent
* SNMP Port
* SNMP Community
* SNMP Version
* SNMP Timeout
* SNMP Retries

### Add a node

onmsctl nodes add \
  --label "test-node-01" \
  --sysOID ".1.3.6.1.4.1.6321"

The output should be something like this:

2020/10/21 10:04:58 Adding node test-node-01
2020/10/21 10:04:58 Node added with ID 2

Take a note about the Node ID as this value will be used on subsequent requests.

:::warning
Defining a node that way will be interpreted by `Provisiond` as an auto-discovered node, meaning `Provisiond` will attempt to scan the node regularly according to the `default-foreign-source.xml`. It is recommended to have a dedicated OpenNMS instance to monitor the C7 rings and make sure there are no detectors and policies on this file to avoid undesired side effects.
:::

:::danger
The `sysObjectId` is a mandatory field that the SNMP Collector will use to know which `systemDef` inside the chosen SNMP Collection within `datacollection-config.xml` will be used to determinate the set of metrics to be collected for the device.
:::

Optionally, the following SNMP attributes can also be set, as those would be visible on the WebUI:

* sysName
* sysContact
* sysDescription
* sysLocation

:::danger
Do not set the `location` element unless the node is going to be reachable via `Minion`.
:::

### Add an SNMP Interface

onmsctl nodes snmpInterfaces add \
  --ifIndex 100 \
  --ifOper 1 \
  --ifAdmin 1 \
  --ifName eth0 \
  --ifDescr Ethernet0 \
  --ifAlias "Important Interface" \
  --ifSpeed 100000000 \
  --ifType 6 \
  --collect \
  2

:::success
The line 11 contains the Node ID as an argument.
:::

The output should be something like this:

2020/10/21 10:07:05 Adding SNMP Interface with index 100

### Add an IP interface

onmsctl nodes ipInterfaces add \
 --ipAddr "10.0.0.1" \
 --hostname "test-node-01.local" \
 --isManaged M \
 --snmpPrimary P \
 --ifIndex 100 \
 2

:::success
The line 7 contains the Node ID as an argument.
:::

The output should be something like this:

2020/10/21 10:11:05 Associating SNMP interface with ID 7 and ifIndex 100 to 10.0.0.1
2020/10/21 10:11:05 Adding IP Interface 10.0.0.1

:::info
To specify an `ifIndex` to associate that IP interface with an existing SNMP Interface, that physical interface *must* exist in the chosen node.
:::

:::warning
The `snmpPrimary` flag has to be `P` (i.e., `Primary`) to let the SNMP Collector know that this IP will be used to retrieve SNMP metrics.
:::

### Add a monitored service

onmsctl nodes ipInterfaces services add \
  2 \
  "10.0.0.1" \
  "SNMP-C7"

:::success
The line 2 contains the Node ID as an argument.
The line 3 contains the IP Address of the parent interface where the service would be added.
The line 4 contains the name of the service.
:::

The command returns nothing on success.

### Add metadata

The `metadata` can be added at the node level, interface level or service level. Assuming the idea is that each unique SNMP Agent on the C7 ring will have a node in OpenNMS, the node-level metadata would be enough for this solution.

onmsctl nodes metadata set \
  --context "C7" \
  --key "snmpPort" \
  --value 5005 \
  2

:::success
The line 5 contains the Node ID as an argument.
:::

The command returns nothing on success.

The `context` is a way to *group* a set of metadata, and a metadata itself is a key-value pair.

:::warning
When using requisitions, the context is *always* going to be `requisitions` and cannot be changed. This is why, for custom ones, we should use a unique name.
:::

The idea is to add one metadata entry for each variable that makes the SNMP agent unique.

:::danger
The `context` and the `key` must match the service configuration inside `collectd-configuration.xml`.
:::

## Create Nodes via YAML

Even if the above commands offer a way to build the inventory, there is a better approach, especially when there is a need to add multiple nodes with multiple IP and SNMP interfaces. This method involves creating a YAML file and then use `onmsctl` to send the request to OpenNMS.

The advantage is that it will take care about all the dependencies like the Node ID automatically for you.

The content of the YAML file should be something like this:

---
nodes:
- label: "node01"
  sysObjectId: "1.3.6.1.4.1.666.1.1"
  sysName: "node01"
  sysLocation: "Earth"
  sysContact: "agalue"
  sysDescription: "Sample Node"
  metadata:
  - context: "C7"
    key: "snmpPort"
    value: "5000"
  ipInterfaces:
  - ipAddress: "192.168.0.10"
    ifIndex: 100 # Must match one of the snmpInterfaces
    services:
    - serviceType:
        name: "SNMP-C7"
  snmpInterfaces:
  - ifIndex: 100
    ifName: "eth0"
    ifDescr: "eth0"
    ifAlias: "admin"
    ifSpeed: 1000000000
    ifType: 6

Assuming the content of the file exist in `nodes.yaml`, you can send it to OpenNMS like this:

onmsctl nodes apply -f nodes.yaml

The output would be something like this:

2020/10/21 10:25:28 Adding node node01
2020/10/21 10:25:28 Node added with ID 3
2020/10/21 10:25:28 Adding SNMP Interface with index 100
2020/10/21 10:25:28 Associating SNMP interface with ID 11 and ifIndex 100 to 192.168.0.10
2020/10/21 10:25:28 Adding IP Interface 192.168.0.10

You can specify multiple nodes, multiple IP or SNMP interfaces per node, multiple services per IP, and multiple metadata elements per node, IP interface or service.

## Useful commands

With a simple command, it is possible to change the metadata, and the collector will use it on the next attempt.

For instance, to change the timeout for node with ID 7, the following command can do it:

onmsctl nodes metadata set -c C7 -k snmpTimeout -v 20000 2

Note that the last argument is the Node ID.

To list the current metadata:

onmsctl nodes metadata list 2

You'll get:

Context	Key		Value
C7	snmpPort	5005
C7	snmpTimeout	20000

To remove an existing metadata:

onmsctl nodes metadata delete -c C7 -k snmpTimeout 2

Note that the last argument is the Node ID.

## Collector Configuration

We recommend having a dedicated package for C7, although, as the service name is unique, this is not strictly required.

There, we will use a special service. That is because there is a custom behavior associated with a monitored service when it is called `SNMP`, and we don't want that for this use case, so we decided to use a custom one called `SNMP-C7`:

<package name="C7" remote="false">
  <filter>IPADDR != '0.0.0.0'</filter>
  <include-range begin="1.1.1.1" end="254.254.254.254"/>
  <service name="SNMP-C7" interval="300000" user-defined="false" status="on">
    <parameter key="collection" value="C7"/>
    <parameter key="port" value="${C7:snmpPort|162}"/>
    <parameter key="version" value="${C7:snmpVersion|2c}"/>
    <parameter key="read-community" value="${C7:snmpCommunity|public}"/>
    <parameter key="thresholding-enabled" value="true"/>
  </service>
</package>

In case there is going to be a different set of metrics depending on the SNMP Agent, the `collection` can also be assigned via metadata.

:::warning
The `collection` should match a valid `snmp-collection` inside the `datacollection-config.xml`. Inside of it, a `datacollection-group` must be specified, and inside of it, a `systemDef` referencing the same `sysObjectId` used when the node was defined.
:::

<datacollection-config xmlns="http://xmlns.opennms.org/xsd/config/datacollection" rrdRepository="/opt/opennms/share/rrd/snmp/">
   <snmp-collection name="C7" snmpStorageFlag="select">
      <rrd step="300">
         <rra>RRA:AVERAGE:0.5:1:2016</rra>
         <rra>RRA:AVERAGE:0.5:12:1488</rra>
         <rra>RRA:AVERAGE:0.5:288:366</rra>
         <rra>RRA:MAX:0.5:288:366</rra>
         <rra>RRA:MIN:0.5:288:366</rra>
      </rrd>
      <include-collection dataCollectionGroup="C7-Metrics"/>
   </snmp-collection>
</datacollection-config>

:::info
Note that the name of the `snmp-collection` matches the `collection` parameters inside the Collection Package.
:::

The `rrd` section is mandatory but only used when the RRDtool or JRobin backend is enabled. This is ignored when using Newts/Cassandra.

<datacollection-group xmlns="http://xmlns.opennms.org/xsd/config/datacollection" name="C7-Metrics">
...
   <systemDef name="C7-Devices">
      <sysoid>.1.3.6.1.4.1.6321</sysoid>
      <collect>
         <includeGroup>mib2-X-interfaces</includeGroup>
         ...
      </collect>
   </systemDef>
</datacollection-group>

The above XML is a sample of what the actual configuration should look like. Note that the name of the `datacollection-group` matches the `include-collection` entry from `datacollectioon-config.xml`. This XML is intended to be stored on a file inside the `etc/datacollection` directory.

It is **crucial** to set the following property inside `/opt/opennms/etc/opennms.properties.d` for `Collectd`, to avoid performing bulk requests against the whole `ifTable` or `ifXTable`, as this is forbidden on a C7:

org.opennms.netmgt.collectd.SnmpCollector.limitCollectionToInstances=true

That is another reason for having a dedicated OpenNMS for monitoring C7, as that is a global property that affects all the monitored nodes.