Google Drive
Gist
Clipboard
or
or
By clicking below, you agree to our terms of service.
Sign in with Wallet
New to HackMD? Sign up
| Syntax | Example | Reference | |
|---|---|---|---|
| # Header | Header | 基本排版 | |
| - Unordered List |
|
||
| 1. Ordered List |
|
||
| - [ ] Todo List |
|
||
| > Blockquote | Blockquote |
||
| **Bold font** | Bold font | ||
| *Italics font* | Italics font | ||
| ~~Strikethrough~~ | |||
| 19^th^ | 19th | ||
| H~2~O | H2O | ||
| ++Inserted text++ | Inserted text | ||
| ==Marked text== | Marked text | ||
| [link text](https:// "title") | Link | ||
|  | Image | ||
| `Code` | Code |
在筆記中貼入程式碼 | |
| ```javascript var i = 0; ``` |
|
||
| :smile: |  |
Emoji list | |
| {%youtube youtube_id %} | Externals | ||
| $L^aT_eX$ | LaTeX | ||
| :::info This is a alert area. ::: |
This is a alert area. |
On a scale of 0-10, how likely is it that you would recommend HackMD to your friends, family or business associates?
Please give us some advice and help us improve HackMD.
Do you want to remove this version name and description?
Syncing
xxxxxxxxxx-
Any changes
Be notified of any changes
-
Mention me
Be notified of mention me
-
Unsubscribe
SubscribeDeploying Multiple OpenNMS Backends sharing the same Data Clusters
The objective here is to deploy multiple OpenNMS backends, where each backend will have a set of Minions/Locations. The idea is to share the same Kafka cluster across all the backend/minions set. For this reason, the topics in Kafka should be private to each backend/minions set. That way, the messages will be sent by the appropriate minions and will be processed by the appropriate backend server.
When the data collection feature is required, the inventory should be unique per the backend server. The Cassandra/ScyllaDB will be shared among all the backends. In this case, each backend will have its own keyspace, so the data from one backend won't interfere with the others.
In terms of the inventory, as each backend is basically completely independent and unaware of the other ones, each OpenNMS backend will have its own PostgreSQL server to simplify maintenance and avoid potential bottlenecks that might affect other backends.
The consequence of having independent backends is that to have a unified view for all the monitored devices, the usage of Grafana/Helm with custom dashboards will be mandatory, as the WebUI of each backend will show information only for the devices it is monitoring.
Deployment
Requirements
org.opennms.instance.idattribute is unique per OpenNMS backend server and its Minions.Constraints
The first requirement is to facilitate the management of the unified Grafana/Helm dashboards, in case combining metrics from multiple backends is desired. Also, it would be useful to be able to identify the source of a given resource/node just by looking at its foreignSource/foreignID.
As Kafka will be in the middle of the communication, one OpenNMS must only get the data from the Minions that it is managing. In theory, we're talking about multiple independent OpenNMS/Minion deployments that don't share inventory. The main purpose of this is to avoid side effects, avoid introducing lag on a topic that belongs to another OpenNMS/Minion combination when one combination is busy, and facilitate troubleshooting.
The location name is part of the RPC topics, but it is not part of the Sink topics. That means the Sink topics will be shared containing information that matters only to certain OpenNMS servers. This is the main motivation for this change.
Solution
On each backend/minion set:
/opt/opennms/etc/opennms.properties.d/directory (for exampleminion.properties) and add an entry fororg.opennms.instance.idinside of it; for instance:org.opennms.instance.idproperty and its value at the endfile called/opt/minion/etc/custom.system.properties; for instance:The suggestion for the ID format would be:
OpenNMS.XXXOr
CompanyName.XXXWhere
XXXis the custom identifier per OpenNMS backend. It is suggested to use a combination like DC+Region. Please make sure it is a short string without special characters, as this will be part of each topic name. There is no need to include that information on the location's name to include other information for the location.The actual value of the Instance ID (
XXXXin the above example) must match between OpenNMS and its Minions.Background
Let's analyze the existing topics from a test environment:
The above is the result when having
single-topic=false(which is the default for some versions of OpenNMS).If you have
single-topic=true, that reduces the number of RPC topics; for instance:On this particular installation, there is one OpenNMS instance with 2 Minions on different locations (Apex and Durham) handling telemetry, flows, syslog traps messages.
Let's analyze the topics that exist on Kafka:
This is an optional feature to forward the inventory, events, alarms and metrics to Kafka topics. The topic names are:
This can be fully customized per OpenNMS server. In fact, these can be shared across all the OpenNMS instances. The only consumers for these topics on this example deployment is the correlation engine (using the first 2 topics).
This is an optional feature for advanced alarm correlation. The topic names are:
These topics are managed by the ALEC (Architecture for Learning Enabled Correlation), a new feature currently under development.
This is part of the core communication between OpenNMS and Minions, and cover all asynchronous messages sent by Minions to OpenNMS. The topic names are:
These are related with async messages like SNMP Traps, Syslog Messages, Flow data and streaming telemetry data sent by the monitored devices to Minions.
These topics are the ones that should not be shared across multiple OpenNMS instances, and required a custom "instanceId" to make sure each OpenNMS backend and its Minions will produce/consume only on their topics.
This is part of the core communication between OpenNMS and Minions, and cover all the messages associated with synchronous monitoring (like polling and data collection).
When not using
single-topic, the topic names are:When using
single-topic, the topic names are:These topics are divided per Location. As you can see the location names are
DurhamandApex.These set works in conjunction with the
rpc-responseset, so even if these topics are constrained by location, the relation with the other topic set force us to use theinstanceIdfeature per OpenNMS backend.These topics are also associated with RPC communication, but don't contain the location name (hence the need to set the
instanceId):When not using
single-topic, the topic names are:When using
single-topic, the topic names are:Using RPC with Kafka is not available on OpenNMS Meridian, this feature requires OpenNMS Horizon 23 or newer. For Meridian, ActiveMQ will manage RPC communication, but the response Queues are handled differently compared with the solution for Kafka.
These are shared topics that handle RPC responses from all the Minions from all the locations.
For similar reasons like the Sink topics, these should be constrained per OpenNMS backend and its Minions, and the way to do it is by setting an
instanceIdper OpenNMS backend.InstanceID Example
All the topics affected by the Instance ID are the Sink and RPC topics. The default value of the
instanceIdisOpenNMS(for OpenNMS Horizon), orMeridian(for OpenNMS Meridian 2018 or newer).When this ID is set with let's say
CompanyName-DC1-G1(for data center 1, group/sub-group 1), the topics will look like the following, assuming a location name ofLoc1:Obviously, if a second OpenNMS backend has an
instanceIdofCompanyName-DC1-G2orCompanyName-DC2-G3, the topics won't interfere between each other, and they can safely share the same Kafka Cluster.About ActiveMQ
The above solution was designed to be implemented using Kafka. Unfortunately, there are situations on which it is required to use ActiveMQ, for example, when using Meridian 2018, or a version of Horizon older than 23.
ActiveMQ requires additional changes on
$OPENNMS_HOME/etc/opennms-activemq.xmlwhen the instance ID is changed. To be more precise, under a section calledauthorizationEntries.The default configuration contains the following:
The prefix
OpenNMSshould be replaced with the chosen Instance ID. In case of Meridian 2018, the default prefix is different but the logic still applies.