Try   HackMD

Install, configure, manage and run OpenNMS as non-root

This guide was designed for Meridian 2018 but should work on Horizon 22 or newer versions of OpenNMS. I'm assuming the operating system will be CentOS/RHEL 7, but all the commands should work in 8.

Any yum command will internally use dnf if you're using CentOS/RHEL 8.

Installation

The installation process requires privileges. For this reason, the administrator must execute all the following steps as root or through sudo.

User for OpenNMS

Create the opennms user before installing the OpenNMS RPMs (ensure that /opt/opennms doesn't exist). If the RPMs are already installed (meaning the /opt/opennms directory exists and it is populated), fix the opennms user to work as a regular user.

The following code section illustrates the conditional behavior based on the existence of /opt/opennms:












USER_EXISTS=$(id -u opennms > /dev/null 2>&1; echo $?)
if [ "$USER_EXISTS" == "1" ]; then
  if [ ! -d /opt/opennms ]; then
    sudo useradd -d /opt/opennms -m -c "OpenNMS" opennms
    sudo chmod 775 /opt/opennms
  else
    sudo useradd -d /opt/opennms -c "OpenNMS" opennms
    sudo cp /etc/skel/.bash* /opt/opennms/
    sudo chown opennms:opennms /opt/opennms/.bash*
  fi
fi

Set the password for the opennms user to let operators and administrator access the OpenNMS server via SSH through that user and control the OpenNMS application:



ONMS_PASSWORD="0p3nNM5;"
echo $ONMS_PASSWORD | sudo passwd --stdin opennms

Systemctl

Enable sudo access to execute systemctl and journalctl against the opennms service for the opennms user.

sudo visudo -f /etc/sudoers.d/opennms

Inside the editor, add the following:



Cmnd_Alias SYSTEMD = /bin/journalctl, /bin/systemctl start opennms, /bin/systemctl restart opennms, /bin/systemctl stop opennms, /bin/systemctl status opennms
opennms ALL = NOPASSWD: SYSTEMD

Save the changes.

If operators are more familiar with the service command, update Cmnd_Alias to include those commands as well; for example service start opennms.

Java

For Meridian 2018 and older, or Horizon 23 and older, it is recommended to use Oracle JDK 8 in production, even if it could work with OpenJDK. For Horizon 24 or newer and Meridian 2019 or newer, OpenJDK 11 is recommended.

The usual way to do this is by manually downloading the RPM from java.oracle.com (the latest version is 8u211 by the time this document was written). Unfortunately, this process cannot be automated as it is required to log in using your Oracle account to download Java 8 after the licensing change.

For Horizon 24+/Meridian2019+:





if ! rpm -qa | grep -q java-11-openjdk-devel; then
  echo "Installing latest OpenJDK 11..."
  sudo yum install -y -q java-11-openjdk-devel
fi

Network Privileges for ICMP

There are two valid methodologies to allow non-root users to send ICMP requests.

The first solution works on every Linux distribution, whereas the second one requires a fairly recent kernel.

Using netcap

The following procedure works regardless of the Kernel version used on your Linux Distribution, and it is the reason why the ping command works for unprivileged users.

Update the network raw packets settings for the Java binary to execute ICMP requests as non-root.

For Oracle JDK:




sudo setcap cap_net_raw+ep /usr/java/latest/bin/java
echo /usr/java/latest/lib/amd64/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For OpenJDK 8:




sudo setcap cap_net_raw+ep /usr/lib/jvm/java-1.8.0/bin/java
echo  /usr/lib/jvm/java-1.8.0/lib/amd64/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For OpenJDK 11:




sudo setcap cap_net_raw+ep  /usr/lib/jvm/java-11/bin/java
echo  /usr/lib/jvm/java-11/lib/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For more information check the documentation about CAP_NET_RAW.

Please keep in mind that the setcap command has to be re-executed every time the JDK is upgraded.

Using sysctl

Using this methodology requires a fairly recent Kernel. For instance, this will work on CentOS/RHEL 8 (and modern versions of any Debian based distribution like Ubuntu)However, in CentOS/RHEL 7 it would work only for IPv4, as support for IPv6 was added in Kernel 3.11.




FILE="/etc/sysctl.d/99-zzz-non-root-icmp.conf"
echo "net.ipv4.ping_group_range=0 429496729" | sudo tee $FILE
sudo sysctl -p $FILE

About CentOS 7 or older. If you attempt to use this, you'll see errors like the following in the logs:

2021-03-13 12:54:04,596 ERROR [Main] o.o.n.i.j.Jni6Pinger: Permission error received while attempting to open ICMP socket. See https://wiki.opennms.org/wiki/ICMP for information on configuring ICMP for non-root.

However, RRD/JDB files for ICMP response times seem to be updated, and ICMP outages are reported; but only for IPv4 interfaces.

This option is more strict than using setcap, and if you reduce the scope even more, reduce the group range to cover only the UID of the opennms user, as the above enables ICMP for all users in the system.

Install Haveged

The following helps to bring entropy to the system, required for encrypting algorithms, for instance, SNMPv3. Even if you're not using it, having haveged running can reduce the startup time of OpenNMS.

This package is part of the EPEL repository.



sudo yum install -y -q epel-release
sudo yum install -y -q haveged

Install OpenNMS

For Horizon:




sudo yum install -y -q http://yum.opennms.org/repofiles/opennms-repo-stable-rhel7.noarch.rpm
sudo yum install -y -q jicmp jicmp6 jrrd2 rrdtool
sudo yum install -y -q opennms-core opennms-webapp-jetty opennms-webapp-hawtio

For Meridian 2018 and newer on CentOS/RHEL 7:
















MERIDIAN_USER="your_username_here"
MERIDIAN_PASSWORD="your_password_here"
MERIDIAN_REPO="2018/stable"

cat <<EOF | sudo tee /etc/yum.repos.d/meridian.repo
[meridian]
name=OpenNMS for Red Hat Enterprise Linux and CentOS 7
baseurl="https://$MERIDIAN_USER:$MERIDIAN_PASSWORD@meridian.opennms.com/packages/$MERIDIAN_REPO/rhel7"
enabled=1
gpgcheck=1
sslverify=false
gpgkey=http://yum.opennms.org/OPENNMS-GPG-KEY
EOF
sudo yum install -y -q jicmp jicmp6 jrrd2 rrdtool
sudo yum install -y -q meridian-core meridian-webapp-jetty meridian-webapp-hawtio

For CentOS/RHEL 8, make sure to adjust the URL accordingly.

Update the systemd definition for OpenNMS, to manage the process as non-root

Systemd does not apply the PAM resource limit settings but provides its own set of directives for defining per-unit resource limits. To raise the limits for the number of open files and the number of user processes, add the following two lines to the [Service] section of the unit definition file.




sudo sed -i -r '/User/s/root/opennms/' /lib/systemd/system/opennms.service
sudo sed -i -r '/^.Service/a LimitNOFILE=204800\nLimitNPROC=63409' /lib/systemd/system/opennms.service
sudo systemctl daemon-reload

After the change, the [Service] section should look like this:





[Service]
LimitNOFILE=204800
LimitNPROC=63409
User=opennms

Additionally, setting the resource limits in the Systemd unit does not apply the same limits for interactive login sessions of the opennms service user. This includes the direct use of /opt/opennms/bin/opennms from the shell to stop or start OpenNMS. To set the same values:





cat <<EOF | sudo tee /etc/security/limits.d/100-opennms.conf
opennms    -       nproc     63409
opennms    -       nofile    204800
EOF

The values for nofile and nproc are just examples. Depending on the environment, those values might be requird to be adjusted.

Fix OpenNMS scripts

The following is required to run the OpenNMS scripts as non-root




sudo sed -i -r '/RUNAS/s/root/opennms/' /opt/opennms/bin/opennms
sudo sed -i -r '/RUNAS/s/root/opennms/' /opt/opennms/bin/install
sudo sed -i -r '/RUNAS/s/root/opennms/' /opt/opennms/bin/upgrade

In theory, you can avoid modifying the script by adding RUNAS=opennms to /opt/opennms/etc/opennms.conf; for instance:

echo "RUNAS=opennms" | sudo tee -a /opt/opennms/etc/opennms.conf

Update directory permissions




sudo chown -R opennms:opennms /opt/opennms
sudo chown -R opennms:opennms /var/log/opennms
sudo chown -R opennms:opennms /var/opennms

Update Listeners

Change Trapd port from 162 to 1162 on /opt/opennms/etc/trapd-configuration.xml

sudo sed -r -i 's/snmp-trap-port="[^"]*"/snmp-trap-port="1162"/' /opt/opennms/etc/trapd-configuration.xml

Configuration

The following content is not related to the non-root nature.

The following contains some general advice about how to configure OpenNMS assuming that Kafka will be used for Sink. On Horizon 23 or newer (and for Meridian 2019+), Kafka is possible for RPC. For older versions, ActiveMQ should be used for RPC, which is what is outlined below as this guide was designed for Meridian 2018.

This guide assumes that a centralized Kafka cluster will be shared across multiple OpenNMS instances, and will be used for the Sink API and for the Kafka Producer feature. On recent versions, as mentioned, it can also be used for the RPC API. because of this, the "Instance ID" must be used not only for Kafka but also for ActiveMQ if applies.

  1. Make sure that the Instance ID is properly defined on the OpenNMS Server. Create a file called /opt/opennms/etc/opennms.properties.d/minion.properties with the following content:
org.opennms.instance.id=XXX

On the above example, XXX is the chosen name for the instance ID, and that will be the prefix used on all the Topic/Queue names in Kafka and ActiveMQ.

Check this link for more information.

  1. Make sure to enable the Kafka Sink API for the OpenNMS Server. Create a file called /opt/opennms/etc/opennms.properties.d/kafka.properties with the following content:


org.opennms.core.ipc.sink.strategy=kafka
org.opennms.core.ipc.sink.kafka.bootstrap.servers=kafka1:9092

Remember to change kafka1 with the IP or FQDN of one of the Kafka broker. Alternatively, a list of brokers is also accepted; for instance, kafka1:9092,kafka2:9092.

  1. For M2019+: Make sure to enable the Kafka RPC API for the OpenNMS Server. Create a file called /opt/opennms/etc/opennms.properties.d/kafka.properties with the following content:


org.opennms.core.ipc.rpc.strategy=kafka
org.opennms.core.ipc.rpc.kafka.bootstrap.servers=kafka1:9092

Remember to change kafka1 with the IP or FQDN of one of the Kafka broker. Alternatively, a list of brokers is also accepted; for instance, kafka1:9092,kafka2:9092.

  1. For M2020+: Enable the single-topic feature for RPC. Add the following content to /opt/opennms/etc/opennms.properties.d/kafka.properties with the following content:
org.opennms.core.ipc.rpc.kafka.single-topic=true
  1. Only M2018: Make sure ActiveMQ is enabled and tuned










INSTANCE_ID=XXX
AMQ_FILE=/opt/opennms/etc/opennms-activemq.xml

sudo sed -r -i '/0.0.0.0:61616/s/[<][!]--//' $AMQ_FILE
sudo sed -r -i '/0.0.0.0:61616/s/--[>]//' $AMQ_FILE

sudo sed -r -i '/memoryUsage limit/s/=".*"/="512 mb"/' $AMQ_FILE
sudo sed -r -i '/tempUsage limit/s/=".*"/="512 mb"/' $AMQ_FILE

sudo sed -r -i "/authorizationEntry queue/s/queue=\"[^.]*\./queue=\"$INSTANCE_ID\./" $AMQ_FILE

The first set is to remove the comments around the AMQ listener.

The second set is to increase the limits to avoid issues with RPC when collecting data from big devices.

The third set is to fix the queue template to authorize requests, based on the chosen Instance ID.

Make sure to replace INSTANCE_ID with the chosen Instance ID configured on step 1 for org.opennms.instance.id.

  1. For Syslog processing, on the OpenNMS server, make sure the Syslogd daemon on /opt/opennms/etc/service-configuration.xml looks like this:








<service>
   <name>OpenNMS:Name=Syslogd</name>
   <class-name>org.opennms.netmgt.syslogd.jmx.Syslogd</class-name>
   <invoke method="init" pass="0" at="start"/>
   <invoke method="start" pass="1" at="start"/>
   <invoke method="status" pass="0" at="status"/>
   <invoke method="stop" pass="0" at="stop"/>
</service>

In other words,

sudo sed -r -i '/enabled="false"/{$!{N;s/ enabled="false"[>]\n(.*OpenNMS:Name=Syslogd.*)/>\n\1/}}' /opt/opennms/etc/service-configuration.xml

Make sure to use org.opennms.netmgt.syslogd.RadixTreeSyslogParser as the chosen parser inside /opt/opennms/etc/syslogd-configuration.xml

  1. [Optional] Configure internal firewall (CentOS/RHEL with Firewalld is assumed)

Enable firewalld:



sudo systemctl enable firewalld
sudo systemctl start firewalld

Open ports to access the application:















sudo firewall-cmd --zone=public --add-port=8980/tcp
sudo firewall-cmd --zone=public --add-port=8980/tcp --permanent

sudo firewall-cmd --zone=public --add-port=5817/tcp
sudo firewall-cmd --zone=public --add-port=5817/tcp --permanent

sudo firewall-cmd --zone=public --add-port=8101/tcp
sudo firewall-cmd --zone=public --add-port=8101/tcp --permanent

sudo firewall-cmd --zone=public --add-port=162/udp
sudo firewall-cmd --zone=public --add-port=162/udp --permanent

sudo firewall-cmd --zone=public --add-forward-port=port=162:proto=udp:toport=1162
sudo firewall-cmd --zone=public --add-forward-port=port=162:proto=udp:toport=1162 --permanent

The last 2 entries is to redircat SNMP Traps traffic from UDP 162 to UDP 1162, so OpenNMS can receive them.

If the local ActivMQ is going to be used:



sudo firewall-cmd --zone=public --add-port=61616/tcp
sudo firewall-cmd --zone=public --add-port=61616/tcp --permanent

If OpenNMS will be using HTTPS or expose the WebUI on another port, make sure to fix the first rule with the appropriate port.

To verify the firewall rules:

sudo firewall-cmd --list-all --zone public

Or to check everything:

sudo firewall-cmd  --list-all-zones
  1. Configure JDK

Prior start OpenNMS, it is important to set the exact path to the Java binary to avoid issues with setcap.

For Oracle JDK:

/opt/opennms/bin/runjava -S /usr/java/latest/bin/java

For OpenJDK 8:

/opt/opennms/bin/runjava -S /usr/lib/jvm/java-1.8.0/bin/java

For OpenJDK 11 (for H24+ or Meridian 2019+):

/opt/opennms/bin/runjava -S /usr/lib/jvm/java-11/bin/java
  1. Execute the install script
/opt/opennms/bin/install -dis
  1. Start OpenNMS
sudo systemctl start opennms
  1. Create a user for the Minions (tested on M2018 only)












MINION_USER="minion"
MINION_PASSWD="minion"

curl -u admin:admin -H "Content-Type: application/xml" -d "
<user>
  <user-id>$MINION_USER</user-id>
  <full-name>Minion User</full-name>
  <user-comments>A user to authenticate Minions</user-comments>
  <password>$MINION_PASSWD</password>
</user>
" http://localhost:8980/opennms/rest/users
curl -u admin:admin -X PUT http://localhost:8980/opennms/rest/users/minion/roles/ROLE_MINION
  1. Change the password for the admin account using the WebUI.
Last changed by 
 
Alejandro Galue
0
747

Read more

Sizing Cassandra

The use case for having Cassandra (or ScyllaDB) as the backend for the performance metrics in OpenNMS is for storing a huge amount of non-aggregated data, which is not possible with RRDtool. RRDtool is very good for those installations with a finite and predictable amount of metrics where the size and the I/O requirements are feasible for modern SSD disks. This is important as RRDtool only scales vertically, meaning that when the current disks&apos; limits are reached (mostly due to speed, not space), a faster disk is required. This is when Cassandra or ScyllaDB can help. Although there is a high learning curve, using any of these applications requires a commitment to have qualified personnel to manage this database. ScyllaDB is binary compatible with Cassandra (even at the SSTable level), but they are implemented very differently. Cassandra is implemented in Java, meaning that all the JVM tuning is required and other internal Cassandra tunings. On the other hand, ScyllaDB is implemented in modern C++, taking advantage of the CPU where it is running. That means ScyllaDB can manage huge machines as nodes, whereas Cassandra would require multiple instances on huge machines. It is feasible to have faster results with ScyllaDB compared with its Java sibling in terms of performance. Configuring and managing both applications is different, even if they will provide the same operational result with OpenNMS, so the decision should be carefully analyzed, especially by the team that will support this database.

May 4, 2022
Simple OpenNMS Environment using ScyllaDB

This lab starts an OpenNMS instance, a 3-node ScyllaDB cluster, and optionally an instance for the Scylla Monitoring Stack in the cloud, for learning purposes. To monitor a network, it is advised to enable ActiveMQ or Kafka and use Minions. For simplicity, the embedded ActiveMQ will be enabled. :::success Follow this guide for general information about how to configure Scylla/Cassandra for OpenNMS/Newts. Most of that knowledge is transferable to ScyllaDB. ::: Requirements

Nov 19, 2021
Simple OpenNMS/Minion Environment using Kafka in Azure

This lab starts an OpenNMS instance and a 3 node ZK/Kafka cluster in the cloud and two Minions on your machine, using Kafka for communication through Multipass and Azure, for learning purposes. :::warning The lab doesn&apos;t cover security by default (user authentication and encryption), which is crucial if we ever want to expose the Kafka cluster to the Internet. A separate section covers the required changes for this. ::: :::success Keep in mind that nothing prevents us from skipping using the cloud provider and do everything with Multipass (or VirtualBox, or Hyper-V, or VMWare). The reason for using a cloud provider is to prove that OpenNMS can monitor unreachable devices via Minion. Similarly, we could use any other cloud provider instead of Azure. However I won&apos;t explain how to port the solution here. :::

Nov 19, 2021
Simple OpenNMS/Minion Environment using the embedded ActiveMQ in Azure

This lab starts an OpenNMS instance in the cloud and two Minions on your machine, using ActiveMQ for communication through Multipass and Azure, for learning purposes. :::danger The lab doesn&apos;t cover security (in terms of encryption), which is crucial if you ever want to expose AMQ to the Internet. Follow this guide to learn how to do it with LetsEncrypt. ::: :::success Keep in mind that nothing prevents you from skipping using the cloud provider and do everything with Multipass (or VirtualBox, or Hyper-V, or VMWare). The reason for using a cloud provider is to prove that OpenNMS can monitor unreachable devices via Minion. Similarly, you could use any other cloud provider instead of Azure. However I won&apos;t explain how to port the solution here. :::

Nov 19, 2021
Read more from Alejandro Galue
Published on HackMD