Install, configure, manage and run OpenNMS as non-root

This guide was designed for Meridian 2018 but should work on Horizon 22 or newer versions of OpenNMS. I'm assuming the operating system will be CentOS/RHEL 7, but all the commands should work in 8.

Any yum command will internally use dnf if you're using CentOS/RHEL 8.

Installation

The installation process requires privileges. For this reason, the administrator must execute all the following steps as root or through sudo.

User for OpenNMS

Create the opennms user before installing the OpenNMS RPMs (ensure that /opt/opennms doesn't exist). If the RPMs are already installed (meaning the /opt/opennms directory exists and it is populated), fix the opennms user to work as a regular user.

The following code section illustrates the conditional behavior based on the existence of /opt/opennms:

USER_EXISTS=$(id -u opennms > /dev/null 2>&1; echo $?)
if [ "$USER_EXISTS" == "1" ]; then
  if [ ! -d /opt/opennms ]; then
    sudo useradd -d /opt/opennms -m -c "OpenNMS" opennms
    sudo chmod 775 /opt/opennms
  else
    sudo useradd -d /opt/opennms -c "OpenNMS" opennms
    sudo cp /etc/skel/.bash* /opt/opennms/
    sudo chown opennms:opennms /opt/opennms/.bash*
  fi
fi

Set the password for the opennms user to let operators and administrator access the OpenNMS server via SSH through that user and control the OpenNMS application:

ONMS_PASSWORD="0p3nNM5;"
echo $ONMS_PASSWORD | sudo passwd --stdin opennms

Systemctl

Enable sudo access to execute systemctl and journalctl against the opennms service for the opennms user.

sudo visudo -f /etc/sudoers.d/opennms

Inside the editor, add the following:

Cmnd_Alias SYSTEMD = /bin/journalctl, /bin/systemctl start opennms, /bin/systemctl restart opennms, /bin/systemctl stop opennms, /bin/systemctl status opennms
opennms ALL = NOPASSWD: SYSTEMD

Save the changes.

Java

For Meridian 2018 and older, or Horizon 23 and older, it is recommended to use Oracle JDK 8 in production, even if it could work with OpenJDK. For Horizon 24 or newer and Meridian 2019 or newer, OpenJDK 11 is recommended.

The usual way to do this is by manually downloading the RPM from java.oracle.com (the latest version is 8u211 by the time this document was written). Unfortunately, this process cannot be automated as it is required to log in using your Oracle account to download Java 8 after the licensing change.

For Horizon 24+/Meridian2019+:

if ! rpm -qa | grep -q java-11-openjdk-devel; then
  echo "Installing latest OpenJDK 11..."
  sudo yum install -y -q java-11-openjdk-devel
fi

Network Privileges for ICMP

There are two valid methodologies to allow non-root users to send ICMP requests.

The first solution works on every Linux distribution, whereas the second one requires a fairly recent kernel.

Using netcap

Update the network raw packets settings for the Java binary to execute ICMP requests as non-root.

For Oracle JDK:

sudo setcap cap_net_raw+ep /usr/java/latest/bin/java
echo /usr/java/latest/lib/amd64/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For OpenJDK 8:

sudo setcap cap_net_raw+ep /usr/lib/jvm/java-1.8.0/bin/java
echo  /usr/lib/jvm/java-1.8.0/lib/amd64/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For OpenJDK 11:

sudo setcap cap_net_raw+ep  /usr/lib/jvm/java-11/bin/java
echo  /usr/lib/jvm/java-11/lib/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For more information check the documentation about CAP_NET_RAW.

Using sysctl

FILE="/etc/sysctl.d/99-zzz-non-root-icmp.conf"
echo "net.ipv4.ping_group_range=0 429496729" | sudo tee $FILE
sudo sysctl -p $FILE

About CentOS 7 or older. If you attempt to use this, you'll see errors like the following in the logs:

2021-03-13 12:54:04,596 ERROR [Main] o.o.n.i.j.Jni6Pinger: Permission error received while attempting to open ICMP socket. See https://wiki.opennms.org/wiki/ICMP for information on configuring ICMP for non-root.

However, RRD/JDB files for ICMP response times seem to be updated, and ICMP outages are reported; but only for IPv4 interfaces.

This option is more strict than using setcap, and if you reduce the scope even more, reduce the group range to cover only the UID of the opennms user, as the above enables ICMP for all users in the system.

Install Haveged

The following helps to bring entropy to the system, required for encrypting algorithms, for instance, SNMPv3. Even if you're not using it, having haveged running can reduce the startup time of OpenNMS.

This package is part of the EPEL repository.

sudo yum install -y -q epel-release
sudo yum install -y -q haveged

Install OpenNMS

For Horizon:

sudo yum install -y -q http://yum.opennms.org/repofiles/opennms-repo-stable-rhel7.noarch.rpm
sudo yum install -y -q jicmp jicmp6 jrrd2 rrdtool
sudo yum install -y -q opennms-core opennms-webapp-jetty opennms-webapp-hawtio

For Meridian 2018 and newer on CentOS/RHEL 7:

MERIDIAN_USER="your_username_here"
MERIDIAN_PASSWORD="your_password_here"
MERIDIAN_REPO="2018/stable"

cat <<EOF | sudo tee /etc/yum.repos.d/meridian.repo
[meridian]
name=OpenNMS for Red Hat Enterprise Linux and CentOS 7
baseurl="https://$MERIDIAN_USER:$MERIDIAN_PASSWORD@meridian.opennms.com/packages/$MERIDIAN_REPO/rhel7"
enabled=1
gpgcheck=1
sslverify=false
gpgkey=http://yum.opennms.org/OPENNMS-GPG-KEY
EOF
sudo yum install -y -q jicmp jicmp6 jrrd2 rrdtool
sudo yum install -y -q meridian-core meridian-webapp-jetty meridian-webapp-hawtio

Update the systemd definition for OpenNMS, to manage the process as non-root

Systemd does not apply the PAM resource limit settings but provides its own set of directives for defining per-unit resource limits. To raise the limits for the number of open files and the number of user processes, add the following two lines to the [Service] section of the unit definition file.

sudo sed -i -r '/User/s/root/opennms/' /lib/systemd/system/opennms.service
sudo sed -i -r '/^.Service/a LimitNOFILE=204800\nLimitNPROC=63409' /lib/systemd/system/opennms.service
sudo systemctl daemon-reload

After the change, the [Service] section should look like this:

[Service]
LimitNOFILE=204800
LimitNPROC=63409
User=opennms

Additionally, setting the resource limits in the Systemd unit does not apply the same limits for interactive login sessions of the opennms service user. This includes the direct use of /opt/opennms/bin/opennms from the shell to stop or start OpenNMS. To set the same values:

cat <<EOF | sudo tee /etc/security/limits.d/100-opennms.conf
opennms    -       nproc     63409
opennms    -       nofile    204800
EOF

Fix OpenNMS scripts

The following is required to run the OpenNMS scripts as non-root

sudo sed -i -r '/RUNAS/s/root/opennms/' /opt/opennms/bin/opennms
sudo sed -i -r '/RUNAS/s/root/opennms/' /opt/opennms/bin/install
sudo sed -i -r '/RUNAS/s/root/opennms/' /opt/opennms/bin/upgrade

In theory, you can avoid modifying the script by adding RUNAS=opennms to /opt/opennms/etc/opennms.conf; for instance:

echo "RUNAS=opennms" | sudo tee -a /opt/opennms/etc/opennms.conf

Update directory permissions

sudo chown -R opennms:opennms /opt/opennms
sudo chown -R opennms:opennms /var/log/opennms
sudo chown -R opennms:opennms /var/opennms

Update Listeners

Change Trapd port from 162 to 1162 on /opt/opennms/etc/trapd-configuration.xml

sudo sed -r -i 's/snmp-trap-port="[^"]*"/snmp-trap-port="1162"/' /opt/opennms/etc/trapd-configuration.xml

Configuration

This guide assumes that a centralized Kafka cluster will be shared across multiple OpenNMS instances, and will be used for the Sink API and for the Kafka Producer feature. On recent versions, as mentioned, it can also be used for the RPC API. because of this, the "Instance ID" must be used not only for Kafka but also for ActiveMQ if applies.

1. Make sure that the Instance ID is properly defined on the OpenNMS Server. Create a file called /opt/opennms/etc/opennms.properties.d/minion.properties with the following content:

org.opennms.instance.id=XXX

On the above example, XXX is the chosen name for the instance ID, and that will be the prefix used on all the Topic/Queue names in Kafka and ActiveMQ.

Check this link for more information.

2. Make sure to enable the Kafka Sink API for the OpenNMS Server. Create a file called /opt/opennms/etc/opennms.properties.d/kafka.properties with the following content:

org.opennms.core.ipc.sink.strategy=kafka
org.opennms.core.ipc.sink.kafka.bootstrap.servers=kafka1:9092

3. For M2019+: Make sure to enable the Kafka RPC API for the OpenNMS Server. Create a file called /opt/opennms/etc/opennms.properties.d/kafka.properties with the following content:

org.opennms.core.ipc.rpc.strategy=kafka
org.opennms.core.ipc.rpc.kafka.bootstrap.servers=kafka1:9092

4. For M2020+: Enable the single-topic feature for RPC. Add the following content to /opt/opennms/etc/opennms.properties.d/kafka.properties with the following content:

org.opennms.core.ipc.rpc.kafka.single-topic=true

5. Only M2018: Make sure ActiveMQ is enabled and tuned

INSTANCE_ID=XXX
AMQ_FILE=/opt/opennms/etc/opennms-activemq.xml

sudo sed -r -i '/0.0.0.0:61616/s/[<][!]--//' $AMQ_FILE
sudo sed -r -i '/0.0.0.0:61616/s/--[>]//' $AMQ_FILE

sudo sed -r -i '/memoryUsage limit/s/=".*"/="512 mb"/' $AMQ_FILE
sudo sed -r -i '/tempUsage limit/s/=".*"/="512 mb"/' $AMQ_FILE

sudo sed -r -i "/authorizationEntry queue/s/queue=\"[^.]*\./queue=\"$INSTANCE_ID\./" $AMQ_FILE

The first set is to remove the comments around the AMQ listener.

The second set is to increase the limits to avoid issues with RPC when collecting data from big devices.

The third set is to fix the queue template to authorize requests, based on the chosen Instance ID.

6. For Syslog processing, on the OpenNMS server, make sure the Syslogd daemon on /opt/opennms/etc/service-configuration.xml looks like this:

<service>
   <name>OpenNMS:Name=Syslogd</name>
   <class-name>org.opennms.netmgt.syslogd.jmx.Syslogd</class-name>
   <invoke method="init" pass="0" at="start"/>
   <invoke method="start" pass="1" at="start"/>
   <invoke method="status" pass="0" at="status"/>
   <invoke method="stop" pass="0" at="stop"/>
</service>

In other words,

sudo sed -r -i '/enabled="false"/{$!{N;s/ enabled="false"[>]\n(.*OpenNMS:Name=Syslogd.*)/>\n\1/}}' /opt/opennms/etc/service-configuration.xml

7. [Optional] Configure internal firewall (CentOS/RHEL with Firewalld is assumed)

Enable firewalld:

sudo systemctl enable firewalld
sudo systemctl start firewalld

Open ports to access the application:

sudo firewall-cmd --zone=public --add-port=8980/tcp
sudo firewall-cmd --zone=public --add-port=8980/tcp --permanent

sudo firewall-cmd --zone=public --add-port=5817/tcp
sudo firewall-cmd --zone=public --add-port=5817/tcp --permanent

sudo firewall-cmd --zone=public --add-port=8101/tcp
sudo firewall-cmd --zone=public --add-port=8101/tcp --permanent

sudo firewall-cmd --zone=public --add-port=162/udp
sudo firewall-cmd --zone=public --add-port=162/udp --permanent

sudo firewall-cmd --zone=public --add-forward-port=port=162:proto=udp:toport=1162
sudo firewall-cmd --zone=public --add-forward-port=port=162:proto=udp:toport=1162 --permanent

The last 2 entries is to redircat SNMP Traps traffic from UDP 162 to UDP 1162, so OpenNMS can receive them.

If the local ActivMQ is going to be used:

sudo firewall-cmd --zone=public --add-port=61616/tcp
sudo firewall-cmd --zone=public --add-port=61616/tcp --permanent

To verify the firewall rules:

sudo firewall-cmd --list-all --zone public

Or to check everything:

sudo firewall-cmd  --list-all-zones

8. Configure JDK

Prior start OpenNMS, it is important to set the exact path to the Java binary to avoid issues with setcap.

For Oracle JDK:

/opt/opennms/bin/runjava -S /usr/java/latest/bin/java

For OpenJDK 8:

/opt/opennms/bin/runjava -S /usr/lib/jvm/java-1.8.0/bin/java

For OpenJDK 11 (for H24+ or Meridian 2019+):

/opt/opennms/bin/runjava -S /usr/lib/jvm/java-11/bin/java

9. Execute the install script

/opt/opennms/bin/install -dis

10. Start OpenNMS

sudo systemctl start opennms

11. Create a user for the Minions (tested on M2018 only)

MINION_USER="minion"
MINION_PASSWD="minion"

curl -u admin:admin -H "Content-Type: application/xml" -d "
<user>
  <user-id>$MINION_USER</user-id>
  <full-name>Minion User</full-name>
  <user-comments>A user to authenticate Minions</user-comments>
  <password>$MINION_PASSWD</password>
</user>
" http://localhost:8980/opennms/rest/users
curl -u admin:admin -X PUT http://localhost:8980/opennms/rest/users/minion/roles/ROLE_MINION

12. Change the password for the admin account using the WebUI.