Install, configure, manage and run Minion as non-root

This guide was designed for Meridian 2018 but should work on Horizon 22 or newer versions of OpenNMS. I'm assuming the operating system will be CentOS/RHEL 7, but all the commands should work in 8.

Any yum command will internally use dnf if you're using CentOS/RHEL 8.

Installation

The installation process requires privileges. For this reason, the administrator must execute all the following steps as root or through sudo.

User for Minion

Create the minion user before installing the OpenNMS RPMs (ensure that /opt/minion doesn't exist). If the RPMs are already installed (meaning the /opt/minion directory exists and it is populated), fix the minion user to work as a regular user.

The following code section illustrates the conditional behavior based on the existence of /opt/minion:











USER_EXISTS=$(id -u minion > /dev/null 2>&1; echo $?)
if [ "$USER_EXISTS" == "1" ]; then
  if [ ! -d /opt/minion ]; then
    sudo useradd -d /opt/minion -m -c "OpenNMS Minion" minion
    sudo chmod 775 /opt/minion
  else
    sudo usermod -s /bin/bash minion
    sudo cp /etc/skel/.bash* /opt/minion/
    sudo chown minion:minion /opt/minion/.bash*
  fi
fi

Set the password for the minion user to let operators and administrator access the Minion server via SSH through that user and control the Minion application:


MINION_PASSWORD="m1n10n;"
echo $MINION_PASSWORD | sudo passwd --stdin minion

Systemctl

Enable sudo access to execute systemctl and journalctl against the minion service for the minion user.


sudo visudo -f /etc/sudoers.d/opennms

Inside the editor, add the following:


Cmnd_Alias SYSTEMD = /usr/bin/journalctl, /usr/bin/systemctl start minion, /usr/bin/systemctl restart minion, /usr/bin/systemctl stop minion, /bin/systemctl status minion
minion ALL = NOPASSWD: SYSTEMD

Save the changes.

If operators are more familiar with the service command, update Cmnd_Alias to include those commands as well; for example service start minion.

Java

For Meridian 2018 and older, or Horizon 23 and older, it is recommended to use Oracle JDK 8 in production, even if it could work with OpenJDK. For Horizon 24 or newer and Meridian 2019 or newer, OpenJDK 11 is recommended.

The usual way to do this is by manually downloading the RPM from java.oracle.com (the latest version is 8u211 by the time this document was written). Unfortunately, this process cannot be automated as it is required to log in using your Oracle account to download Java 8 after the licensing change.

For Horizon 24+/Meridian2019+:




if ! rpm -qa | grep -q java-11-openjdk-devel; then
  echo "Installing latest OpenJDK 11..."
  sudo yum install -y -q java-11-openjdk-devel
fi

Network Privileges for ICMP

There are two valid methodologies to allow non-root users to send ICMP requests.

The first solution works on every Linux distribution, whereas the second one requires a fairly recent kernel.

Using netcap

The following procedure works regardless of the Kernel version used on your Linux Distribution, and it is the reason why the ping command works for unprivileged users.

Update the network raw packets settings for the Java binary to execute ICMP requests as non-root.

For Oracle JDK:



sudo setcap cap_net_raw+ep /usr/java/latest/bin/java
echo /usr/java/latest/lib/amd64/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For OpenJDK 8:



sudo setcap cap_net_raw+ep /usr/lib/jvm/java-1.8.0/bin/java
echo  /usr/lib/jvm/java-1.8.0/lib/amd64/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For OpenJDK 11:



sudo setcap cap_net_raw+ep  /usr/lib/jvm/java-11/bin/java
echo  /usr/lib/jvm/java-11/lib/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For more information check the documentation about CAP_NET_RAW.

Please keep in mind that the setcap command has to be re-executed every time the JDK is upgraded.

Using sysctl

Using this methodology requires a fairly recent Kernel. For instance, this will work on CentOS/RHEL 8 (and modern versions of any Debian based distribution like Ubuntu). However, in CentOS/RHEL 7 it would work only for IPv4, as support for IPv6 was added in Kernel 3.11.



FILE="/etc/sysctl.d/99-zzz-non-root-icmp.conf"
echo "net.ipv4.ping_group_range=0 429496729" | sudo tee $FILE
sudo sysctl -p $FILE

About CentOS 7 or older. If you attempt to use this, you'll see errors like the following in the logs:


2021-03-13 12:54:04,596 ERROR [Main] o.o.n.i.j.Jni6Pinger: Permission error received while attempting to open ICMP socket. See https://wiki.opennms.org/wiki/ICMP for information on configuring ICMP for non-root.

However, RRD/JDB files for ICMP response times seem to be updated, and ICMP outages are reported; but only for IPv4 interfaces.

This option is more strict than using setcap, and if you reduce the scope even more, reduce the group range to cover only the UID of the opennms user, as the above enables ICMP for all users in the system.

Install Minion

Follow the usual procedure to install RPMs on CentOS/RHEL.

For Horizon:



sudo yum install -y -q http://yum.opennms.org/repofiles/opennms-repo-stable-rhel7.noarch.rpm
sudo yum install -y -q jicmp jicmp6
sudo yum install -y -q opennms-minion

For Meridian 2018 and newer:















MERIDIAN_USER="your_username_here"
MERIDIAN_PASSWORD="your_password_here"
MERIDIAN_REPO="2018/stable"

cat <<EOF | sudo tee /etc/yum.repos.d/meridian.repo
[meridian]
name=OpenNMS for Red Hat Enterprise Linux and CentOS 7
baseurl="https://$MERIDIAN_USER:$MERIDIAN_PASSWORD@meridian.opennms.com/packages/$MERIDIAN_REPO/rhel7"
enabled=1
gpgcheck=1
sslverify=false
gpgkey=http://yum.opennms.org/OPENNMS-GPG-KEY
EOF
sudo yum install -y -q jicmp jicmp6
sudo yum install -y -q meridian-minion

Move the `init` script


sudo mv /etc/init.d/minion /opt/minion/bin/

Due to a known issue, a fixed version of /etc/init.d/minion is required when running Meridian 2018.1.5 or older. If this is the case, make sure to grab a copy of this script from the Horizon RPM. This is not required with latest version.

Create a Systemd unit for Minion

The following is optional but recommended:























cat <<EOF | sudo tee /etc/systemd/system/minion.service
[Unit]
Description=OpenNMS Minion server
Requires=network.target network-online.target
After=network.target network-online.target

[Service]
User=minion
LimitNOFILE=204800
LimitNPROC=63409

Type=forking
PIDFile=/opt/minion/karaf.pid

ExecStart=/opt/minion/bin/minion start
ExecStop=/opt/minion/bin/minion stop

[Install]
WantedBy=multi-user.target
EOF
sudo chown minion:minion /etc/systemd/system/minion.service
sudo systemctl daemon-reload
sudo systemctl enable minion

At this point, a user can login with the minion account, and should be able to manage the minion application through systemctl. All the content of /opt/minion is owned by the minion user, which makes the whole application fully configurable.

JVM Settings

Update /etc/sysconfig/minion for standard tuning and enable non-root nature.
















total_mem_in_mb=$(free -m | awk '/:/ {print $2;exit}')
mem_in_mb=$(expr $total_mem_in_mb / 2)
if [ "$mem_in_mb" -gt "8192" ]; then
mem_in_mb="8192"
fi
half_mem_in_mb=$(expr $mem_in_mb / 2)
java_opts="-XX:+UseG1GC"
sysconfig=/etc/sysconfig/minion
sudo sed -r -i "/export JAVA_MAX_MIM/s/^# //" $sysconfig
sudo sed -i -r "/export JAVA_MAX_MIM/s/=.*/=${half_mem_in_mb}M/" $sysconfig
sudo sed -r -i "/export JAVA_MAX_MEM/s/^# //" $sysconfig
sudo sed -i -r "/export JAVA_MAX_MEM/s/=.*/=${mem_in_mb}M/" $sysconfig
sudo sed -r -i "/export RUNAS/s/^# //" $sysconfig
sudo sed -i -r "/export RUNAS/s/=.*/=minion/" $sysconfig
sudo sed -r -i "/export JAVA_OPTS/s/^# //" $sysconfig
sudo sed -i -r "/export JAVA_OPTS/s/=.*/=\"$java_opts\"/" $sysconfig

The last entry is to avoid checking ICMP bits at kernel level, which won't work on CentOS/RHEL 7. The setcap on tha java binary is enough to perform ICMP requests through a non-root user on OS with old kernels.

The following applies only when using Meridian 2018.1.5 or older:



cat <<EOF | sudo tee -a /etc/sysconfig/minion
export PING_REQUIRED=FALSE
EOF

Credentials

Configure the credentials to access the OpenNMS ReST API and the ActiveMQ broker (if apply).

By default, Minion will use the admin user with the admin password. The idea is create a user in OpenNMS called minion, and assign the ROLE_MINION to it. Then, update the credentials:




MINION_USER="minion"
MINION_PASSWD="minion"
sudo /opt/minion/bin/scvcli set opennms.http $MINION_USER $MINION_PASSWD
sudo /opt/minion/bin/scvcli set opennms.broker $MINION_USER $MINION_PASSWD

WARNING: MINION_USER and MINION_PASSWD must match the user created inside users.xml in the OpenNMS server.

Permissions

If Minion was previously updated, fix the permissions


sudo chown -R minion:minion /opt/minion

Configuration

The following content is not related to the non-root nature.

The following contains some general advice about how to configure Minion assuming that Kafka will be used for Sink. On Horizon 23 or newer (and for Meridian 2019+), Kafka is possible for RPC. For older versions, ActiveMQ should be used for RPC, which is what is outlined below as this guide was designed for Meridian 2018.

This guide assumes that a centralized Kafka cluster will be shared across multiple OpenNMS instances, and will be used for the Sink API and for the Kafka Producer feature. On recent versions, as mentioned, it can also be used for the RPC API. because of this, the "Instance ID" must be used not only for Kafka but also for ActiveMQ if applies.

Make sure that OpenNMS has been configured as described here.

Make sure that the Instance ID is properly defined on the Minions associated with the backend configured on step 1. Append the following content to /opt/minion/etc/system.properties:


org.opennms.instance.id=XXX

On the above example, XXX is the chosen name for the instance ID, and that will be the prefix used on all the Topic/Queue names in Kafka and ActiveMQ.

It has to be be same content used for OpenNMS.

Make sure to enable the Kafka Sink API for the Minions. Create a file called /opt/minion/etc/featuresBoot.d/kafka.boot with the following content:


!opennms-core-ipc-sink-camel
opennms-core-ipc-sink-kafka

And configure the Kafka broker on /opt/minion/etc/org.opennms.core.ipc.sink.kafka.cfg, the content should look like this:


bootstrap.servers=kafka1:9092

Use the same broker list used for OpenNMS. For more information, check here.

Remember to change kafka1 with the IP or FQDN of one of the Kafka broker. Alternatively, a list of brokers is also accepted; for instance, kafka1:9092,kafka2:9092.

For M2019+. Enable the Kafka RPC API for the Minions.

Create a file called /opt/minion/etc/featuresBoot.d/kafka.boot with the following content:


!opennms-core-ipc-rpc-camel
opennms-core-ipc-rpc-kafka

And the bootstrap servers to /opt/minion/etc/org.opennms.core.ipc.rpc.kafka.cfg. The content should look like this:


bootstrap.servers=kafka1:9092

Use the same broker list used for OpenNMS. For more information, check here.

Remember to change kafka1 with the IP or FQDN of one of the Kafka broker. Alternatively, a list of brokers is also accepted; for instance, kafka1:9092,kafka2:9092.

For M2020+: Enable the single-topic feature for RPC. Add the following content to /opt/minion/etc/org.opennms.core.ipc.rpc.kafka.cfg with the following content:


single-topic=true

Configure the location, opennms-url and broker-url for Minion on org.opennms.minion.controller.cfg; for example:




location=Apex
id=onms-minion.local
http-url=http://192.168.205.168:8980/opennms
broker-url=failover:tcp://192.168.205.168:61616

Remember to change 192.168.205.168 with the IP or FQDN of the OpenNMS server. The above example assumes the usaged of the embedded AMQ within OpenNMS. Alternatively, An external AMQ server can be used. When using Kafka for RPC and Sink, broker-url can be omitted (it will be ignored).

Configure Trap/Syslog reception on Minion

Add the following to /opt/minion/etc/org.opennms.netmgt.trapd.cfg:








trapd.listen.interface=0.0.0.0
trapd.listen.port=1162

# controls how many traps are included in a single message sent to Kafka
trapd.batch.size=10

# limits how many messages are kept in memory if Kafka is unreachable
trapd.queue.size=10000

Add the following to /opt/minion/etc/org.opennms.netmgt.syslog.cfg:








syslog.listen.interface=0.0.0.0
syslog.listen.port=1514

# controls how many traps are included in a single message sent to Kafka
syslog.batch.size=10

# limits how many messages are kept in memory if Kafka is unreachable
syslog.queue.size=10000

The queue size must be consistent with the message/buffer limits at Kafka Broker level. This is important for Meridian 2018 when using Kafka, as with Horizon 24+ or newer, the messages are split into multiple chunks to avoid hitting Kafka limits.

Speaking about Kafka Limits, the following is required for the Sink API:


echo "max.request.size=5000000" >> /opt/minion/etc/org.opennms.core.ipc.sink.kafka.cfg

For RPC (not available on Meridian 2018):


echo "max.request.size=5000000" >> /opt/minion/etc/org.opennms.core.rpc.sink.kafka.cfg

Minion would have to be restarted after the above changes.

That also requires changes on each Kafka broker and OpenNMS.

At Kafka broker level:





cat <<EOF | tee -a /opt/kafka/conf/server.properties
message.max.bytes=5000000
replica.fetch.max.bytes=5000000
compression.type=producer
EOF

At OpenNMS level, for Sink:



cat <<EOF | tee -a /opt/opennms/etc/opennms.properties.d/kafka.properties
org.opennms.core.ipc.sink.kafka.max.partition.fetch.bytes=5000000
EOF

And for RPC (not available on Meridian 2018):







cat <<EOF | tee -a /opt/opennms/etc/opennms.properties.d/kafka.properties
# RPC Consumer (verify Kafka broker configuration)
org.opennms.core.ipc.rpc.kafka.max.partition.fetch.bytes=5000000
org.opennms.core.ipc.rpc.kafka.auto.offset.reset=latest
# RPC Producer (verify Kafka broker configuration)
org.opennms.core.ipc.rpc.kafka.max.request.size=5000000
EOF

Configure internal firewall (CentOS/RHEL with Firewalld is assumed)

Enable firewalld:


sudo systemctl enable firewalld
sudo systemctl start firewalld

Open the UDP ports 162 and 514 on the local firewall, to allow the Minion Server to receive information through them:





sudo firewall-cmd --zone=public --add-port=162/udp
sudo firewall-cmd --zone=public --add-port=162/udp --permanent

sudo firewall-cmd --zone=public --add-port=514/udp
sudo firewall-cmd --zone=public --add-port=514/udp --permanent

Enable port forwarding, in order to forward from UDP 162 to UDP 1162, and from UDP 514 to UDP 1514; to guarantee the Minion application receives the messages.





sudo firewall-cmd --zone="public" --add-forward-port=port=162:proto=udp:toport=1162
sudo firewall-cmd --zone="public" --add-forward-port=port=162:proto=udp:toport=1162 --permanent

sudo firewall-cmd --zone="public" --add-forward-port=port=514:proto=udp:toport=1514
sudo firewall-cmd --zone="public" --add-forward-port=port=514:proto=udp:toport=1514 --permanent

Do not associate or constraint the rules against 127.0.0.1 as the official documentation suggests, to be consistent with the listen.interface entry used on the configuration files.

To verify the firewall rules:


sudo firewall-cmd --list-all --zone public

Or to check everything:


sudo firewall-cmd  --list-all-zones

Start Minion


systemctl start minion