# Install, configure, manage and run Minion as non-root

This guide was designed for Meridian 2018 but should work on Horizon 22 or newer versions of OpenNMS. I'm assuming the operating system will be CentOS/RHEL 7, but all the commands should work in 8.

Any `yum` command will internally use `dnf` if you're using CentOS/RHEL 8.

## Installation

The installation process requires privileges. For this reason, the administrator must execute all the following steps as `root` or through `sudo`.

### User for Minion

Create the `minion` user before installing the OpenNMS RPMs (ensure that `/opt/minion` doesn't exist). If the RPMs are already installed (meaning the `/opt/minion` directory exists and it is populated), fix the `minion` user to work as a regular user.

The following code section illustrates the conditional behavior based on the existence of `/opt/minion`:

USER_EXISTS=$(id -u minion > /dev/null 2>&1; echo $?)
if [ "$USER_EXISTS" == "1" ]; then
  if [ ! -d /opt/minion ]; then
    sudo useradd -d /opt/minion -m -c "OpenNMS Minion" minion
    sudo chmod 775 /opt/minion
  else
    sudo usermod -s /bin/bash minion
    sudo cp /etc/skel/.bash* /opt/minion/
    sudo chown minion:minion /opt/minion/.bash*
  fi
fi

Set the password for the `minion` user to let operators and administrator access the Minion server via SSH through that user and control the Minion application:

MINION_PASSWORD="m1n10n;"
echo $MINION_PASSWORD | sudo passwd --stdin minion 

### Systemctl

Enable `sudo` access to execute `systemctl` and `journalctl` against the `minion` service for the `minion` user.

sudo visudo -f /etc/sudoers.d/opennms

Inside the editor, add the following:

Cmnd_Alias SYSTEMD = /usr/bin/journalctl, /usr/bin/systemctl start minion, /usr/bin/systemctl restart minion, /usr/bin/systemctl stop minion, /bin/systemctl status minion
minion ALL = NOPASSWD: SYSTEMD

Save the changes.

:::info
If operators are more familiar with the `service` command, update `Cmnd_Alias` to include those commands as well; for example `service start minion`.
:::

### Java

For Meridian 2018 and older, or Horizon 23 and older, it is recommended to use Oracle JDK 8 in production, even if it could work with OpenJDK. For Horizon 24 or newer and Meridian 2019 or newer, OpenJDK 11 is recommended.

The usual way to do this is by manually downloading the RPM from `java.oracle.com` (the latest version is 8u211 by the time this document was written). Unfortunately, this process cannot be automated as it is required to log in using your Oracle account to download Java 8 after the licensing change.

For Horizon 24+/Meridian2019+:

if ! rpm -qa | grep -q java-11-openjdk-devel; then
  echo "Installing latest OpenJDK 11..."
  sudo yum install -y -q java-11-openjdk-devel
fi

### Network Privileges for ICMP

There are two valid methodologies to allow non-root users to send ICMP requests.

The first solution works on every Linux distribution, whereas the second one requires a fairly recent kernel.

#### Using netcap

:::info
The following procedure works regardless of the Kernel version used on your Linux Distribution, and it is the reason why the `ping` command works for unprivileged users.
:::

Update the network raw packets settings for the Java binary to execute ICMP requests as non-root.

For Oracle JDK:

sudo setcap cap_net_raw+ep /usr/java/latest/bin/java
echo /usr/java/latest/lib/amd64/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For OpenJDK 8:

sudo setcap cap_net_raw+ep /usr/lib/jvm/java-1.8.0/bin/java
echo  /usr/lib/jvm/java-1.8.0/lib/amd64/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For OpenJDK 11:

sudo setcap cap_net_raw+ep  /usr/lib/jvm/java-11/bin/java
echo  /usr/lib/jvm/java-11/lib/jli | sudo tee /etc/ld.so.conf.d/java-latest.conf
sudo ldconfig

For more information check the documentation about [CAP_NET_RAW](http://man7.org/linux/man-pages/man7/capabilities.7.html).

:::warning
Please keep in mind that the `setcap` command has to be re-executed every time the JDK is upgraded.
:::

#### Using sysctl

:::warning
Using this methodology requires a fairly recent Kernel. For instance, this will work on CentOS/RHEL 8 (and modern versions of any Debian based distribution like Ubuntu). However, in CentOS/RHEL 7 it would work only for IPv4, as support for IPv6 was added in Kernel 3.11.
:::

FILE="/etc/sysctl.d/99-zzz-non-root-icmp.conf"
echo "net.ipv4.ping_group_range=0 429496729" | sudo tee $FILE
sudo sysctl -p $FILE

About CentOS 7 or older. If you attempt to use this, you'll see errors like the following in the logs:

2021-03-13 12:54:04,596 ERROR [Main] o.o.n.i.j.Jni6Pinger: Permission error received while attempting to open ICMP socket. See https://wiki.opennms.org/wiki/ICMP for information on configuring ICMP for non-root.

However, RRD/JDB files for ICMP response times seem to be updated, and ICMP outages are reported; but only for IPv4 interfaces.

This option is more strict than using `setcap`, and if you reduce the scope even more, reduce the group range to cover only the UID of the `opennms` user, as the above enables ICMP for all users in the system.

### Install Minion

Follow the usual procedure to install RPMs on CentOS/RHEL.

For Horizon:

sudo yum install -y -q http://yum.opennms.org/repofiles/opennms-repo-stable-rhel7.noarch.rpm
sudo yum install -y -q jicmp jicmp6
sudo yum install -y -q opennms-minion

For Meridian 2018 and newer:

MERIDIAN_USER="your_username_here"
MERIDIAN_PASSWORD="your_password_here"
MERIDIAN_REPO="2018/stable"

cat <<EOF | sudo tee /etc/yum.repos.d/meridian.repo
[meridian]
name=OpenNMS for Red Hat Enterprise Linux and CentOS 7
baseurl="https://$MERIDIAN_USER:$[e-mail]/packages/$MERIDIAN_REPO/rhel7"
enabled=1
gpgcheck=1
sslverify=false
gpgkey=http://yum.opennms.org/OPENNMS-GPG-KEY
EOF
sudo yum install -y -q jicmp jicmp6
sudo yum install -y -q meridian-minion

#### Move the `init` script

sudo mv /etc/init.d/minion /opt/minion/bin/

:::warning
Due to a [known issue](https://issues.opennms.org/browse/LTS-231), a fixed version of `/etc/init.d/minion` is required when running Meridian 2018.1.5 or older. If this is the case, make sure to grab a copy of this script from the [Horizon RPM](http://yum.opennms.org/stable/common/opennms/opennms-minion-container-23.0.3-1.noarch.rpm). This is not required with latest version.
:::

#### Create a Systemd unit for Minion

The following is optional but recommended:

cat <<EOF | sudo tee /etc/systemd/system/minion.service
[Unit]
Description=OpenNMS Minion server
Requires=network.target network-online.target
After=network.target network-online.target

[Service]
User=minion
LimitNOFILE=204800
LimitNPROC=63409

Type=forking
PIDFile=/opt/minion/karaf.pid

ExecStart=/opt/minion/bin/minion start
ExecStop=/opt/minion/bin/minion stop

[Install]
WantedBy=multi-user.target
EOF
sudo chown minion:minion /etc/systemd/system/minion.service
sudo systemctl daemon-reload
sudo systemctl enable minion

:::info
At this point, a user can login with the `minion` account, and should be able to manage the `minion` application through `systemctl`. All the content of `/opt/minion` is owned by the `minion` user, which makes the whole application fully configurable.
:::

### JVM Settings

Update `/etc/sysconfig/minion` for standard tuning and enable non-root nature.

total_mem_in_mb=$(free -m | awk '/:/ {print $2;exit}')
mem_in_mb=$(expr $total_mem_in_mb / 2)
if [ "$mem_in_mb" -gt "8192" ]; then
mem_in_mb="8192"
fi
half_mem_in_mb=$(expr $mem_in_mb / 2)
java_opts="-XX:+UseG1GC"
sysconfig=/etc/sysconfig/minion
sudo sed -r -i "/export JAVA_MAX_MIM/s/^# //" $sysconfig
sudo sed -i -r "/export JAVA_MAX_MIM/s/=.*/=${half_mem_in_mb}M/" $sysconfig
sudo sed -r -i "/export JAVA_MAX_MEM/s/^# //" $sysconfig
sudo sed -i -r "/export JAVA_MAX_MEM/s/=.*/=${mem_in_mb}M/" $sysconfig
sudo sed -r -i "/export RUNAS/s/^# //" $sysconfig
sudo sed -i -r "/export RUNAS/s/=.*/=minion/" $sysconfig
sudo sed -r -i "/export JAVA_OPTS/s/^# //" $sysconfig
sudo sed -i -r "/export JAVA_OPTS/s/=.*/=\"$java_opts\"/" $sysconfig

:::info
The last entry is to avoid checking ICMP bits at kernel level, which won't work on CentOS/RHEL 7. The `setcap` on tha `java` binary is enough to perform ICMP requests through a non-root user on OS with old kernels.
:::

The following applies only when using Meridian 2018.1.5 or older:

cat <<EOF | sudo tee -a /etc/sysconfig/minion
export PING_REQUIRED=FALSE
EOF

### Credentials

Configure the credentials to access the OpenNMS ReST API and the ActiveMQ broker (if apply).

By default, Minion will use the `admin` user with the `admin` password. The idea is create a user in OpenNMS called `minion`, and assign the `ROLE_MINION` to it. Then, update the credentials:

MINION_USER="minion"
MINION_PASSWD="minion"
sudo /opt/minion/bin/scvcli set opennms.http $MINION_USER $MINION_PASSWD
sudo /opt/minion/bin/scvcli set opennms.broker $MINION_USER $MINION_PASSWD

> **WARNING**: `MINION_USER` and `MINION_PASSWD` must match the user created inside `users.xml` in the OpenNMS server.

### Permissions

If Minion was previously updated, fix the permissions

sudo chown -R minion:minion /opt/minion

## Configuration

:::danger
The following content is not related to the non-root nature.
:::

:::warning
The following contains some general advice about how to configure Minion assuming that Kafka will be used for Sink. On Horizon 23 or newer (and for Meridian 2019+), Kafka is possible for RPC. For older versions, ActiveMQ should be used for RPC, which is what is outlined below as this guide was designed for Meridian 2018.
:::

This guide assumes that a centralized Kafka cluster will be shared across multiple OpenNMS instances, and will be used for the Sink API and for the Kafka Producer feature. On recent versions, as mentioned, it can also be used for the RPC API. because of this, the "Instance ID" must be used not only for Kafka but also for ActiveMQ if applies.

:::info
Make sure that OpenNMS has been configured as described [here](https://hackmd.io/0hICulQpSZm0J6OmnzfMvA?view#Configuration).
:::

1. Make sure that the Instance ID is properly defined on the Minions associated with the backend configured on step 1. Append the following content to `/opt/minion/etc/system.properties`:

org.opennms.instance.id=XXX

On the above example, `XXX` is the chosen name for the instance ID, and that will be the prefix used on *all* the Topic/Queue names in Kafka and ActiveMQ.

:::info
It has to be be same content used for OpenNMS.
:::

2. Make sure to enable the Kafka Sink API for the Minions. Create a file called `/opt/minion/etc/featuresBoot.d/kafka.boot` with the following content:

!opennms-core-ipc-sink-camel
opennms-core-ipc-sink-kafka

And configure the Kafka broker on `/opt/minion/etc/org.opennms.core.ipc.sink.kafka.cfg`, the content should look like this:

bootstrap.servers=kafka1:9092

:::info
Use the same broker list used for OpenNMS. For more information, check [here](https://hackmd.io/0hICulQpSZm0J6OmnzfMvA).
:::

:::warning
Remember to change `kafka1` with the IP or FQDN of one of the Kafka broker. Alternatively, a list of brokers is also accepted; for instance, `kafka1:9092,kafka2:9092`.
:::

3. **For M2019+**. Enable the Kafka RPC API for the Minions.

Create a file called `/opt/minion/etc/featuresBoot.d/kafka.boot` with the following content:

!opennms-core-ipc-rpc-camel
opennms-core-ipc-rpc-kafka

And the bootstrap servers to `/opt/minion/etc/org.opennms.core.ipc.rpc.kafka.cfg`. The content should look like this:

bootstrap.servers=kafka1:9092

:::info
Use the same broker list used for OpenNMS. For more information, check [here](https://hackmd.io/0hICulQpSZm0J6OmnzfMvA).
:::

:::warning
Remember to change `kafka1` with the IP or FQDN of one of the Kafka broker. Alternatively, a list of brokers is also accepted; for instance, `kafka1:9092,kafka2:9092`.
:::

5. **For M2020+**: Enable the single-topic feature for RPC. Add the following content to `/opt/minion/etc/org.opennms.core.ipc.rpc.kafka.cfg` with the following content:

single-topic=true

6. Configure the location, opennms-url and broker-url for Minion on org.opennms.minion.controller.cfg; for example:

location=Apex
id=onms-minion.local
http-url=http://192.168.205.168:8980/opennms
broker-url=failover:tcp://192.168.205.168:61616

:::warning
Remember to change `192.168.205.168` with the IP or FQDN of the OpenNMS server. The above example assumes the usaged of the embedded AMQ within OpenNMS. Alternatively, An external AMQ server can be used. When using Kafka for RPC and Sink, `broker-url` can be omitted (it will be ignored).
:::

7. Configure Trap/Syslog reception on Minion

Add the following to `/opt/minion/etc/org.opennms.netmgt.trapd.cfg`:

trapd.listen.interface=0.0.0.0
trapd.listen.port=1162

# controls how many traps are included in a single message sent to Kafka
trapd.batch.size=10

# limits how many messages are kept in memory if Kafka is unreachable
trapd.queue.size=10000

Add the following to `/opt/minion/etc/org.opennms.netmgt.syslog.cfg`:

syslog.listen.interface=0.0.0.0
syslog.listen.port=1514

# controls how many traps are included in a single message sent to Kafka
syslog.batch.size=10

# limits how many messages are kept in memory if Kafka is unreachable
syslog.queue.size=10000

:::warning
The queue size must be consistent with the message/buffer limits at Kafka Broker level. This is important for Meridian 2018 when using Kafka, as with Horizon 24+ or newer, the messages are split into multiple chunks to avoid hitting Kafka limits.
:::

Speaking about Kafka Limits, the following is required for the Sink API:

echo "max.request.size=5000000" >> /opt/minion/etc/org.opennms.core.ipc.sink.kafka.cfg

For RPC (not available on Meridian 2018):

echo "max.request.size=5000000" >> /opt/minion/etc/org.opennms.core.rpc.sink.kafka.cfg

Minion would have to be restarted after the above changes.

That also requires changes on each Kafka broker and OpenNMS.

At Kafka broker level:

cat <<EOF | tee -a /opt/kafka/conf/server.properties
message.max.bytes=5000000
replica.fetch.max.bytes=5000000
compression.type=producer
EOF

At OpenNMS level, for Sink:

cat <<EOF | tee -a /opt/opennms/etc/opennms.properties.d/kafka.properties
org.opennms.core.ipc.sink.kafka.max.partition.fetch.bytes=5000000
EOF

And for RPC (not available on Meridian 2018):

cat <<EOF | tee -a /opt/opennms/etc/opennms.properties.d/kafka.properties
# RPC Consumer (verify Kafka broker configuration)
org.opennms.core.ipc.rpc.kafka.max.partition.fetch.bytes=5000000
org.opennms.core.ipc.rpc.kafka.auto.offset.reset=latest
# RPC Producer (verify Kafka broker configuration)
org.opennms.core.ipc.rpc.kafka.max.request.size=5000000
EOF

8. Configure internal firewall (CentOS/RHEL with Firewalld is assumed)

Enable firewalld:

sudo systemctl enable firewalld
sudo systemctl start firewalld

Open the UDP ports 162 and 514 on the local firewall, to allow the Minion Server to receive information through them:

sudo firewall-cmd --zone=public --add-port=162/udp
sudo firewall-cmd --zone=public --add-port=162/udp --permanent

sudo firewall-cmd --zone=public --add-port=514/udp
sudo firewall-cmd --zone=public --add-port=514/udp --permanent

Enable port forwarding, in order to forward from UDP 162 to UDP 1162, and from UDP 514 to UDP 1514; to guarantee the Minion application receives the messages.

sudo firewall-cmd --zone="public" --add-forward-port=port=162:proto=udp:toport=1162
sudo firewall-cmd --zone="public" --add-forward-port=port=162:proto=udp:toport=1162 --permanent

sudo firewall-cmd --zone="public" --add-forward-port=port=514:proto=udp:toport=1514
sudo firewall-cmd --zone="public" --add-forward-port=port=514:proto=udp:toport=1514 --permanent

:::warning
Do not associate or constraint the rules against `127.0.0.1` as the official documentation suggests, to be consistent with the `listen.interface` entry used on the configuration files.
:::

To verify the firewall rules:

sudo firewall-cmd --list-all --zone public

Or to check everything:

sudo firewall-cmd  --list-all-zones

9. Start Minion

systemctl start minion