# Syslog
:::success
[RFC 5424 - The Syslog Protocol](https://datatracker.ietf.org/doc/html/rfc5424)
:::
[TOC]

## What is syslog
- A standard for **sending and receiving notification messages** in a particular format from various network devices.
- **Monitor network devices and systems** to send out notification messages if there are any issues with functioning.
- **Sends out alerts** for pre-notified events and monitors suspicious activity.
> - Hardware errors
> - Application failures
> - Lost contact
> - Mis-configuration
- Syslog server offers **a central repository for logs** from multiple sources.
### Concept
- Syslog daemon **`rsyslog`**
- Listens for logs and writes them to a specific location.
> The location(s) is defined in the configuration file for the daemon.
- Syslog Message Format defined in *RFC5424*
- Syslog Protocol
- Used for remote logging.
- Modern: TCP and TLS
- Legacy: UDP

### Components of Syslog Servers
- A Syslog listener
- The listener gathers and processes syslog data sent over **port 514**.
- A database
- Syslog servers need databases to store massive amounts of data for quick access.
- Management and filtering software
- Since there can be enormous amounts of data, it can take excessive amounts of time to find specific log entries.
### Different system log tools
1. **syslog**
- Traditional syslog daemon
- Only support **UDP** transmission
2. **rsyslog**
- An "advanced" version of sysklog
- support **TCP** transmission
- can discriminate the **log filtering** by program, source, message, pid.
- The config file remains the same (you can just copy and paste and it will still be able to run up)
3. **syslog-ng**
- Next generation
- Everything is object (source, destination, filter, and the very forwarding rule) and the syntax is clear.

## Definitions

### Three conceptual layers
- **syslog content**
- The management information contained in a syslog message.
- **syslog application**
- Handles generation, interpretation, routing, and storage of syslog messages.
- **syslog transport**
- Puts messages on the wire and takes them off the wire.

### Functions
- **originator**
- **Generates syslog content** to be carried in a message.
- **collector**
- **Gathers syslog content** for further analysis.
- **relay**
- **Forwards messages**, accepting messages from originators or other relays and sending them to collectors or other relays.
- **transport sender**
- Passes syslog messages to a specific transport protocol.
- **transport receiver**
- Takes syslog messages from a specific transport protocol.

### Severity levels
- There are 8 levels of severity defined by the standard

| Severity | Keyword | Description |
| -------- | -------- | -------- |
| Emergency | `emerg` | System is unusable |
| Alert | `alert` | Action must be taken immediately |
| Critical | `crit` | Critical conditions |
| Error | `err` | Error conditions |
| Warning | `warning` | Warning conditions |
| Notice | `notice` | Normal but significant conditions |
| Informational | `info` | Informational messages |
| Debug | `debug` | Debug-level messages |

### Basic Principles
- The syslog protocol does not provide acknowledgment of message delivery, it is a **pure simplex communications protocol**.
- Originators and relays may be configured to send the same message to multiple collectors and relays.
- Originator, relay, and collector functionality may reside on the same system.

## Operation

### `journald`
- Logs are written in binary, view via command

journalctl

### `/var/log`
- Logs are written under the */var/log* directory by a Syslog daemon(`rsyslog`)

phoebe@env_ubuntu1804:~$ sudo ls /var/log
alternatives.log       dpkg.log.1      syslog.3.gz
alternatives.log.1     fail2ban.log    syslog.4.gz
apport.log             fail2ban.log.1  syslog.5.gz
apport.log.1           faillog         syslog.6.gz
apt                    installer       syslog.7.gz
auth.log               journal         tallylog
auth.log.1             kern.log        unattended-upgrades
bootstrap.log          kern.log.1      vmware-vmsvc-root.1.log
btmp                   landscape       vmware-vmsvc-root.2.log
btmp.1                 lastlog         vmware-vmsvc-root.log
cloud-init.log         lxd             vmware-vmtoolsd-root.log
cloud-init-output.log  syslog          wtmp
dist-upgrade           syslog.1        wtmp.1
dpkg.log               syslog.2.gz

#### `tail`
- View the last few logs

sudo tail -f /var/log/syslog

> `-f`: watch logs in real time
> Path for RedHat based systems: `/var/log/messages`
- Can be used to view kernel logs (kern.log), boot logs (boot.log), etc.

#### Configuration file
- The rules for which logs go where are defined in the Syslog daemon’s configuration file `/etc/rsyslog.conf`.

sudo nano /etc/rsyslog.conf

- For some distros the location rules are defined separately in `/etc/rsyslog.d/50-default.conf`
:::spoiler
:::

- Facility codes
- The `kern.*`, `mail.*`, etc at the start of some lines are facility codes as defined by the Syslog standard.
> See more details in [wiki page](https://en.wikipedia.org/wiki/Syslog#Facility)

- Severity levels
- The `*.=info;`,`*.=debug;`, etc means the severity level selected to log.
> See defination for [severity levels](https://hackmd.io/r9A9nMFvSOKBG3Mzic1ZoQ?both#Severity-levels)

### SOP
- Check if `rsyslog` is installed

rsyslogd -v

:::spoiler
:::

#### Edit `rsyslog`’s configuration file
- Enable grouping of logs

	$template FILENAME,"/var/log/%HOSTNAME%/syslog.log"
	*.* ?FILENAME

> Group the logs by creating separate directories for separate client systems using what rsyslog calls `templates`.
- Enable TCP

	# provides TCP syslog reception
	module(load="imtcp")
	input(type="imtcp" port="514")

- Enable UDP

	# provides UDP syslog reception
	module(load="imudp")
	input(type="imudp" port="514")

#### Configure the firewall to open the ==port 514==
- View opened ports on the host firewall

sudo ufw status

- Enable firewall

sudo ufw enable

- Add port 514

sudo ufw allow 514/tcp

:::warning
See more about [Ubuntu `ufw` howto](https://www.cyberciti.biz/faq/how-to-open-firewall-port-on-ubuntu-linux-12-04-14-04-lts/)
:::

#### Restart `rsyslog`

sudo systemctl restart rsyslog
sudo systemctl enable rsyslog

#### List all the open ports

sudo netstat -pnlt

:::spoiler
:::
> `rsyslog` is listening on port 514

#### Edit client’s `/etc/rsyslog.conf` file
- Add

	*.* @@10.21.24.96:514

> `*.*` tells rsyslog to forward all logs.
> `@@` means it's a TCP connection.
> `10.21.24.96` is Syslog Server's IP address.
- Remember to restart client's `rsyslog`, also check if client's port 514 is opened.

#### Test if Server can receive logs from client
- type command on client

logger "Here is the message from client"

- Use `tail -f` to check if logs were sent to server

sudo tail -f /var/log/attacker1/syslog.log

> Because of the configuration which set template in `rsyslog`, under server's `/var/log/` there would create a directory which named after client's hostname.

## [Message Format](https://datatracker.ietf.org/doc/html/rfc5424#section-6)

      SYSLOG-MSG      = HEADER SP STRUCTURED-DATA [SP MSG]

      HEADER          = PRI VERSION SP TIMESTAMP SP HOSTNAME
                        SP APP-NAME SP PROCID SP MSGID
      PRI             = "<" PRIVAL ">"
      PRIVAL          = 1*3DIGIT ; range 0 .. 191
      VERSION         = NONZERO-DIGIT 0*2DIGIT
      HOSTNAME        = NILVALUE / 1*255PRINTUSASCII

      APP-NAME        = NILVALUE / 1*48PRINTUSASCII
      PROCID          = NILVALUE / 1*128PRINTUSASCII
      MSGID           = NILVALUE / 1*32PRINTUSASCII

      TIMESTAMP       = NILVALUE / FULL-DATE "T" FULL-TIME
      FULL-DATE       = DATE-FULLYEAR "-" DATE-MONTH "-" DATE-MDAY
      DATE-FULLYEAR   = 4DIGIT
      DATE-MONTH      = 2DIGIT  ; 01-12
      DATE-MDAY       = 2DIGIT  ; 01-28, 01-29, 01-30, 01-31 based on
                                ; month/year
      FULL-TIME       = PARTIAL-TIME TIME-OFFSET
      PARTIAL-TIME    = TIME-HOUR ":" TIME-MINUTE ":" TIME-SECOND
                        [TIME-SECFRAC]
      TIME-HOUR       = 2DIGIT  ; 00-23
      TIME-MINUTE     = 2DIGIT  ; 00-59
      TIME-SECOND     = 2DIGIT  ; 00-59
      TIME-SECFRAC    = "." 1*6DIGIT
      TIME-OFFSET     = "Z" / TIME-NUMOFFSET
      TIME-NUMOFFSET  = ("+" / "-") TIME-HOUR ":" TIME-MINUTE


      STRUCTURED-DATA = NILVALUE / 1*SD-ELEMENT
      SD-ELEMENT      = "[" SD-ID *(SP SD-PARAM) "]"
      SD-PARAM        = PARAM-NAME "=" %d34 PARAM-VALUE %d34
      SD-ID           = SD-NAME
      PARAM-NAME      = SD-NAME
      PARAM-VALUE     = UTF-8-STRING ; characters '"', '\' and
                                     ; ']' MUST be escaped.
      SD-NAME         = 1*32PRINTUSASCII
                        ; except '=', SP, ']', %d34 (")

      MSG             = MSG-ANY / MSG-UTF8
      MSG-ANY         = *OCTET ; not starting with BOM
      MSG-UTF8        = BOM UTF-8-STRING
      BOM             = %xEF.BB.BF
      UTF-8-STRING    = *OCTET ; UTF-8 string as specified
                        ; in RFC 3629

      OCTET           = %d00-255
      SP              = %d32
      PRINTUSASCII    = %d33-126
      NONZERO-DIGIT   = %d49-57
      DIGIT           = %d48 / NONZERO-DIGIT
      NILVALUE        = "-"

### Syslog message size >= 480 octets
- There is **no upper limit** per se. Each transport mapping defines the minimum maximum required message length support, and the minimum maximum MUST be **at least 480 octets in length**.
> - Any transport receiver MUST be able to accept messages of up to and including 480 octets in length.
> - All transport receiver implementations SHOULD be able to accept messages of up to and including 2048 octets in length.
> - Transport receivers MAY receive messages larger than 2048 octets in length.
- If a transport receiver receives a message with a length larger than it supports, the transport receiver SHOULD **truncate the payload**. Or, it MAY **discard the message**.

### Header
- The character set used in the HEADER MUST be **seven-bit ASCII in an eight-bit field**.

### PRI
- Starts with `<` & followed by a `>`
- The PRI part MUST have three, four, or five characters and will be bound with angle brackets as the first and last characters.
- The number contained within these angle brackets is known as the **Priority value** (PRIVAL) and represents both the **Facility** and **Severity**.
#### Facility
- Facility values MUST be in the range of 0 to 23 inclusive.

    Numerical             Facility
    Code

        0             kernel messages
        1             user-level messages
        2             mail system
        3             system daemons
        4             security/authorization messages
        5             messages generated internally by syslogd
        6             line printer subsystem
        7             network news subsystem
        8             UUCP subsystem
        9             clock daemon
        10             security/authorization messages
        11             FTP daemon
        12             NTP subsystem
        13             log audit
        14             log alert
        15             clock daemon (note 2)
        16             local use 0  (local0)
        17             local use 1  (local1)
        18             local use 2  (local2)
        19             local use 3  (local3)
        20             local use 4  (local4)
        21             local use 5  (local5)
        22             local use 6  (local6)
        23             local use 7  (local7)

### Severity
- Each message Priority has a decimal Severity level indicator.
- Severity values MUST be in the range of 0 to 7 inclusive.

    Numerical		Severity
    Code

	0       	Emergency: system is unusable
	1       	Alert: action must be taken immediately
	2       	Critical: critical conditions
	3       	Error: error conditions
	4       	Warning: warning conditions
	5       	Notice: normal but significant condition
	6       	Informational: informational messages
	7       	Debug: debug-level messages

### Timestamp
- Restrictions
- The "T" and "Z" characters in this syntax MUST be upper case.
- Usage of the "T" character is REQUIRED.
- Leap seconds MUST NOT be used.

### Hostname
- Identifies the machine that originally sent the syslog message.
- SHOULD contain the hostname and the domain name of the originator in the Fully Qualified Domain Name (FQDN) format.
- > The order of preference: **FQDN > Static IP address > hostname > Dynamic IP address > the NILVALUE**

### APP-NAME
- SHOULD identify the device or application that originated the message.
- It is intended for **filtering messages** on a relay or collector.
- This field MAY be operator-assigned.

### PROCID
- A **value** that is included in the message, having **no interoperable meaning**, except that a change in the value indicates there has been a discontinuity in Syslog reporting.
- The field does not have any specific syntax or semantics.
- The value is implementation-dependent and/or operator-assigned.
- Often used to provide the process name or process ID associated with a Syslog system.
- On an embedded system without any operating system process ID, PROCID might be a reboot ID.
> PROCID can enable log analyzers to detect discontinuities in syslog reporting by detecting a change in the syslog process ID.
> Also, it can be used to identify which messages belong to a group of messages (same service, same ID).

### MSGID
- SHOULD identify **the type of message**.
- Messages with the same MSGID should reflect events of the same semantics.
- The MSGID itself is a string without further semantics. It is intended for **filtering messages** on a relay or collector.
- This field MAY be operator-assigned.

### STRUCTURED-DATA
- Provides a mechanism to express information in a well-defined, easily parseable, and interpretable data format.
- STRUCTURED-DATA can contain zero, one, or multiple structured data elements, which are referred to as "SD-ELEMENT" in this document.
- The character set used in STRUCTURED-DATA MUST be seven-bit ASCII in an eight-bit field.
> An exception is the PARAM-VALUE field, in which UTF-8 encoding MUST be used.
- A collector MAY ignore malformed STRUCTURED-DATA elements.
- A relay MUST forward malformed STRUCTURED-DATA without any alteration.
### SD-ELEMENT
- An SD-ELEMENT consists of a name and parameter name-value pairs.
- The name is referred to as SD-ID.
- The name-value pairs are referred to as "SD-PARAM".
### SD-ID
- SD-IDs are case-sensitive and uniquely identify the type and purpose of the SD-ELEMENT.
- The same SD-ID MUST NOT exist more than once in a message.
- Two formats for SD-ID names

### SD-PARAM
- Each SD-PARAM consists of a name, referred to as **PARAM-NAME**, and a value, referred to as **PARAM-VALUE**.
- Case-sensitive
- IANA controls all PARAM-NAMEs, with the exception of those in SD-IDs whose names contain an at-sign(`@`).
- The PARAM-NAME scope is within a specific SD-ID.
> Thus, equally named PARAM-NAME values contained in two different SD-IDs are not the same.
- The PARAM-VALUE field MUST be encoded using UTF-8.
- It MUST NOT fail if control characters are present in PARAM-VALUE.
- The syslog application MAY modify messages containing control characters.
- An SD-PARAM MAY be repeated multiple times inside an SD-ELEMENT.
:::info
Once SD-IDs and PARAM-NAMEs are defined, the syntax and semantics of these objects MUST NOT be altered. Should a change to an existing object be desired, a new SD-ID or PARAM-NAME MUST be created and the old one remains unchanged.
:::
- OPTIONAL PARAM-NAMEs MAY be added to an existing SD-ID.

### MSG
- Contains a free-form message that provides information about the event.
- The character set used in MSG SHOULD be UNICODE, encoded using UTF-8.

:::spoiler
OPTIONAL Structured Data IDs [details](https://datatracker.ietf.org/doc/html/rfc5424#section-7).
:::

## Reference
### Official
- [RFC 5424 - The Syslog Protocol](https://datatracker.ietf.org/doc/html/rfc5424)
### Article
- [What is Syslog in Linux? A Step-by-Step Guide to Set up Remote System Logging](https://www.linuxfordevices.com/tutorials/remote-syslog-in-linux)
- [Syslog : The Complete System Administrator Guide](https://devconnected.com/syslog-the-complete-system-administrator-guide/)
- [Syslog Tutorial: How It Works, Examples, Best Practices, and More](https://stackify.com/syslog-101/)
- [Set up a Central Log Server with syslog-ng daemon on FreeBSD](https://blog.matrixpost.net/set-up-a-central-monitoring-server-with-syslog-ng-on-freebsd/)