Syslog - HackMD

# Syslog :::success [RFC 5424 - The Syslog Protocol](https://datatracker.ietf.org/doc/html/rfc5424) ::: [TOC] ## What is syslog - A standard for **sending and receiving notification messages** in a particular format from various network devices. - **Monitor network devices and systems** to send out notification messages if there are any issues with functioning. - **Sends out alerts** for pre-notified events and monitors suspicious activity. > - Hardware errors > - Application failures > - Lost contact > - Mis-configuration - Syslog server offers **a central repository for logs** from multiple sources. ### Concept - Syslog daemon **`rsyslog`** - Listens for logs and writes them to a specific location. > The location(s) is defined in the configuration file for the daemon. - Syslog Message Format defined in *RFC5424* - Syslog Protocol - Used for remote logging. - Modern: TCP and TLS - Legacy: UDP ### Components of Syslog Servers - A Syslog listener - The listener gathers and processes syslog data sent over **port 514**. - A database - Syslog servers need databases to store massive amounts of data for quick access. - Management and filtering software - Since there can be enormous amounts of data, it can take excessive amounts of time to find specific log entries. ### Different system log tools 1. **syslog** - Traditional syslog daemon - Only support **UDP** transmission 2. **rsyslog** - An "advanced" version of sysklog - support **TCP** transmission - can discriminate the **log filtering** by program, source, message, pid. - The config file remains the same (you can just copy and paste and it will still be able to run up) 3. **syslog-ng** - Next generation - Everything is object (source, destination, filter, and the very forwarding rule) and the syntax is clear. ## Definitions ### Three conceptual layers - **syslog content** - The management information contained in a syslog message. - **syslog application** - Handles generation, interpretation, routing, and storage of syslog messages. - **syslog transport** - Puts messages on the wire and takes them off the wire. ### Functions - **originator** - **Generates syslog content** to be carried in a message. - **collector** - **Gathers syslog content** for further analysis. - **relay** - **Forwards messages**, accepting messages from originators or other relays and sending them to collectors or other relays. - **transport sender** - Passes syslog messages to a specific transport protocol. - **transport receiver** - Takes syslog messages from a specific transport protocol. ### Severity levels - There are 8 levels of severity defined by the standard | Severity | Keyword | Description | | -------- | -------- | -------- | | Emergency | `emerg` | System is unusable | | Alert | `alert` | Action must be taken immediately | | Critical | `crit` | Critical conditions | | Error | `err` | Error conditions | | Warning | `warning` | Warning conditions | | Notice | `notice` | Normal but significant conditions | | Informational | `info` | Informational messages | | Debug | `debug` | Debug-level messages | ### Basic Principles - The syslog protocol does not provide acknowledgment of message delivery, it is a **pure simplex communications protocol**. - Originators and relays may be configured to send the same message to multiple collectors and relays. - Originator, relay, and collector functionality may reside on the same system. ## Operation ### `journald` - Logs are written in binary, view via command ```bash= journalctl ``` ### `/var/log` - Logs are written under the */var/log* directory by a Syslog daemon(`rsyslog`) ``` phoebe@env_ubuntu1804:~$ sudo ls /var/log alternatives.log dpkg.log.1 syslog.3.gz alternatives.log.1 fail2ban.log syslog.4.gz apport.log fail2ban.log.1 syslog.5.gz apport.log.1 faillog syslog.6.gz apt installer syslog.7.gz auth.log journal tallylog auth.log.1 kern.log unattended-upgrades bootstrap.log kern.log.1 vmware-vmsvc-root.1.log btmp landscape vmware-vmsvc-root.2.log btmp.1 lastlog vmware-vmsvc-root.log cloud-init.log lxd vmware-vmtoolsd-root.log cloud-init-output.log syslog wtmp dist-upgrade syslog.1 wtmp.1 dpkg.log syslog.2.gz ``` #### `tail` - View the last few logs ```bash= sudo tail -f /var/log/syslog ``` > `-f`: watch logs in real time > Path for RedHat based systems: `/var/log/messages` - Can be used to view kernel logs (kern.log), boot logs (boot.log), etc. #### Configuration file - The rules for which logs go where are defined in the Syslog daemon’s configuration file `/etc/rsyslog.conf`. ```bash= sudo nano /etc/rsyslog.conf ``` - For some distros the location rules are defined separately in `/etc/rsyslog.d/50-default.conf` :::spoiler ![](https://i.imgur.com/DQM0ARE.png) ![](https://i.imgur.com/mFohLUN.png) ::: - Facility codes - The `kern.*`, `mail.*`, etc at the start of some lines are facility codes as defined by the Syslog standard. ![](https://i.imgur.com/bzJQtPh.png) > See more details in [wiki page](https://en.wikipedia.org/wiki/Syslog#Facility) - Severity levels - The `*.=info;`,`*.=debug;`, etc means the severity level selected to log. ![](https://i.imgur.com/gAEx9pO.png) > See defination for [severity levels](https://hackmd.io/r9A9nMFvSOKBG3Mzic1ZoQ?both#Severity-levels) ### SOP - Check if `rsyslog` is installed ```bash= rsyslogd -v ``` :::spoiler ![](https://i.imgur.com/TO2Voul.png) ::: #### Edit `rsyslog`’s configuration file - Enable grouping of logs ``` $template FILENAME,"/var/log/%HOSTNAME%/syslog.log" *.* ?FILENAME ``` > Group the logs by creating separate directories for separate client systems using what rsyslog calls `templates`. - Enable TCP ``` # provides TCP syslog reception module(load="imtcp") input(type="imtcp" port="514") ``` - Enable UDP ``` # provides UDP syslog reception module(load="imudp") input(type="imudp" port="514") ``` #### Configure the firewall to open the ==port 514== - View opened ports on the host firewall ```bash= sudo ufw status ``` - Enable firewall ```bash= sudo ufw enable ``` - Add port 514 ```bash= sudo ufw allow 514/tcp ``` :::warning See more about [Ubuntu `ufw` howto](https://www.cyberciti.biz/faq/how-to-open-firewall-port-on-ubuntu-linux-12-04-14-04-lts/) ::: #### Restart `rsyslog` ```bash= sudo systemctl restart rsyslog sudo systemctl enable rsyslog ``` #### List all the open ports ```bash= sudo netstat -pnlt ``` :::spoiler ![](https://i.imgur.com/TOjSiaM.png) ::: > `rsyslog` is listening on port 514 #### Edit client’s `/etc/rsyslog.conf` file - Add ``` *.* @@10.21.24.96:514 ``` > `*.*` tells rsyslog to forward all logs. > `@@` means it's a TCP connection. > `10.21.24.96` is Syslog Server's IP address. - Remember to restart client's `rsyslog`, also check if client's port 514 is opened. #### Test if Server can receive logs from client - type command on client ```bash= logger "Here is the message from client" ``` - Use `tail -f` to check if logs were sent to server ``` sudo tail -f /var/log/attacker1/syslog.log ``` > Because of the configuration which set template in `rsyslog`, under server's `/var/log/` there would create a directory which named after client's hostname. ## [Message Format](https://datatracker.ietf.org/doc/html/rfc5424#section-6) ``` SYSLOG-MSG = HEADER SP STRUCTURED-DATA [SP MSG] HEADER = PRI VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID PRI = "<" PRIVAL ">" PRIVAL = 1*3DIGIT ; range 0 .. 191 VERSION = NONZERO-DIGIT 0*2DIGIT HOSTNAME = NILVALUE / 1*255PRINTUSASCII APP-NAME = NILVALUE / 1*48PRINTUSASCII PROCID = NILVALUE / 1*128PRINTUSASCII MSGID = NILVALUE / 1*32PRINTUSASCII TIMESTAMP = NILVALUE / FULL-DATE "T" FULL-TIME FULL-DATE = DATE-FULLYEAR "-" DATE-MONTH "-" DATE-MDAY DATE-FULLYEAR = 4DIGIT DATE-MONTH = 2DIGIT ; 01-12 DATE-MDAY = 2DIGIT ; 01-28, 01-29, 01-30, 01-31 based on ; month/year FULL-TIME = PARTIAL-TIME TIME-OFFSET PARTIAL-TIME = TIME-HOUR ":" TIME-MINUTE ":" TIME-SECOND [TIME-SECFRAC] TIME-HOUR = 2DIGIT ; 00-23 TIME-MINUTE = 2DIGIT ; 00-59 TIME-SECOND = 2DIGIT ; 00-59 TIME-SECFRAC = "." 1*6DIGIT TIME-OFFSET = "Z" / TIME-NUMOFFSET TIME-NUMOFFSET = ("+" / "-") TIME-HOUR ":" TIME-MINUTE STRUCTURED-DATA = NILVALUE / 1*SD-ELEMENT SD-ELEMENT = "[" SD-ID *(SP SD-PARAM) "]" SD-PARAM = PARAM-NAME "=" %d34 PARAM-VALUE %d34 SD-ID = SD-NAME PARAM-NAME = SD-NAME PARAM-VALUE = UTF-8-STRING ; characters '"', '\' and ; ']' MUST be escaped. SD-NAME = 1*32PRINTUSASCII ; except '=', SP, ']', %d34 (") MSG = MSG-ANY / MSG-UTF8 MSG-ANY = *OCTET ; not starting with BOM MSG-UTF8 = BOM UTF-8-STRING BOM = %xEF.BB.BF UTF-8-STRING = *OCTET ; UTF-8 string as specified ; in RFC 3629 OCTET = %d00-255 SP = %d32 PRINTUSASCII = %d33-126 NONZERO-DIGIT = %d49-57 DIGIT = %d48 / NONZERO-DIGIT NILVALUE = "-" ``` ### Syslog message size >= 480 octets - There is **no upper limit** per se. Each transport mapping defines the minimum maximum required message length support, and the minimum maximum MUST be **at least 480 octets in length**. > - Any transport receiver MUST be able to accept messages of up to and including 480 octets in length. > - All transport receiver implementations SHOULD be able to accept messages of up to and including 2048 octets in length. > - Transport receivers MAY receive messages larger than 2048 octets in length. - If a transport receiver receives a message with a length larger than it supports, the transport receiver SHOULD **truncate the payload**. Or, it MAY **discard the message**. ### Header - The character set used in the HEADER MUST be **seven-bit ASCII in an eight-bit field**. ### PRI - Starts with `<` & followed by a `>` - The PRI part MUST have three, four, or five characters and will be bound with angle brackets as the first and last characters. - The number contained within these angle brackets is known as the **Priority value** (PRIVAL) and represents both the **Facility** and **Severity**. #### Facility - Facility values MUST be in the range of 0 to 23 inclusive. ``` Numerical Facility Code 0 kernel messages 1 user-level messages 2 mail system 3 system daemons 4 security/authorization messages 5 messages generated internally by syslogd 6 line printer subsystem 7 network news subsystem 8 UUCP subsystem 9 clock daemon 10 security/authorization messages 11 FTP daemon 12 NTP subsystem 13 log audit 14 log alert 15 clock daemon (note 2) 16 local use 0 (local0) 17 local use 1 (local1) 18 local use 2 (local2) 19 local use 3 (local3) 20 local use 4 (local4) 21 local use 5 (local5) 22 local use 6 (local6) 23 local use 7 (local7) ``` ### Severity - Each message Priority has a decimal Severity level indicator. - Severity values MUST be in the range of 0 to 7 inclusive. ``` Numerical Severity Code 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages ``` ### Timestamp - Restrictions - The "T" and "Z" characters in this syntax MUST be upper case. - Usage of the "T" character is REQUIRED. - Leap seconds MUST NOT be used. ### Hostname - Identifies the machine that originally sent the syslog message. - SHOULD contain the hostname and the domain name of the originator in the Fully Qualified Domain Name (FQDN) format. - > The order of preference: **FQDN > Static IP address > hostname > Dynamic IP address > the NILVALUE** ### APP-NAME - SHOULD identify the device or application that originated the message. - It is intended for **filtering messages** on a relay or collector. - This field MAY be operator-assigned. ### PROCID - A **value** that is included in the message, having **no interoperable meaning**, except that a change in the value indicates there has been a discontinuity in Syslog reporting. - The field does not have any specific syntax or semantics. - The value is implementation-dependent and/or operator-assigned. - Often used to provide the process name or process ID associated with a Syslog system. - On an embedded system without any operating system process ID, PROCID might be a reboot ID. > PROCID can enable log analyzers to detect discontinuities in syslog reporting by detecting a change in the syslog process ID. > Also, it can be used to identify which messages belong to a group of messages (same service, same ID). ### MSGID - SHOULD identify **the type of message**. - Messages with the same MSGID should reflect events of the same semantics. - The MSGID itself is a string without further semantics. It is intended for **filtering messages** on a relay or collector. - This field MAY be operator-assigned. ### STRUCTURED-DATA - Provides a mechanism to express information in a well-defined, easily parseable, and interpretable data format. - STRUCTURED-DATA can contain zero, one, or multiple structured data elements, which are referred to as "SD-ELEMENT" in this document. - The character set used in STRUCTURED-DATA MUST be seven-bit ASCII in an eight-bit field. > An exception is the PARAM-VALUE field, in which UTF-8 encoding MUST be used. - A collector MAY ignore malformed STRUCTURED-DATA elements. - A relay MUST forward malformed STRUCTURED-DATA without any alteration. ### SD-ELEMENT - An SD-ELEMENT consists of a name and parameter name-value pairs. - The name is referred to as SD-ID. - The name-value pairs are referred to as "SD-PARAM". ### SD-ID - SD-IDs are case-sensitive and uniquely identify the type and purpose of the SD-ELEMENT. - The same SD-ID MUST NOT exist more than once in a message. - Two formats for SD-ID names ### SD-PARAM - Each SD-PARAM consists of a name, referred to as **PARAM-NAME**, and a value, referred to as **PARAM-VALUE**. - Case-sensitive - IANA controls all PARAM-NAMEs, with the exception of those in SD-IDs whose names contain an at-sign(`@`). - The PARAM-NAME scope is within a specific SD-ID. > Thus, equally named PARAM-NAME values contained in two different SD-IDs are not the same. - The PARAM-VALUE field MUST be encoded using UTF-8. - It MUST NOT fail if control characters are present in PARAM-VALUE. - The syslog application MAY modify messages containing control characters. - An SD-PARAM MAY be repeated multiple times inside an SD-ELEMENT. :::info Once SD-IDs and PARAM-NAMEs are defined, the syntax and semantics of these objects MUST NOT be altered. Should a change to an existing object be desired, a new SD-ID or PARAM-NAME MUST be created and the old one remains unchanged. ::: - OPTIONAL PARAM-NAMEs MAY be added to an existing SD-ID. ### MSG - Contains a free-form message that provides information about the event. - The character set used in MSG SHOULD be UNICODE, encoded using UTF-8. :::spoiler OPTIONAL Structured Data IDs [details](https://datatracker.ietf.org/doc/html/rfc5424#section-7). ::: ## Reference ### Official - [RFC 5424 - The Syslog Protocol](https://datatracker.ietf.org/doc/html/rfc5424) ### Article - [What is Syslog in Linux? A Step-by-Step Guide to Set up Remote System Logging](https://www.linuxfordevices.com/tutorials/remote-syslog-in-linux) - [Syslog : The Complete System Administrator Guide](https://devconnected.com/syslog-the-complete-system-administrator-guide/) - [Syslog Tutorial: How It Works, Examples, Best Practices, and More](https://stackify.com/syslog-101/) - [Set up a Central Log Server with syslog-ng daemon on FreeBSD](https://blog.matrixpost.net/set-up-a-central-monitoring-server-with-syslog-ng-on-freebsd/)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.