Log all traffic in Ubuntu

tags: NTUToolmenLab

Add logger in iptable

about iptable
https://www.hostinger.com/tutorials/iptables-tutorial
https://help.ubuntu.com/community/IptablesHowTo

show it

sudo iptables --line-numbers -L
sudo iptables --line-numbers -L OUTPUT

Add logger

iptables -A OUTPUT -j LOG --log-prefix 'iptable log: ' --log-level 7

or more formally

iptables -N LOGGER
iptables -A LOGGER -j LOG --log-prefix 'iptable log: ' --log-level 7
iptables -A OUTPUT -j LOGGER

You should add at the top of the chain to avoid skip it by another chain before.
iptables -I OUTPUT -j LOGGER
iptables -I INPUT -j LOGGER

If you want to monitor in docker
iptables -I FORWARD -j LOGGER

delete it if needed
iptables -D OUTPUT 1

Reference

• https://ubuntuforums.org/showthread.php?t=1158091

Separate from syslog

The concept of syslog
https://www.the-art-of-web.com/system/rsyslog-config/

For ubuntu 1804
cat /var/log/syslog or cat /var/log/kern.log to see the log

method 1

Filter the log of iptable logger to specific file

edit
/etc/rsyslog.d/iptable_logger.conf

:msg,contains,"iptable log: " /var/log/iptable.log

method 2(Recommanded)

Don't want to output in kern.log and syslog

Add below code in the begining of /etc/rsyslog.d/50-default.conf

:msg,startswith," iptable log: " /var/log/iptable.log
:msg,startswith," iptable log: " stop

Better format

/etc/rsyslog.conf

module(load="imklog" ParseKernelTimestamp="on" KeepKernelTimestamp="off")
# $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$template mycleanformat,"%TIMESTAMP:::date-rfc3339% %syslogtag% %syslogseverity-text% %msg%\n"
$ActionFileDefaultTemplate mycleanformat

reload

and finally restart
systemctl restart syslog

Reference

• https://askubuntu.com/questions/348439/where-can-i-find-the-iptables-log-file-and-how-can-i-change-its-location
• https://www.rsyslog.com/doc/v8-stable/configuration/filters.html
• https://www.rsyslog.com/doc/v8-stable/rainerscript/expressions.html
• https://unix.stackexchange.com/questions/302972/how-to-stop-rsyslog-output-timestamp
• https://linux.die.net/man/5/rsyslog.conf
• https://www.rsyslog.com/doc/v8-stable/configuration/modules/imklog.html
• https://stackoverflow.com/questions/22755005/rsyslog-property-based-filtering-not-working
• https://rsyslog-5-8-6-doc.neocities.org/rsyslog_conf_templates.html

Make it smaller

Using logrotated
/etc/logrotate.d/rsyslog

/var/log/iptable.log
{
    rotate 99
    size 500M
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Because logrotate run everyday not every second by default,
create a crontab to run it.
/etc/cron.d/iptable-log-rotate

*/10  *  *  *  *   root /usr/sbin/logrotate /etc/logrotate.conf

sudo systemctl restart cron

In my mechine, the 100MB log file can be compressed into about 5MB

And same as systemd-journal

edit /etc/systemd/journald.conf to set
SystemMaxUse=1G

sudo systemctl restart systemd-journald

or stop all of journal
sudo systemctl stop systemd-journald*

Reference:

• https://www.digitalocean.com/community/tutorials/how-to-manage-logfiles-with-logrotate-on-ubuntu-16-04
• https://linux.die.net/man/8/logrotate
• https://www.digitalocean.com/community/tutorials/how-to-manage-logfiles-with-logrotate-on-ubuntu-16-04
• https://unix.stackexchange.com/questions/139513/how-to-clear-journalctl

With calico(Brute force)

Force to write it because calico checks its iptable chain every 60 seconds

while true; do
    if ! iptables -L cali-FORWARD | grep -q LOGGER; then
        iptables -I  cali-FORWARD -j LOGGER;
        echo `date` "ADD"
    fi
    sleep 0.1;
done

And set calico checking seconds more longger in calico.yml

- name: FELIX_IPTABLESPOSTWRITECHECKINTERVALSECS
  value: "10"

Reference

• https://docs.projectcalico.org/v3.7/reference/felix/configuration

CODE

Overall code

	# backup
	sudo cp /etc/rsyslog.d/50-default.conf /etc/rsyslog.d/50-default.conf.bak
	sudo cp /etc/rsyslog.conf /etc/rsyslog.conf.bak
	sudo cp /etc/logrotate.d/rsyslog /opt/rsyslog.bak
	# iptable filter
	sudo sed -i '/iptable log/d' /etc/rsyslog.d/50-default.conf
	sudo sed -i '/Logging iptable/d' /etc/rsyslog.d/50-default.conf
	cat << EOF | cat - /etc/rsyslog.d/50-default.conf > tmp && sudo mv tmp /etc/rsyslog.d/50-default.conf
	# Logging iptable in other logfile
	:msg,startswith," iptable log: " /var/log/iptable.log
	:msg,startswith," iptable log: " stop
	EOF
	# syslog format
	sudo sed -i '/template mycleanformat/d' /etc/rsyslog.conf
	sudo sed -i 's/.*imklog.*/module(load="imklog" ParseKernelTimestamp="on" KeepKernelTimestamp="off")/g' /etc/rsyslog.conf
	sudo sed -i '1 i\$template mycleanformat,"%TIMESTAMP:::date-rfc3339% %syslogtag% %syslogseverity-text% %msg%\\n"' /etc/rsyslog.conf
	sudo sed -i 's/ActionFileDefaultTemplate.*/ActionFileDefaultTemplate mycleanformat/g' /etc/rsyslog.conf
	# log
	echo "*/10 * * * * root /usr/sbin/logrotate /etc/logrotate.conf" > tmp | sudo mv tmp /etc/cron.d/iptable-log-rotate
	cat /etc/logrotate.d/rsyslog | tr '\n' '\r' | sed -e 's~/var/log/iptable.log.*{.*}~~g' | tr '\r' '\n' > tmp
	cat << EOF >> tmp
	/var/log/iptable.log
	{
	rotate 99
	size 500M
	missingok
	notifempty
	delaycompress
	compress
	postrotate
	/usr/lib/rsyslog/rsyslog-rotate
	endscript
	}
	EOF
	sudo mv tmp /etc/logrotate.d/rsyslog
	# set permission
	sudo chmod 644 /etc/logrotate.d/rsyslog
	sudo chown root:root /etc/logrotate.d/rsyslog
	sudo chmod 644 /etc/cron.d/iptable-log-rotate
	sudo chown root:root /etc/cron.d/iptable-log-rotate
	sudo chmod 644 /etc/rsyslog.d/50-default.conf
	sudo chown root:root /etc/rsyslog.d/50-default.conf
	# restart
	sudo systemctl stop systemd-journald*
	sudo systemctl restart cron
	sudo systemctl restart syslog
	# Force to write iptable
	cat << EOF > tmp
	#! /bin/bash
	# create chain
	iptables -N LOGGER
	iptables -F LOGGER
	iptables -A LOGGER -j LOG --log-prefix 'iptable log: ' --log-level 7
	# force create rule
	while true; do
	if ! iptables -L FORWARD --line-numbers | grep -q "1\\s*LOGGER"; then
	iptables -D FORWARD \$(iptables -L FORWARD --line-numbers | grep LOGGER | awk '{ print \$1}')
	iptables -I FORWARD -j LOGGER;
	echo \`date\` "ADD FORWARD"
	fi
	if ! iptables -L INPUT --line-numbers | grep -q "1\\s*LOGGER"; then
	iptables -D INPUT \$(iptables -L INPUT --line-numbers | grep LOGGER | awk '{ print \$1}')
	iptables -I INPUT -j LOGGER;
	echo \`date\` "ADD INPUT"
	fi
	if ! iptables -L OUTPUT --line-numbers | grep -q "1\\s*LOGGER"; then
	iptables -D OUTPUT \$(iptables -L OUTPUT --line-numbers | grep LOGGER | awk '{ print \$1}')
	iptables -I OUTPUT -j LOGGER;
	echo \`date\` "ADD OUTPUT"
	fi
	sleep 0.1;
	done
	EOF
	chmod +x tmp
	sudo mv tmp /opt/change_iptable.sh
	# start script when reboot
	sudo crontab -l > tmp
	sudo sed -i '/change_iptable.sh/d' tmp
	echo "@reboot /opt/change_iptable.sh &" >> tmp
	sudo crontab tmp
	rm tmp

view raw montior_traffic.sh hosted with ❤ by GitHub

TODO

• Monitor in kubernete with calico-cni in more formal way.