Syslog demuxer for detailed iptables logging

[ml70]

Fed up with the misery of logging iptables to syslog I made a log demuxer which makes it possible to demux the syslog feed into any number of separate log files.
All that is needed is to change syslog to output to a named pipe, and an awk script which does the demuxing.

The purpose is to demux all iptables messages out of the syslog into other log files, but can be used for any regexp matched messages.
It's still a work in progress but runs without errors, this far.
Runs on vanilla Merlin/forks without Opt/Entware installed, my development router is an AC66 (mips) with John's fork of Asuswrt-Merlin.

This is not a ready copy paste install, you may need to adapt it to your configuration, prerequisites are

mkdir /jffs/fifo
# usb key or hdd, here labeled "usb"
mkdir /tmp/mnt/usb/log
# any number of log files, these are used in the example
>/tmp/mnt/usb/log/ipt-icmp.log
>/tmp/mnt/usb/log/ipt-other.log
>/tmp/mnt/usb/log/syslog.log
# you may want to link syslog to this syslog file
# so it stays viewable in web interface
#  ln -fns /tmp/mnt/usb/log/syslog.log /tmp/syslog.log

/jffs/scripts/syslog-fifo.sh for starting the services, post-mount is a good place to start this after the usb/hdd has been mounted

#!/bin/sh
# /jffs/scripts/syslog-fifo.sh
rm -f /jffs/fifo/syslogfifo
killall -9 logdemux.sh
killall -9 syslogd
# create the named pipe
mknod /jffs/fifo/syslogfifo p
chmod 765 /jffs/fifo/syslogfifo
# to know nohup.out location
cd /jffs/scripts
# to keep the named pipe open
nohup sleep 4294967295 > /jffs/fifo/syslogfifo 2> /jffs/fifo/syslogfifo.sleep.errlog &
# start the log demuxer
nohup /jffs/fifo/logdemux.sh 2> /jffs/fifo/logdemux.error.log &
# installed syslogd has a bug it won't accept more than -b 1 --help says 99 max
nohup syslogd -m 0 -O /jffs/fifo/syslogfifo -b 1 -S -s 4096 -l 7 2> /jffs/fifo/syslogd.error.log &

/jffs/fifo/logdemux.sh

#!/bin/sh
# /jffs/fifo/logdemux.sh
while :
do
awk '   /kernel: IPT ICMP/ {print $0 >> "/tmp/mnt/usb/log/ipt-icmp.log" ; next}
        /kernel: IPT/ {print $0 >> "/tmp/mnt/usb/log/ipt-other.log" ; next}
        // {print $0 >> "/tmp/mnt/usb/log/syslog.log" ; next}   ' /jffs/fifo/syslogfifo
done

You can easily filter out lines from the syslog just by matching with a regex, here you need to include

-j LOG --log-prefix "IPT "
# or
-j LOG --log-prefix "IPT ICMP "

when logging from iptables.
/match/ and where it's output to, removed from syslog.
Awk offers great possibilities for massaging the output in case you find the default format too verbose/redundant.

If you want to keep the match in syslog and only copy it to another log file, remove "; next" from the match and see that it will hit the default // match later, or copy the match and change the output file to syslog.log . (untested but should work)

Comments, questions, especially improvements welcome, I wanted to have a single awk script in an infinite loop (like tail -f) but my awk-fu failed miserably, so this one is executed once for every line received through the named pipe.

 

[ml70]

CHANGE: logdemux.sh line 5: the fifo is best read from as

cat<>/jffs/fifo/syslogfifo | awk '   /kernel: IPT ICMP/ ....

Delete /jffs/fifo/syslogfifo from end of last awk line.

CHANGE: syslog-fifo.sh line 12: the fifo is no longer needed to kept open, comment the line out

# nohup sleep 4294967295 > /jffs/fifo/ ...

Still using it, the more keywords one adds for separating message streams to different logfiles the more powerful it becomes. Great for exporting ip addresses or port numbers to userland to then process them in various ways; ipset will auto increase the hashsize when necessary when called from shell.

Example of matching a source ip address in awk and calling ipset:

/kernel: LOWPORT-UDP/ { ipaddr=gensub(/(.* <1>SRC=)(([0-9]+\.){3}[0-9]+)( .*)/,"\\2",1,$0) ; system("ipset -A lowport-udp " ipaddr) }

Don't know if that <1>SRC is a peculiarity of AC66 linux/syslogd version, check your syslog.

 

[ml70]

It's very useful to create a CURRENT match

/kernel: CURRENT / {print $0 >> "/tmp/mnt/usb/log/current.log" ; next}

it enables one to very quickly check for current issues by
-j LOG --log-prefix "CURRENT " --log-tcp-sequence --log-tcp-options --log-ip-option
and tail -f current.log