Configuring syslog-ng with merlin firmware

[Xentrk]

For anyone else landing here wanting to install and configure. I found that v3.9.1 places a default config file in /opt/etc/syslog-ng.conf. You then place your custom config file in /opt/etc/syslog-ng.d/syslog-ng.conf. I used the @kvic version posted here with modifications. I had to remove the entries that were already in the /opt/etc/syslog-ng.conf file. That got it to work. Here is the conf file I used:

destination d_iptables { file("/opt/var/log/iptables.log"); };
destination d_pixelserv { file("/opt/var/log/pixelserv.log"); };

filter f_iptables { facility(kern) and message("DROP IN="); };
filter f_pixelserv { facility(daemon) and program("pixelserv"); };
filter f_default { not filter(f_pixelserv) and not filter(f_iptables); };

log { source(src); filter(f_pixelserv); destination(d_pixelserv); };
log { source(src); filter(f_iptables); destination(d_iptables); };

 

[tomsk]

now that diversion has put its dnsmasq.log in the Entware directory, if you want syslog-ng to grab that you can select it as a file source

source s_file {
        file("/opt/var/log/dnsmasq.log" default-facility(daemon) flags(no-hostname));

then add the source to the which ever log statement you want

log { source(s_file);source(src); destination(messages); };

 

[Xentrk]

@tomsk, Thank you for posting your "how to" tips as you progressed thru the syslog-ng and logrotate entware packages. I took some notes along the way and will write a how-to configure guide on my blog site. I'll ask you and @kvic to look it over when I get the first draft done.

I had time today to create cron jobs to rotate the log files and purge files in the /opt/var/log directory > 30 days. I added the &> /dev/null to the logrotate command to suppress the error message that appears because syslogd is not running.

/jffs/scripts/services-start

#!/bin/sh
# rotate logs
cru a LogRotate "0 0 * * * /opt/sbin/logrotate -f /opt/etc/logrotate.conf &>/dev/null"

# purge logs
cru a PurgeLogs "0 1 * * * find /opt/var/log -atime +30 -delete"

Spoiler: logrotate.conf

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /opt/etc/logrotate.d

# system-specific logs may be also be configured here.

/opt/var/log/iptables.log {
size 1024k
weekly
rotate 9
postrotate
killall -HUP syslogd
endscript
}

/opt/var/log/messages {
size 1024k
weekly
rotate 9
postrotate
killall -HUP syslogd
endscript
}

/opt/var/log/pixelserv.log {
size 1024k
weekly
rotate 9
postrotate
killall -HUP syslogd
endscript
}

 

[tomsk]

If you are going to purge your logs with a cron job like that , would it be safer to use -mtime?
The "rotate 9" means that anything older than 9 weeks will be deleted anyway. if you want to delete logs based on days you could use "maxage 30" instead

 

[kvic]

You have the option not to specify "weekly". Then it'll go with default if I don't remember wrong. That means everyday logrotate runs and check the conditions. It will roll forward when size > specified say 1024k (which btw is too small). Once it reaches 9 1024K files. The oldest will be automatically purged on a daily basis. Not very smart but good enough for everyday use on a very busy router.

 

[Xentrk]

@tomsk & @kvic
Thank you for clarifying how the rotate and purging of the log files work. I'll ditch the cron job to purge the files and let logrotate manage it using the values in the logrotate.conf file.

 

[tomsk]

You have the option not to specify "weekly". Then it'll go with default if I don't remember wrong. That means everyday logrotate runs and check the conditions. It will roll forward when size > specified say 1024k (which btw is too small). Once it reaches 9 1024K files. The oldest will be automatically purged on a daily basis. Not very smart but good enough for everyday use on a very busy router.

Hi @kvic agreed 1024k is too small (certainly for the pixelserv log). In the case on my "messages" log, i symlink it to the syslog so that it can be read in the router UI. I find if i let it grow too large the UI will become slower and more unresponsive until it becomes unusable. Im currently set at 2048K, do you think thats a reasonable figure?

 

[kvic]

Hi @kvic agreed 1024k is too small (certainly for the pixelserv log). In the case on my "messages" log, i symlink it to the syslog so that it can be read in the router UI. I find if i let it grow too large the UI will become slower and more unresponsive until it becomes unusable. Im currently set at 2048K, do you think thats a reasonable figure?

I believe I used to set it at 256KB and tried 500KB as well as 1000KB. The log handling on WebUI is less than desirable..

For a very long time, my script killed off the httpd process under the hold right after boot. I recall once I need some quick change but the process can't be brought back online quickly. Since then I didn't kill off the process lol.. though I haven't connected to WebUI for many months.

I believe my current size is 1000KB. Most messages are filtered into separate files, it's less chatty. So you might be able to live on a smaller size..even down to 256KB the default.

In the other news, in pixelserv-tls 2.2, there will be improvement in terms of logging. In current versions, when you enable log LEVEL >=4, it adds an extra 50ms to 100ms to tav on a typical router. In v2.2, this will be negligible even on LEVEL=5 full time. Should be great news for syslog tinkers who want to keep a copy of all URL & data ever accessed to blocked domains!

 

[elorimer]

I wanted to note here two things I needed to do in getting syslog-ng working as I wanted it to.

1. In order for syslog-ng's own log messages to have the correct time stamp, I needed to include

export TZ=$(cat /etc/TZ)

in /opt/etc/init.d/S01syslog-ng.
2. Running "syslog-ng --syntax-only" was invaluable in hunting down errors in the .conf file. Among other things, it told me that the .conf file should start now with 3.16. Also, pure match() is now deprecated without combining it with value(), which means probably flags("prefix") or flags("substring") also is now required.

Also, because I was sifting the log into five separate logs, I found it easier to have a final log path of

log { source(src); destination(messages); flags(fallback); };

instead of a filter that was {not filter1 and not filter2 and not filter3, etc.

 

[tomsk]

Running "syslog-ng --syntax-only" was invaluable in hunting down errors in the .conf file. Among other things, it told me that the .conf file should start now with 3.16

Yes for unknown reasons the config file supplied with the entware version of syslog 3.16 still starts with @version:3.9, so don't forget to change that when you add your own sources, filters and destinations.
There will be a syslog message reminding you of this whenever the config is reloaded.
Interesting feature of 3.16 is that it can now send information to telegram.

 

[kvic]

2. Running "syslog-ng --syntax-only" was invaluable in hunting down errors in the .conf file. Among other things, it told me that the .conf file should start now with 3.16.

My conf still begins with "@version 3.9" and everything created inside back then still work as-is in v3.16

The bigger problem for Entware when you have more than a few services installed is dealing with config related files overwritten on every upgrade. People have to come up with their own innovative solutions..

Interesting feature of 3.16 is that it can now send information to telegram.

One useful feature for me is suppressing duplicate lines like in rsyslogd. I believe it was added in one of the earlier 3.x version.

 

[Xentrk]

I wanted to note here two things I needed to do in getting syslog-ng working as I wanted it to.

1. In order for syslog-ng's own log messages to have the correct time stamp, I needed to include

export TZ=$(cat /etc/TZ)

in /opt/etc/init.d/S01syslog-ng.
2. Running "syslog-ng --syntax-only" was invaluable in hunting down errors in the .conf file. Among other things, it told me that the .conf file should start now with 3.16. Also, pure match() is now deprecated without combining it with value(), which means probably flags("prefix") or flags("substring") also is now required.

Also, because I was sifting the log into five separate logs, I found it easier to have a final log path of

log { source(src); destination(messages); flags(fallback); };

instead of a filter that was {not filter1 and not filter2 and not filter3, etc.

Thanks for sharing the syntax check. I also changed the 3.9 reference to 3.16. I also removed the code below as the syntax-checker said sync and stats were obsolete.

options {
        sync(0);

        # The default action of syslog-ng 1.6.0 is to log a STATS line
        # to the file every 10 minutes.  That's pretty ugly after a while.
        # Change it to every 12 hours so you get a nice daily update of
        # how many messages syslog-ng missed (0).
        stats(43200);
};

 

[elorimer]

I'm using this:

options {
    chain_hostnames(no);
    flush_lines(0);
    stats_freq(43200);
   
};

sync() is a deprecated alias for flush_lines(), and stats_freq() is time in seconds between STATS messages (default 600).

 

[Xentrk]

I'm using this:

options {
    chain_hostnames(no);
    flush_lines(0);
    stats_freq(43200);
  
};

sync() is a deprecated alias for flush_lines(), and stats_freq() is time in seconds between STATS messages (default 600).

Thanks for sharing.

I'm still having issues with parsing messages using a filter to match on key words in a message. I've tried several examples I've gathered from my web searches with no luck. I'll try again with the syntax checker to see if it helps point me in the right direction.

 

[elorimer]

I've found the same in parsing messages. Still working that out. The syntax checker isn't sufficiently helpful with that since I get unexpected results with perfect syntax.

 

[tomsk]

Thanks for sharing.

I'm still having issues with parsing messages using a filter to match on key words in a message. I've tried several examples I've gathered from my web searches with no luck. I'll try again with the syntax checker to see if it helps point me in the right direction.

if you are using wildcards you might need to use type(glob) in your filter, otherwise it will treat the asterisk as a regex.
https://www.syslog-ng.com/technical...ion/3.16/administration-guide/52#TOPIC-956583

 

[elorimer]

I figure someone has come across this. My symbolic link in /tmp to my syslog-ng log regularly gets deleted and replaced by a syslog.log. Syslog-ng keeps running.

Setup: 87U on 7_2 firmware, with skynet, pixelserv and diversion. My syslog-ng separates out some messages, and leaves the rest in /opt/var/log/messages. In my init.d script I have

kill_syslogd () {
    if [[ ! -z `pidof syslogd` ]]; then
        logger -t syslog-ng "kill_syslogd run and link to messages formed"
    killall syslogd
        cat /tmp/syslog.log >> /opt/var/log/messages
        rm  /tmp/syslog.log /tmp/syslog.log-1
        ln -s /opt/var/log/messages /tmp/syslog.log       
    fi
}

In my /jffs I have replaced syslog.log and syslog.log-1 with directories.

Everything works fine, but some time after a reboot that link is broken. Any idea what is stomping on this?

 

[kvic]

I figure someone has come across this. My symbolic link in /tmp to my syslog-ng log regularly gets deleted and replaced by a syslog.log. Syslog-ng keeps running.

I recall I never ran into this issue before. But it was on an old firmware..very old by now.

@Butterfly Bones told me he was mad about this issue so he might be interested in a solution too.

Perhaps you could sort it out?

 

[elorimer]

I think I've figured this out. I think skynet regularly operates on syslog.log with sed -i, (in-place editing) which is destructive. I'll post over there.

EDIT: So yes. Nothing to be done. Skynet runs a chron at the top of each hour, so I'll add another one at five after to restore the symlink.

 

[kvic]

I think I've figured this out. I think skynet regularly operates on syslog.log with sed -i, (in-place editing) which is destructive. I'll post over there.

EDIT: So yes. Nothing to be done. Skynet runs a chron at the top of each hour, so I'll add another one at five after to restore the symlink.

Sounds like the destructive "sed" operation is for poor souls running the vanilla syslog. So for you running syslog-ng, commenting that out without performance degradation. No? A better way than to have a band-aid over another band-aid.

I've never run these scripts. So you'll have to experiment. But if you further figure it out the better way, it'll certainly benefit syslog-ng users in terms of efficiency, robustness and what else?!