Configuring syslog-ng with merlin firmware

[elorimer]

Something weird is happening. I can't consistently filter the hourly summary statistics.

Your first filter worked for the hourlies for me:

filter f_skynet { match("Skynet: " value("MESSAGE")); };

Which I think is the same as:

filter f_skynet { program("Skynet"); };

The same doesn't work for the BLOCKED messages because they come from the kernel.

And then by redirecting skynet to the skynet log file, the BLOCKED messages are stripped every hour, leaving the hourlies.

 

[cmkelley]

All the indent-multiline does is add a tab character to the new line character, I think. Of course it will pass the syntax check, as that won't tell you why nothing matches it.

On a slightly different note, I think it would be helpful to leave all the syslog-ng stuff in this thread, even all the grousing. No offense, but having a discussion about openvpn logging in the stubby thread makes it harder to help others.

I will post later my openvpn1 &2 server filters.

Right, but I viewed indent-multi-line as litmus test - if it was truly a multi-line message, then the indent function would indent it. Since it didn't indent, either I used it wrong, or it's not a multi-line message. I'm pretty convinced at this point that it's not a multi-line message.

Agree on putting stuff here. My bad.

Your first filter worked for the hourlies for me:

filter f_skynet { match("Skynet: " value("MESSAGE")); };

Which I think is the same as:

filter f_skynet { program("Skynet"); };

The same doesn't work for the BLOCKED messages because they come from the kernel.

And then by redirecting skynet to the skynet log file, the BLOCKED messages are stripped every hour, leaving the hourlies.

Really? Okay, lemme change it back and see if it magically starts working again. Can't say I've never had that experience working with syslog-ng. I think some of my difficulties may be from using syslog-ng-ctl to reload the configuration rather than restarting the service through the init.d script.

 

[cmkelley]

Heres what I use for logs for my openvpn servers:

destination openvpn1 { file("/opt/var/log/openvpn1"); };
destination openvpn2 { file("/opt/var/log/openvpn2"); };
filter f_openvpn1 { facility(daemon) and program("ovpn-server1"); };
filter f_openvpn2 { facility(daemon) and program("ovpn-server2"); };
log { source(src); filter(f_openvpn1); destination(openvpn1); };
log { source(src); filter(f_openvpn2); destination(openvpn2); };

So, openvpn stuff has literally never showed up in my system logs. Because I came from pfsense running on an old Pentium Pro, I had gone through the OpenVPN manual quite extensively looking at how to set it up. So when I started using Merlin, and it didn't have a spot for where to store the log, I just added "log-append /opt/var/log/openvpn.log" to the custom configuration box. So from day one, I've had the openvpn stuff in its own log. In my view, having OpenVPN skip the system log entirely is preferable. YMMV.

 

[cmkelley]

Your first filter worked for the hourlies for me:

filter f_skynet { match("Skynet: " value("MESSAGE")); };

Which I think is the same as:

filter f_skynet { program("Skynet"); };

The same doesn't work for the BLOCKED messages because they come from the kernel.

And then by redirecting skynet to the skynet log file, the BLOCKED messages are stripped every hour, leaving the hourlies.

Shot myself square in the foot. They both work. Given when it started working, I think the problem had something to do with my using tmux. Hella useful tool, but I think I need to exit completely out of it (not just detach from the session) when making system-level changes. Or maybe I'm just losing my mind.

For your example I _think_ that's not how the program() function works. The program() function is for use in the source directive, like file() or unix-stream(). I think the match on MSGHDR is the same as:

filter f_skynet { match("Skynet" value("PROGRAM")); };

But I read somewhere that the PROGRAM template may or may not be working correctly. Like, despite the documentation, many of the templates weren't actually implemented. But it was an old message and tmux may have been causing some problems as well.

Do you have any files in syslog-ng.d with hyphens or underscores in them? I almost have myself convinced those may not work correctly. Again see tmux above and losing my mind.

 

[cmkelley]

Shot myself square in the foot. They both work. Given when it started working, I think the problem had something to do with my using tmux. Hella useful tool, but I think I need to exit completely out of it (not just detach from the session) when making system-level changes. Or maybe I'm just losing my mind.

For your example I _think_ that's not how the program() function works. The program() function is for use in the source directive, like file() or unix-stream(). I think the match on MSGHDR is the same as:

filter f_skynet { match("Skynet" value("PROGRAM")); };

But I read somewhere that the PROGRAM template may or may not be working correctly. Like, despite the documentation, many of the templates weren't actually implemented. But it was an old message and tmux may have been causing some problems as well.

Do you have any files in syslog-ng.d with hyphens or underscores in them? I almost have myself convinced those may not work correctly. Again see tmux above and losing my mind.

Arrrrrrrrghhhhhh!!! Looking at my logs more carefully, it's freaking sporadic if it filters it or not!!! Whiskey. Tango. Fox.

 

[elorimer]

Maybe it would help if we looked at the structure of a message that is going into the syslog so we are on the same page. I can't pretend to understand this, and of course we are using a big hammer on a small thumbtack. (For example, I don't think we have any multiline messages from AsusWRT, in the sense that syslog-ng understands them).
Here's one log entry at random:

Feb 10 11:49:07 RT-AC87R miniupnpd[2616]: Listening for NAT-PMP/PCP traffic on port 5351

If I follow the way syslog-ng parses this, there is no priority at the beginning of the line, so no PRI part of this.
The date stamp and "RT-AC87R" is the HEADER part of the message, divided into the timestamp and the hostname.
All the rest is MSG part, divided into the name of the program and the message text, divided by the ":" Here, the name of the program includes the pid in brackets. I believe program() returns the program name, without the PID. msghdr() returns both. (Fun, isnt it: msghdr parses a portion of the msg part, and not the header part)

That's why I think program("Skynet") works. match("Skynet:") in value("MESSAGES") does too, but wouldn't if there was a PID. Also, match() in value() is now just message().

Note that used in the filter, this program() is different than the program() in the sources, which uses another program to collect log messages.

 

[cmkelley]

Maybe it would help if we looked at the structure of a message that is going into the syslog so we are on the same page. I can't pretend to understand this, and of course we are using a big hammer on a small thumbtack.
Here's one log entry at random:

Feb 10 11:49:07 RT-AC87R miniupnpd[2616]: Listening for NAT-PMP/PCP traffic on port 5351

If I follow the way syslog-ng parses this, there is no priority at the beginning of the line, so no PRI part of this.
The date stamp and "RT-AC87R" is the HEADER part of the message, divided into the timestamp and the hostname.
All the rest is MSG part, divided into the name of the program and the message text, divided by the ":" Here, the name of the program includes the pid in brackets. I believe program() returns the program name, without the PID. msghdr() returns both. (Fun, isnt it: msghdr parses a portion of the msg part, and not the header part)

That's why I think program("Skynet") works. match("Skynet:") in value("MESSAGES") does too, but wouldn't if there was a PID. Also, match() in value() is now just message().

Note that used in the filter, this program() is different than the program() in the sources, which uses another program to collect log messages.

Ahhhh, found that bit in the manual finally. Geh. Lots of the examples on the web are probably written against earlier versions.

So, what I've used to look at what "actually" happening is to use the following:

# print a huge log telling everything about the incoming log

template t_expandlog {
        template("================================================
        PROGRAM=$PROGRAM#
        PID=$PID#
        BSDTAG=$BSDTAG#
        TAG=$TAG#
        TAGS=$TAGS#
        FACILITY=$FACILITY#
        FACILITY_NUM=$FACILITY_NUM#
        LEVEL=$LEVEL#
        LEVEL_NUM=$LEVEL_NUM#
        PRI=$PRI#
        PRIORITY=$PRIORITY#
        FULLHOST=$FULLHOST#
        FULLHOST_FROM=$FULLHOST_FROM#
        HOST=$HOST#
        HOST_FROM=$HOST_FROM#
        LOGHOST=$LOGHOST#
        MSGHDR=$MSGHDR#
        MSGID=$MSGID#
        MSGONLY=$MSGONLY#
        MSG=$MSG#
        MESSAGE=$MESSAGE#
        SOURCE=$SOURCE#
        SOURCEIP=$SOURCEIP#
        SOURCE_IP=$SOURCE_IP#
        SEQNUM=$SEQNUM#
        UNIXTIME=$UNIXTIME#
        FULLDATE=$FULLDATE#
        ISODATE=$ISODATE#
        DATE=$DATE#
        STAMP=$STAMP#
        TZ=$TZ#
        TZOFFSET=$TZOFFSET#
        SEC=$SEC#
        MIN=$MIN#
        HOUR=$HOUR#
        HOUR12=$HOUR12#
        DAY=$DAY#
        WEEK=$WEEK#
        WEEK_DAY=$WEEK_DAY#
        WEEK_DAY_ABBREV=$WEEK_DAY_ABBREV#
        WEEK_DAY_NAME=$WEEK_DAY_NAME#
        MONTH=$MONTH#
        MONTH_ABBREV=$MONTH_ABBREV#
        MONTH_NAME=$MONTH_NAME#
        MONTH_WEEK=$MONTH_WEEK#
        YEAR=$YEAR#
        YEAR_DAY=$YEAR_DAY#\n\n");
        template_escape(no);
};

destination d_expandlog {
        file("/opt/var/log/syslog-expanded.log" template(t_expandlog));
};

log {
        source(src);
        source(kernel);
        destination(d_expandlog);
};

#eof

Obviously this creates a huge log file, so should only be used to see what's going on. I added the "#" at the end of every line to see where there are "hidden" spaces. At least it tells you what's in every field. I didn't come up with this myself, I stole it from someone's web post.

 

[cmkelley]

Maybe it would help if we looked at the structure of a message that is going into the syslog so we are on the same page. I can't pretend to understand this, and of course we are using a big hammer on a small thumbtack. (For example, I don't think we have any multiline messages from AsusWRT, in the sense that syslog-ng understands them).
Here's one log entry at random:

Feb 10 11:49:07 RT-AC87R miniupnpd[2616]: Listening for NAT-PMP/PCP traffic on port 5351

If I follow the way syslog-ng parses this, there is no priority at the beginning of the line, so no PRI part of this.
The date stamp and "RT-AC87R" is the HEADER part of the message, divided into the timestamp and the hostname.
All the rest is MSG part, divided into the name of the program and the message text, divided by the ":" Here, the name of the program includes the pid in brackets. I believe program() returns the program name, without the PID. msghdr() returns both. (Fun, isnt it: msghdr parses a portion of the msg part, and not the header part)

That's why I think program("Skynet") works. match("Skynet:") in value("MESSAGES") does too, but wouldn't if there was a PID. Also, match() in value() is now just message().

Note that used in the filter, this program() is different than the program() in the sources, which uses another program to collect log messages.

Okay, narrowing it down. I think the "match(X value(Y))" is unreliable, or at least has nuances I don't understand. I've changed some of my log scripts to use the program() and message() functions as you suggested, and they seem to be more reliable; I had a hack work-around for scrubbing crashes off into their own file, I was able to remove the hack by changing it to all message() functions.

 

[Zonkd]

@elorimer regarding my other post: (in the Diversion thread I asked whether it was possible to send dnsmasq logs to a Remote Log Server. I asked if this could be done by including dnsmasq inside the syslog.) I have since abandoned the idea because the log file would probably become too large very quickly. The router syslog.log file sent to my Remote Log Server has already reached 16.7MB in 1 day. I can only imagine how large it'd become with dnsmasq included. Probably a better idea to keep it simple and cron cp dnsmasq.log file to the USB storage. I just need to retain 2 weeks of dnsmasq logs to inspect as required. If anyone has discovered the best way to do this then I'd love to hear about it.

 

[cmkelley]

@elorimer regarding my other post: (in the Diversion thread I asked whether it was possible to send dnsmasq logs to a Remote Log Server. I asked if this could be done by including dnsmasq inside the syslog.) I have since abandoned the idea because the log file would probably become too large very quickly. The router syslog.log file sent to my Remote Log Server has already reached 16.7MB in 1 day. I can only imagine how large it'd become with dnsmasq included. Probably a better idea to keep it simple and cron cp dnsmasq.log file to the USB storage. I just need to retain 2 weeks of dnsmasq logs to inspect as required. If anyone has discovered the best way to do this then I'd love to hear about it.

Still sounds like a syslog-ng / logrotate solution. Push the logs to syslog-ng and have it filter them to your usb. Then use logrotate to keep the file sizes however big you deem manageable and auto-delete after 2 weeks. Or set up syslog-ng & logrotate on your remote server as well and do all that there.

 

[elorimer]

The router syslog.log file sent to my Remote Log Server has already reached 16.7MB in 1 day.

Gosh, that is way out of line with my experience, like by two orders of magnitude. The dnsmasq.log that Diversion manages and rotates every week isn't close to that either. Whatever you are logging you don't want to handle with syslogd. Whatever are you logging?

If you are using Diversion, then it is already sending the dnsmasq log to /opt/var/log/dsnmasq.log, and rotating it weekly into log1 and log2. So you can see now exactly how large this portion of the log would be, and you don't need a cron job.

 

[Butterfly Bones]

I tried to set this up again on my AC86U last weekend, and had the same failure that I had earlier in this thread (maybe a year or so back). I'll wait for the installation method being worked on to try again. I'm an amateur with no formal networking background a mental block about scripting.

One thing I have been researching along with this is a good way to remote monitor the separate logs that syslog-ng allows. I expect to have the regular syslog, and want to separate Skynet, ChkWan, VPN_Failover, pixelserv-tls, ovpnvpn-clientX, IPSec server, WLCEVENTD, kernel: ethX, kernel: broX, and so on. (Maybe not all these, but most of them.

I want to monitor the router from my Linux desktop, both at home, and possibly a chromebook or iPad when away from home. I've search for remote monitoring programs, but all seem to be aimed at large corporate hosts or businesses. I just need something simple and no or low cost. A web based option would be best I think, but all I have found are too expensive for my meager budget.

 

[cmkelley]

Here is what the default .conf file included for me:

source s_network {
    default-network-drivers(
        # NOTE: TLS support
        #
        # the default-network-drivers() source driver opens the TLS
        # enabled ports as well, however without an actual key/cert
        # pair they will not operate and syslog-ng would display a
        # warning at startup.
        #
        #tls(key-file("/path/to/ssl-private-key") cert-file("/path/to/ssl-cert"))
    );
};

Using this, syslog-ng failed to start for me. Checking it with the -s parameter flagged an error on the default-network-drivers line. So I think the whole block needs to be commented out or it will choke on what is left uncommented.

Thanks, by the way, for answering whether the placement of the @include statement was a problem.

When it comes to updating both logrotate and syslog-ng, it notices that the default .conf files are different than the existing .conf files, and puts up a warning and names the new default files .conf-pkg. Pretty slick.

Okay, took the dive. syslog-ng -s only fails if I comment out the @include "scl.conf" line. But if I don't comment that out, it loads a whole lot of routines (everything under /opt/share/syslog-ng/include/scl). Like pages and pages. Useful to some I'm sure, looks like it really brings out the features of syslog-ng. I'm struggling with how to handle that. I think I'll (eventually) have to add an option to comment all that out, because most people won't want it.

 

[elorimer]

Okay, took the dive. syslog-ng -s only fails if I comment out the @include "scl.conf" line. But if I don't comment that out, it loads a whole lot of routines (everything under /opt/share/syslog-ng/include/scl). Like pages and pages. Useful to some I'm sure, looks like it really brings out the features of syslog-ng. I'm struggling with how to handle that. I think I'll (eventually) have to add an option to comment all that out, because most people won't want it.

I think I'm following. The block default-network-drivers is defined in the scl conf files. So if you comment out that @include line, the block reference fails. If you don't comment it out, then perhaps the inclusion of the block fails the -s (for me) because of no parameters, but when run it is okay because it doesn't do anything.

 

[cmkelley]

I think I'm following. The block default-network-drivers is defined in the scl conf files. So if you comment out that @include line, the block reference fails. If you don't comment it out, then perhaps the inclusion of the block fails the -s (for me) because of no parameters, but when run it is okay because it doesn't do anything.

For me, if I don't comment out the @include scl.conf line, then syslog-ng -s does not fail. And yes, I am assuming that the default-network-drivers() routine is defined in one of the included files, though I didn't go looking for it.

You can see the entire loaded configuration including stuff loaded with @include statements with:

syslog-ng --preprocess-into=/tmp/syslog-ng-complete.conf && cat /tmp/syslog-ng-complete.conf | less