Scribe scribe - syslog-ng and logrotate installer

[cmkelley]

Hi all, really liking this tool, thanks to the developer! I added one line to /opt/etc/syslog-ng.d/openvpn to catch the 'openvpn-event' entries, would it be possible to add to the example file in future? Thanks.

Please post the exact line you're adding.

 

[elorimer]

You might change the A00remote example, given the other thread, to something like this:

# send _ALL_ messages to a remote logging facility; it is recommended to set the remote log
# server here instead of in syslog-ng.conf if you are using this method to keep all remote
# information in one file - if you do set the log_server destination in syslog-ng.conf,
# you will need to comment out the lines here that define it.  If you define it here
# comment out one or the other destination
# the following definition will send messages to a remote syslog-ng server
# the format is RFC-3164, which is normally used by syslog-ng
destination log_server {
    udp("192.168.x.y" port(514));
};
# the following definition will send messages to a remote server in the IETF 5424 format.
# If the hostname of the target server will not change, insert below the IP of the remote server.
# It is not recommended to configure syslog-ng to do name resolution, as it will stall if the name
# server is unavailable
# the default transport protocol is tcp, and the default port is 601.  These can be changed.
# It is also possible to use TLS encryption, for which see the manual
# destination log_server {
#    syslog("10.1.2.3");
#};
log {
    source(src);
    destination(log_server);
};
# eof

I don't have a remote server like that to test.

 

[elorimer]

You might also add to examples a loggly configuration, named 0loggly:

### Syslog-ng Logging Directives for Loggly.com ###
# This will send all messages to loggly, without disturbing the normal scribe processing
# Use filter definitions here to exclude messages to be sent to loggly, cutting down the traffic so as to remain within the free cap.
# These should use the filters defined in the separate logs that will be excluded
filter f_loggly1 { not filter("f_pixelserv"); };
filter f_loggly2 { not filter("f_skynet"); };
# Change this template to insert your own unique loggly code and the name of your router
# Loggly will accept messages from other sources and use this tag to identify them
#
template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [LogglyCode tag=\"RouterName\" ] $MSG\n");
    template_escape(no);
};
destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log { 
    source(src); 
    filter(f_loggly1);
    filter(f_loggly2);
    destination(d_loggly); 
};
### END Syslog-ng Logging Directives for Loggly.com ###

 

[andrew2018]

Please post the exact line you're adding.

Here's the whole file /opt/etc/syslog-ng.d/openvpn with the line added for 'openvpn-event'. Thanks.

# log all openvpn server and client logs into one file - /opt/var/log/openvpn.log and stop processing openvpn logs

destination d_openvpn {
    file("/opt/var/log/openvpn.log");
};

filter f_openvpn {
    program("ovpn-server1") or
    program("ovpn-server2") or
    program("ovpn-client1") or
    program("ovpn-client2") or
    program("ovpn-client3") or
    program("ovpn-client4") or
    program("ovpn-client5") or
    program("openvpn-event") or
    program("openvpn-routing");
};

log {
    source(src);
    filter(f_openvpn);
    destination(d_openvpn);
    flags(final);
};

#eof

 

[cmkelley]

Here's the whole file /opt/etc/syslog-ng.d/openvpn with the line added for 'openvpn-event'. Thanks.

# log all openvpn server and client logs into one file - /opt/var/log/openvpn.log and stop processing openvpn logs

destination d_openvpn {
    file("/opt/var/log/openvpn.log");
};

filter f_openvpn {
    program("ovpn-server1") or
    program("ovpn-server2") or
    program("ovpn-client1") or
    program("ovpn-client2") or
    program("ovpn-client3") or
    program("ovpn-client4") or
    program("ovpn-client5") or
    program("openvpn-event") or
    program("openvpn-routing");
};

log {
    source(src);
    filter(f_openvpn);
    destination(d_openvpn);
    flags(final);
};

#eof

Have you verified this addition catches anything? I ask because openvpn-event is a script, not a program. The only log entry I have that mentions openvpn-event is

custom_script: Running /jffs/scripts/openvpn-event (args: REDACTED init)

And I'm pretty sure the program filter wouldn't see the program as openvpn-event.

 

[cmkelley]

You might change the A00remote example, given the other thread, to something like this:

# send _ALL_ messages to a remote logging facility; it is recommended to set the remote log
# server here instead of in syslog-ng.conf if you are using this method to keep all remote
# information in one file - if you do set the log_server destination in syslog-ng.conf,
# you will need to comment out the lines here that define it.  If you define it here
# comment out one or the other destination
# the following definition will send messages to a remote syslog-ng server
# the format is RFC-3164, which is normally used by syslog-ng
destination log_server {
    udp("192.168.x.y" port(514));
};
# the following definition will send messages to a remote server in the IETF 5424 format.
# If the hostname of the target server will not change, insert below the IP of the remote server.
# It is not recommended to configure syslog-ng to do name resolution, as it will stall if the name
# server is unavailable
# the default transport protocol is tcp, and the default port is 601.  These can be changed.
# It is also possible to use TLS encryption, for which see the manual
# destination log_server {
#    syslog("10.1.2.3");
#};
log {
    source(src);
    destination(log_server);
};
# eof

I don't have a remote server like that to test.

I'd prefer not to add untested stuff. I also don't know what "other thread" you're talking about .

You might also add to examples a loggly configuration, named 0loggly:

### Syslog-ng Logging Directives for Loggly.com ###
# This will send all messages to loggly, without disturbing the normal scribe processing
# Use filter definitions here to exclude messages to be sent to loggly, cutting down the traffic so as to remain within the free cap.
# These should use the filters defined in the separate logs that will be excluded
filter f_loggly1 { not filter("f_pixelserv"); };
filter f_loggly2 { not filter("f_skynet"); };
# Change this template to insert your own unique loggly code and the name of your router
# Loggly will accept messages from other sources and use this tag to identify them
#
template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [LogglyCode tag=\"RouterName\" ] $MSG\n");
    template_escape(no);
};
destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log {
    source(src);
    filter(f_loggly1);
    filter(f_loggly2);
    destination(d_loggly);
};
### END Syslog-ng Logging Directives for Loggly.com ###

Is this the actual template you're using or have used and verified to work? If so, I'll add, but I'm still gonna stick "submitted by elorimer" in the file..

 

[joe scian]

I'd prefer not to add untested stuff. I also don't know what "other thread" you're talking about .

Is this the actual template you're using or have used and verified to work? If so, I'll add, but I'm still gonna stick "submitted by elorimer" in the file..

template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [LogglyCode@41058 tag=\"RouterName\" ] $MSG\n");
    template_escape(no);

The loggly conf above is missing a parameter - I had to use LogglyCode@41058 where LogglyCode = Customer Token consistent with syslog-ng conf as described in Loggly directions.

 

[Dabombber]

You might also add to examples a loggly configuration, named 0loggly

There's already includes for loggy (/opt/share/syslog-ng/include/scl/loggly/loggly.conf), documentation here. Just include @include "scl.conf" in the main syslog-ng configuration file and you can use something like:

### Syslog-ng Logging Directives for Loggly.com ###
# This will send all messages to loggly, without disturbing the normal scribe processing
# Use filter definitions here to exclude messages to be sent to loggly, cutting down the traffic so as to remain within the free cap.
# These should use the filters defined in the separate logs that will be excluded
filter f_loggly1 { not filter("f_pixelserv"); };
filter f_loggly2 { not filter("f_skynet"); };

log {
    source(src);
    filter(f_loggly1);
    filter(f_loggly2);
   destination { loggly(token("<USER-TOKEN-AS-PROVIDED-BY-LOGGLY>")); };
};
### END Syslog-ng Logging Directives for Loggly.com ###

 

[elorimer]

I'd prefer not to add untested stuff. I also don't know what "other thread" you're talking about .

Here: Remote Log Server, RFC 5424

 

[elorimer]

Is this the actual template you're using or have used and verified to work? I

Yes, with the exception as noted of sanitizing the loggly code--or maybe over-sanitizing it.

 

[elorimer]

There's already includes for loggy (/opt/share/syslog-ng/include/scl/loggly/loggly.conf), documentation here. Just include @include "scl.conf" in the main syslog-ng configuration file and you can use something like:

Nice find. This isn't part of the loggly documentation at SolarWinds. I had included scl.conf when I was fooling with the collectd driver. I'm not sure what the memory implications of including all of the other drivers are. The scribe version of syslog-ng.conf strips out this include statement, and I'm not sure what happens on an update but at this point my .conf file doesn't include it anymore.

 

[andrew2018]

Have you verified this addition catches anything? I ask because openvpn-event is a script, not a program. The only log entry I have that mentions openvpn-event is

custom_script: Running /jffs/scripts/openvpn-event (args: REDACTED init)

And I'm pretty sure the program filter wouldn't see the program as openvpn-event.

You're right. I remembered I made that script file a long time ago from this reference:
https://www.snbforums.com/threads/f...lts-releases-v43e6.18914/page-238#post-294825

 

[kidd232]

Hello, I installed the scribe and uiScribe, and updated the filter in scribe, then found the wlceventd.log is empty.
I have already tried to reinstall both scribe and uiScribe, nothing changed, please help to correct the log setting.

 

[elorimer]

Is there stuff being written to /opt/var/log/wlceventd.log? What router do you have?

 

[kidd232]

Is there stuff being written to /opt/var/log/wlceventd.log? What router do you have?

Hi elorimer,

There's no wlceventd.log in the path, my router is rt-n18u and running gzenux merlin fork.

 

[cmkelley]

Hi elorimer,

There's no wlceventd.log in the path, my router is rt-n18u and running gzenux merlin fork.

I don't know anything about that router but I'd guess that it simply doesn't have the wlceventd daemon, so it doesn't create any log entries. The file won't exist until something is written to it. Might be unique to 801.11ac routers.

 

[elorimer]

Hi elorimer,

There's no wlceventd.log in the path, my router is rt-n18u and running gzenux merlin fork.

If there is no log file then having it be empty in the webgui isn't a surprise. If you don't want it to appear it in the webgui then delete the config file for it, rerun uiScribe, or use uiScribe to exclude it.

 

[amoney]

Are there any issues with wlceventd on RT-AC88U? Just setup scribe and wlceventd are still present in the default log file

Asus: 384.17
scribe: v2.4.2

cat /opt/etc/syslog-ng.d/wlceventd
# put wlceventd Assoc/ReAssoc/Disassoc messages into /opt/var/log/wlceventd.log

destination d_wlceventd {
    file("/opt/var/log/wlceventd.log");
};

filter f_wlceventd {
    ( program("WLCEVENTD") or
    program ("wlceventd") ) and
    ( message("ssoc") or
    message("uth") );
};

log {
    source(src);
    filter(f_wlceventd);
    destination(d_wlceventd);
    flags(final);
};

#eof

grep 'WLCEVENTD' /opt/var/log/messages

May  9 17:19:40 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(401): eth1: Disassoc F0:5C:77:C2:6D:C6, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8)
May  9 18:21:23 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(420): eth1: Auth F0:5C:77:C2:6D:C6, status: 0, reason: d11 RC reserved (0)
May  9 18:21:23 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(449): eth1: Assoc F0:5C:77:C2:6D:C6, status: 0, reason: d11 RC reserved (0)
May  9 18:22:28 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(420): eth1: Auth C0:B6:F9:16:68:B8, status: 0, reason: d11 RC reserved (0)
May  9 18:22:28 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(449): eth1: Assoc C0:B6:F9:16:68:B8, status: 0, reason: d11 RC reserved (0)

Also thanks for this amazing tool.

Also does every syslog-ng.d require a corresponding log rotate config or will it pick up pre defined defaults?

 

[cmkelley]

Are there any issues with wlceventd on RT-AC88U? Just setup scribe and wlceventd are still present in the default log file

Asus: 384.17
scribe: v2.4.2

cat /opt/etc/syslog-ng.d/wlceventd
# put wlceventd Assoc/ReAssoc/Disassoc messages into /opt/var/log/wlceventd.log

destination d_wlceventd {
    file("/opt/var/log/wlceventd.log");
};

filter f_wlceventd {
    ( program("WLCEVENTD") or
    program ("wlceventd") ) and
    ( message("ssoc") or
    message("uth") );
};

log {
    source(src);
    filter(f_wlceventd);
    destination(d_wlceventd);
    flags(final);
};

#eof

grep 'WLCEVENTD' /opt/var/log/messages

May  9 17:19:40 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(401): eth1: Disassoc F0:5C:77:C2:6D:C6, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8)
May  9 18:21:23 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(420): eth1: Auth F0:5C:77:C2:6D:C6, status: 0, reason: d11 RC reserved (0)
May  9 18:21:23 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(449): eth1: Assoc F0:5C:77:C2:6D:C6, status: 0, reason: d11 RC reserved (0)
May  9 18:22:28 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(420): eth1: Auth C0:B6:F9:16:68:B8, status: 0, reason: d11 RC reserved (0)
May  9 18:22:28 RT-AC88U-BC40 syslog: WLCEVENTD wlceventd_proc_event(449): eth1: Assoc C0:B6:F9:16:68:B8, status: 0, reason: d11 RC reserved (0)

Also thanks for this amazing tool.

Also does every syslog-ng.d require a corresponding log rotate config or will it pick up pre defined defaults?

I'm guessing RT-AC88U-BC40 is an AiMesh node and you're directing its logs to your main router? The problem is that for some reason, the program name gets changed from "wlceventd" to "syslog". It appears to only happen to the wlceventd log entries, I've no idea why. I've been accumulating a few bits here and there, including this, probably time to push them out. In the meantime, you can change the filter to:

filter f_wlceventd {
    ( program("WLCEVENTD") or
    program("wlceventd") ) and
    ( message("ssoc") or
    message("uth") ) or
    ( program("syslog") and
    message("wlceventd") );
};

in either your AiMesh node or your main router. If you change it on you AiMesh node, the wlceventd messages won't get passed to the main router at all, they'll stay on the AiMesh node router.

 

[cmkelley]

Also does every syslog-ng.d require a corresponding log rotate config or will it pick up pre defined defaults?

No, it doesn't pick up pre-defined defaults, BUT some logs that live there shouldn't be touched by logrotate, particularly any skynet logs (skynet-0, skynet.log, & skynet_events.log).

In general, if you write your own syslog-ng filter, there needs to be a corresponding logrotate config. Any you copy from /opt/share/syslog-ng.d/examples should have a corresponding logrotate config in /opt/share/logrotate/examples, and it should be copied to /opt/etc/logrotate.d when you copy the syslog filter.