Scribe scribe - syslog-ng and logrotate installer

[cmkelley]

issue with WLCEVENTD filter
they are getting placed in regular system messages instead of getting logged by the filter option provided through scribe.

syslog: WLCEVENTD wlceventd_proc_event(420): eth1:

Are you running an Aimesh setup? Are the messages coming from an Aimesh node?

 

[SomeWhereOverTheRainBow]

Are you running an Aimesh setup? Are the messages coming from an Aimesh node?

No log is locally on router, but it has nodes attached.

 

[elorimer]

I like to follow along, even when the error isn't affecting me.

I'm finding this discussion about WLCEVENTD pretty frustrating, because there are statements about "the regular system log" or "regular system messages". If one is running scribe, these are not intelligible statements. Scribe stops syslogd and klogd, which are the "regular" system processes, and replaces them with syslog-ng. Messages that are not otherwise filtered end up in the messages file, specifically /opt/var/log/messages. They do not end up in /tmp/syslog.log, which should be a link and not a file; when it is a file you might call it the regular system log, but then you are in the wrong thread.

So I can't tell from this train whether syslog-ng is running, or syslogd, or both. I can't tell where the log messages are being quoted from. @cmkelley is saying something similar, but more politely.

I am not in a place where I can check my own logs to be sure, but is it not true there cannot be a message logged to /opt/var/log/messages that has a program id of "syslog"? Is it not always "syslog-ng"? Is this not a sign that syslog-ng is not set up correctly? Purely rhetorical. EDIT: No, it is not true. But maybe the absence of a hostname is similarly indicative, unless it has been redacted as suggested

Also, to lighten the mood, 3.25 is out! And congrats to @cmkelley on being elected to the 1,000 message thread club!

 

[cmkelley]

No log is locally on router, but it has nodes attached.

Right, the log is on the router, but can you tell from IPs that are associating and disassociating if those IPs are connecting to the main router or one of the nodes? I suspect (but I could be wrong) that the nodes are sending their log messages to the main router.

 

[SomeWhereOverTheRainBow]

Right, the log is on the router, but can you tell from IPs that are associating and disassociating if those IPs are connecting to the main router or one of the nodes? I suspect (but I could be wrong) that the nodes are sending their log messages to the main router.

I believe they are the nodes.

 

[Swistheater]

Quite interesting. Something is getting between the logging facility and syslog-ng as near as I can tell. It appears to be saying the program is "syslog" and everything after "syslog:" is the message. I see you're running Aimesh ... can you tell if those messages coming from one of your mesh nodes and being pushed to the main router?

Just to verify, did you redact your router hostname from between the date and "syslog:" or was it not there?

Would you (both you and @SomeWhereOverTheRainBow ) either post or PM me the output of:

ps | grep log

Or, PM me the debug file from the scribe menu su, then d.

the XXXXX represents the admin account name

  427 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  429 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  430 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  431 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  432 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  433 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  434 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  435 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  436 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  437 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  438 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  439 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  440 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  441 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  442 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  443 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  444 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  445 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
 5289 XXXXXXXX  8492 S    {syslog-ng} supervising syslog-ng
 6505 nobody   99.2m S    dnsmasq --log-async
 6512 nobody   99.2m S    dnsmasq --log-async
10362 nobody   99.2m S    dnsmasq --log-async
10363 XXXXXXXX  3296 S    dnsmasq --log-async
31875 XXXXXXXX 15372 S    syslog-ng

 

[cmkelley]

I believe they are the nodes.

That starts to make sense then. The syslog on the nodes is sending its logs to the main router, that's why they're being tagged "syslog:". I don't have an Aimesh setup, nor the ability to do so, my testing router is a 3200, which doesn't support Aimesh, so I can't really do any testing to find a good way to test or sort them out. @Martineau's suggestion above to edit the filter seems like a reasonable hack. There is probably an elegant way to read and sort syslog messages from the nodes, but without an Aimesh setup to play with, I've really no ability to figure it out.

Syslog-ng isn't seeing the messages as WLCEVENTD messages, it's seeing them as syslog messages from the remote node, which is why they're not being filtered by the existing wlceventd filter.

 

[cmkelley]

the XXXXX represents the admin account name

  427 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  429 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  430 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  431 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  432 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  433 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  434 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  435 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  436 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  437 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  438 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  439 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  440 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  441 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  442 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  443 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  444 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
  445 XXXXXXXX  7072 S    /usr/lib/ipsec/charon --use-syslog
 5289 XXXXXXXX  8492 S    {syslog-ng} supervising syslog-ng
 6505 nobody   99.2m S    dnsmasq --log-async
 6512 nobody   99.2m S    dnsmasq --log-async
10362 nobody   99.2m S    dnsmasq --log-async
10363 XXXXXXXX  3296 S    dnsmasq --log-async
31875 XXXXXXXX 15372 S    syslog-ng

Okay, so syslog isn't running. I'm almost certain the messages are being passed from one of your Aimesh nodes to your main router, and the message header is showing the message coming from syslog on the node rather than wlceventd.

 

[Swistheater]

I have some very good news....

Quick hack - try modifying the filter:

filter f_wlceventd {
    ( program("WLCEVENTD") or
    program ("wlceventd") ) and
    ( message("ssoc") or
    message("uth") ) or
    message("wlceventd");
};

I have employeed martineaus HACK,
and it has resolved the issue.

These new logs are being sent to wlceventd filter now.

Once again, @Martineau has provided his shocking revolations and saves the day.

Thanks guys, and @cmkelley

when you come up with a more elegant approach just tag me and i will be glad to test as well.

 

[Martineau]

That starts to make sense then. The syslog on the nodes is sending its logs to the main router, that's why they're being tagged "syslog:". I don't have an Aimesh setup, nor the ability to do so, my testing router is a 3200, which doesn't support Aimesh, so I can't really do any testing to find a good way to test or sort them out. @Martineau's suggestion above to edit the filter seems like a reasonable hack. There is probably an elegant way to read and sort syslog messages from the nodes, but without an Aimesh setup to play with, I've really no ability to figure it out.

Syslog-ng isn't seeing the messages as WLCEVENTD messages, it's seeing them as syslog messages from the remote node, which is why they're not being filtered by the existing wlceventd filter.

I don't have AiMesh but in '/opt/var/messages' I can see for a specific MAC

Dec 14 08:43:42 RT-AC68U WLCEVENTD: eth2: Assoc XX:XX:XX:XX:XX:XX

<snip>

Dec 14 12:56:02 RT-AC68U-XXXX syslog: WLCEVENTD wlceventd_proc_event(386): eth2: Deauth_ind XX:XX:XX:XX:XX:XX, status: 0, reason: Unspecified reason (1)
Dec 14 12:56:33 RT-AC68U-XXXX syslog: WLCEVENTD wlceventd_proc_event(420): eth2: Auth XX:XX:XX:XX:XX:XX, status: 0, reason: d11 RC reserved (0)

Clearly '/usr/sbin/wlceventd'? is simply (legitimately) using 'logger -t' to generate the 'wlceventd_proc_event' entries.

NOTE: I can also spoof the same message from the command line:

logger -t syslog "WLCEVENTD wlceventd_proc_event(???): eth9: Auth xx:xx:xx:xx:xx:xx, status: 0, reason: d11 RC reserved (0)"

Guess what, I upgraded to v384.14 around lunchtime today!

 

[cmkelley]

I like to follow along, even when the error isn't affecting me.

I'm finding this discussion about WLCEVENTD pretty frustrating, because there are statements about "the regular system log" or "regular system messages". If one is running scribe, these are not intelligible statements. Scribe stops syslogd and klogd, which are the "regular" system processes, and replaces them with syslog-ng. Messages that are not otherwise filtered end up in the messages file, specifically /opt/var/log/messages. They do not end up in /tmp/syslog.log, which should be a link and not a file; when it is a file you might call it the regular system log, but then you are in the wrong thread.

So I can't tell from this train whether syslog-ng is running, or syslogd, or both. I can't tell where the log messages are being quoted from. @cmkelley is saying something similar, but more politely.

I am not in a place where I can check my own logs to be sure, but is it not true there cannot be a message logged to /opt/var/log/messages that has a program id of "syslog"? Is it not always "syslog-ng"? Is this not a sign that syslog-ng is not set up correctly? Purely rhetorical. EDIT: No, it is not true. But maybe the absence of a hostname is similarly indicative, unless it has been redacted as suggested

Also, to lighten the mood, 3.25 is out! And congrats to @cmkelley on being elected to the 1,000 message thread club!

I understand (and support) people's need to redact certain information, it's just that I couldn't tell if stuff was redacted or not there. And the parenthesis rather than square brackets for the pid threw me as well, but I'm pretty sure I understand it now, I just can't do anything to make it better without another router that I can use to set up as an Aimesh node.

So, syslogd and klogd are not the "system logging facility". At 10,000 feet, the system logging facility is an inherent part of the kernel - programs such as ntpd write to the system logging facility, and syslogd, or syslog-ng, or other logging programs read from the system logging facility and sort those messages out to either a single file (syslogd) or have the ability to sort them into multiple files (syslog-ng). I think the kernel uses a different internal logging facility, hence klogd on stock firmware to read those messages, but syslog-ng is able to read the kernel's messages as well through the /proc/kmsg interface. The system and kernel logging facilities act as an interface between the programs/kernel and the program used to process the system logs.

 

[cmkelley]

I don't have AiMesh but in '/opt/var/messages' I can see for a specific MAC

Dec 14 08:43:42 RT-AC68U WLCEVENTD: eth2: Assoc XX:XX:XX:XX:XX:XX

<snip>

Dec 14 12:56:02 RT-AC68U-XXXX syslog: WLCEVENTD wlceventd_proc_event(386): eth2: Deauth_ind XX:XX:XX:XX:XX:XX, status: 0, reason: Unspecified reason (1)
Dec 14 12:56:33 RT-AC68U-XXXX syslog: WLCEVENTD wlceventd_proc_event(420): eth2: Auth XX:XX:XX:XX:XX:XX, status: 0, reason: d11 RC reserved (0)

Clearly '/usr/sbin/wlceventd'? is simply (legitimately) using 'logger -t' to generate the 'wlceventd_proc_event' entries.

NOTE: I can also spoof the same message from the command line:

logger -t syslog "WLCEVENTD wlceventd_proc_event(???): eth9: Auth xx:xx:xx:xx:xx:xx, status: 0, reason: d11 RC reserved (0)"

Guess what, I upgraded to v384.14 around lunchtime today!

Oh, goody, new firmware = new log message formatting. Sigh. I haven't touched 384.14 (and it isn't available for my AC3200 test router), so I wasn't thinking about new messages from the firmware, I thought it was just an Aimesh node thing. Is anything else showing up with a "syslog" process id?

I'd argue changing the wlceventd logging from a standard format to what appears at first blush to be non-standard is not "legitimate", but that's just my opinion.

 

[Swistheater]

Oh, goody, new firmware = new log message formatting. Sigh. I haven't touched 384.14 (and it isn't available for my AC3200 test router), so I wasn't thinking about new messages from the firmware, I thought it was just an Aimesh node thing. Is anything else showing up with a "syslog" process id?

I'd argue changing the wlceventd logging from a standard format to what appears at first blush to be non-standard is not "legitimate", but that's just my opinion.

my system messages are free from the hateful glow of the new messages, i will let you know if i notice anything else different. btw i am on 384.14 as well. I just haven't updated signature yet.

one thing we could add a filter example for would be roamast , but i honestly don't think i get enough of these to really care to have them filtered.

 

[Butterfly Bones]

I'm trying to troubleshoot my Loggly issues, but no luck. I want to try setting up syslog-ng TLS using the Loggly instructions.
https://www.loggly.com/docs/syslog-ng-tls-configuration/

The first line in their code box for the directory to place the certificates is not how the ASUS directory structure is configured. Doing a recursive search, these look like the places to point the Loggly certificate (I'll use only one of course). Comments?

/opt/etc/ssl/certs (symlinked) -> /tmp/mnt/SNB/entware/etc/ssl/certs

/rom/etc/ssl/certs

 

[cmkelley]

I'm trying to troubleshoot my Loggly issues, but no luck. I want to try setting up syslog-ng TLS using the Loggly instructions.
https://www.loggly.com/docs/syslog-ng-tls-configuration/

The first line in their code box for the directory to place the certificates is not how the ASUS directory structure is configured. Doing a recursive search, these look like the places to point the Loggly certificate (I'll use only one of course). Comments?

/opt/etc/ssl/certs (symlinked) -> /tmp/mnt/SNB/entware/etc/ssl/certs

It's late, I may be confused, but that looks circular ... on my system, /opt is symlinked to /tmp/opt/, which is symlinked to /tmp/mnt/[USBDISK]/entware/ ... so the above would be a symlink back to itself?

/rom/etc/ssl/certs

You can't put anything in /rom/... that's the ROM, it's read-only.

 

[Butterfly Bones]

It's late, I may be confused, but that looks circular ... on my system, /opt is symlinked to /tmp/opt/, which is symlinked to /tmp/mnt/[USBDISK]/entware/ ... so the above would be a symlink back to itself?

Yeah, I was on a maniacal hunt last night.

Scrolling back through all three terminal buffers I was using, I cannot find what led me to think and type that.

You can't put anything in /rom/... that's the ROM, it's read-only.

I suspected as such, and did not even look seriously at it.

 

[cmkelley]

scribe v2.4_0 pushed

• Adds check for Skynet version; currently requires 6.9.2 or later
• Removed time_reap(2) from Skynet filter; as a result syslog-ng 3.19 is lowest working version, was 3.23 before Skynet update
• Added an ntpd log handler & logrotate, Entware's ntpd logs to /opt/var/spool/ntp/ntp.log; read that file and ntpd logs in system log and put into /opt/var/log/ntp.log
• Clarified the show config shows the on-disk configuration, not the loaded configuration; added the ability to view the loaded configuration
• Both on-disk and loaded configuration are now output in debugging file
• Removed "update-filters" as an accepted command line option; "filters" still works, but is "undocumented", command line use is intended for use by the update process only
• Added "reload" as a command line option to reload the configuration; previously was available only in the menu
  
• Added log handler & logrotate file for "bcm63xxx" messages
• Added redaction of usb drive names in debugging log; note drive names with a comma in them won't be redacted (I had to pick SOMETHING for a sed delimiter), will print a message informing that drive name wasn't redacted if it has a comma in it
• Updated the wlceventd filter; I used a slightly different method than proposed in this thread, but I haven't updated to Merlin 384.14 yet, so I haven't tested it

Have fun, I hope I didn't break anything.

 

[Adamm]

Added redaction of usb drive names in debugging log; note drive names with a comma in them won't be redacted (I had to pick SOMETHING for a sed delimiter), will print a message informing that drive name wasn't redacted if it has a comma in it

If your looking for a good delimiter, check out ~

 

[cmkelley]

If your looking for a good delimiter, check out ~

Also legal in drive names, AFAICT. I thought comma was less likely to be used in a drive name than tilde. I do already use ~ for a sed delimiter in a couple places in the code.

 

[joe scian]

scribe v2.4_0 pushed

• Adds check for Skynet version; currently requires 6.9.2 or later
• Removed time_reap(2) from Skynet filter; as a result syslog-ng 3.19 is lowest working version, was 3.23 before Skynet update
• Added an ntpd log handler & logrotate, Entware's ntpd logs to /opt/var/spool/ntp/ntp.log; read that file and ntpd logs in system log and put into /opt/var/log/ntp.log
• Clarified the show config shows the on-disk configuration, not the loaded configuration; added the ability to view the loaded configuration
• Both on-disk and loaded configuration are now output in debugging file
• Removed "update-filters" as an accepted command line option; "filters" still works, but is "undocumented", command line use is intended for use by the update process only
• Added "reload" as a command line option to reload the configuration; previously was available only in the menu
  
• Added log handler & logrotate file for "bcm63xxx" messages
• Added redaction of usb drive names in debugging log; note drive names with a comma in them won't be redacted (I had to pick SOMETHING for a sed delimiter), will print a message informing that drive name wasn't redacted if it has a comma in it
• Updated the wlceventd filter; I used a slightly different method than proposed in this thread, but I haven't updated to Merlin 384.14 yet, so I haven't tested it

Have fun, I hope I didn't break anything.

ntp log and v2.4 working perfectly including wlceventd filter under 384.14 on RT-AC5300 thank you