Scribe scribe - syslog-ng and logrotate installer

[elorimer]

There is a thing here to watch out for--I fell into it today. It is possible for the /tmp/syslog.log file to be recursive. When syslogd is killed, the next step is to cat /tmp/syslog.log to messages. But if /tmp/syslog.log is a symlink to messages, then it goes around and around. I hit 400 meg before I saw what happened.

 

[L&LD]

I'll be back in soon. I did not have time to let the router "settle" as I needed to respond to some urgent email requests for GPS data, watch games (I only stream TV), and I knew that I could do a full restore, since I do it fairly often and make one frequently and always before trying something beta like this.

Reading the previous messages, I see some fixes, and I want to contribute to the process. I'm reading thorough the recent commits on @cmkelley git to increase my confusion level this evening.

After dinner, I'll be jumping head first into the deep end.

Great! That means I have to follow you now!

 

[cmkelley]

There is a thing here to watch out for--I fell into it today. It is possible for the /tmp/syslog.log file to be recursive. When syslogd is killed, the next step is to cat /tmp/syslog.log to messages. But if /tmp/syslog.log is a symlink to messages, then it goes around and around. I hit 400 meg before I saw what happened.

Errr, I'm not at all clear how that's happening. The relevant bit in rc.func.syslog-ng is:

# webGUI System Log = /tmp/syslog.log
if [ ! -L "/tmp/syslog.log" ]; then
    cat /tmp/syslog.log >> /opt/var/log/messages
    rm -f /tmp/syslog.log /tmp/syslog.log-1
    ln -s /opt/var/log/messages /tmp/syslog.log
fi

"if [ ! -L "/tmp/syslog.log" ]" is literally, "if /tmp/syslog.log is not a symbolic link". I seriously have no clue how that function could return true (and execute the "then" commands) if /tmp/syslog.log were linked to anything.

 

[cmkelley]

I did and got bit. I had to do a full restore of my AC86U after it lost Internet access and refused SSH access.

--- snip, snip ---

In about two more minutes as I looked at Skynet (log direction not pointed to syslog-ng only default) the webGUI got unresponsive and the Chrome window with the router just spun, could not move forward or back. I was able to get in with SSH ok, but none of the menu items in AMTM would respond, so I did
"system reboot".

After reboot I got into the webGUI but it was very unresponsive and very sluggish, taking 30-60 seconds to get to the next menu choice. I used the SSH to reboot again. Same thing, webGUI took long time to respond, so with SSH I did "scribe uninstall".

Next time the log in screen came up, I logged in and webGUI was worse, taking near 60 seconds to menu items, so I went to Admin and did a reset and reboot. When it came back up I restored the cfg file I made before I started this shenanigan.

If this happens again, could you run top (or htop if you have it installed)?

 

[Butterfly Bones]

Okayyyy,
Installed again.
No syslog in webGUI - last two lines (all normal up to here) then no more entries

Apr  5 19:41:37 syslogd exiting
Apr  5 19:41:37 RT-AC86U-4608 syslog-ng[18432]: syslog-ng starting up; version='3.19.1'

Went to Skynet to try and follow debug events

[i] Watching Logs For Debug Entries (ctrl +c) To Stop
tail: can't open '/opt/var/log/skynet-0.log': No such file or directory

Checked the /opt/entware/etc/syslog-ng.d/ directory
skynet is a directory there

ls -al /opt/enware/etc/syslog-ng.d/
drwxr-xr-x    3 [redacted] root          4096 Apr  5 19:49 .
drwxr-xr-x   11 [redacted] root          4096 Apr  5 19:41 ..
-rw-r--r--    1 [redacted] root             0 Mar 20 00:45 .keep
-rw-rw-rw-    1 [redacted] root           574 Apr  5 19:41 crashes
drwxrwxrwx    2 [redacted] root          4096 Apr  5 19:41 skynet
-rw-rw-rw-    1 [redacted] root           319 Apr  5 19:41 syslogng
-rw-rw-rw-    1 [redacted] root           328 Apr  5 19:41 wlceventd

Uninstalled.

Sorry I just saw your last message and keep forgetting to run scribe debug....

EDIT - yes I have htop

 

[cmkelley]

Okayyyy,
Installed again.
No syslog in webGUI - last two lines (all normal up to here) then no more entries

Apr  5 19:41:37 syslogd exiting
Apr  5 19:41:37 RT-AC86U-4608 syslog-ng[18432]: syslog-ng starting up; version='3.19.1'

Went to Skynet to try and follow debug events

[i] Watching Logs For Debug Entries (ctrl +c) To Stop
tail: can't open '/opt/var/log/skynet-0.log': No such file or directory

Checked the /opt/entware/etc/syslog-ng.d/ directory
skynet is a directory there

ls -al /opt/enware/etc/syslog-ng.d/
drwxr-xr-x    3 [redacted] root          4096 Apr  5 19:49 .
drwxr-xr-x   11 [redacted] root          4096 Apr  5 19:41 ..
-rw-r--r--    1 [redacted] root             0 Mar 20 00:45 .keep
-rw-rw-rw-    1 [redacted] root           574 Apr  5 19:41 crashes
drwxrwxrwx    2 [redacted] root          4096 Apr  5 19:41 skynet
-rw-rw-rw-    1 [redacted] root           319 Apr  5 19:41 syslogng
-rw-rw-rw-    1 [redacted] root           328 Apr  5 19:41 wlceventd

Uninstalled.

Sorry I just saw your last message and keep forgetting to run scribe debug....

EDIT - yes I have htop

I'm adding output of the first few lines of top to my debug routine. Lemme have a think on what could be happening here before I push the next version.

 

[cmkelley]

So, there's a reason for the warning on my GitHub page. To avoid familial disruption, I was only testing scribe on my test router, which didn't have Skynet installed. Yes, that was stoooopid. Installing when skynet is present is b0rken.

Please stand by ...

EDIT: okay, not quite so bad as I thought, I actually only broke it when I changed the location of the skynet filter for syslog-ng. Still, should have tested it.

 

[cmkelley]

v0.6_3 pushed.

* fix error copying skynet filter when skynet is installed
* add top to debug

Sorry guys, this one requires remove & re-install for sure. The first error was a doosey.

 

[cmkelley]

v0.6_4 pushed.

• Tests for ASUSWRT-Merlin as well as Entware
• Moved some code around for more logical (to me) order
• Added question to install smaller syslog-ng.conf file
• Respects existing logrotate cru
• Scrub router username from debug output

I'm feeling that this is Beta quality code, or nearly so, at this point. I don't think I'm going to add anymore features before 1.x, adding the question for installing the smaller syslog-ng.conf file was the last thing on my list.

 

[cmkelley]

v0.6_5 pushed

Mostly cosmetic, but now copies both the package syslog-ng.conf and the small version to /opt/share/syslog-ng/examples.
It's 3am here, I'm out for a while...

 

[elorimer]

6.5 is working perfectly for me. I uninstalled it, reinstalled it with "small", working fine, rebooted, working fine. Uninstalled again and repeated. Status, debug, check-syntax, all good. So yes, beta.

Things to consider:

1. line 390, "redacted" instead of "redcated"
2 Because it is so prevalent, you might offer in installation to create a pixelserv-tls log/logrotate using your examples
3. Same for openvpn. I don't use it--I have my own openvpn1 & 2 server files.
4. In examples, change pixelserv to pixelserv-tls. (consistency)
5. Either as part of uninstall, or part of install, offer to move an existing messages to a date+timestamped messages file and start fresh.
6. As part of uninstall, move any syslog-ng.d and logrotate.d file you aren't managing to /opt/etc/scribe-backup/syslog-ng.d and /opt/etc/scribe-backup/logrotate.d, and as part of install, offer to move them back.

Curious why your kill_all function uses a ! "X" formulation, as opposed to the ! -z pidof formula.

 

[cmkelley]

6.5 is working perfectly for me. I uninstalled it, reinstalled it with "small", working fine, rebooted, working fine. Uninstalled again and repeated. Status, debug, check-syntax, all good. So yes, beta.

Things to consider:

1. line 390, "redacted" instead of "redcated"
2 Because it is so prevalent, you might offer in installation to create a pixelserv-tls log/logrotate using your examples
3. Same for openvpn. I don't use it--I have my own openvpn1 & 2 server files.
4. In examples, change pixelserv to pixelserv-tls. (consistency)
5. Either as part of uninstall, or part of install, offer to move an existing messages to a date+timestamped messages file and start fresh.
6. As part of uninstall, move any syslog-ng.d and logrotate.d file you aren't managing to /opt/etc/scribe-backup/syslog-ng.d and /opt/etc/scribe-backup/logrotate.d, and as part of install, offer to move them back.

Curious why your kill_all function uses a ! "X" formulation, as opposed to the ! -z pidof formula.

Super, thank you for your patience in seeing the ugly side of development. I don't know how the other folks get their stuff so polished before unleashing it.

1. Fixed.
2. Noted, will add in 1.x cycle. I haven't go around to it simply because the log grows VERY slowly.
3. Ditto.
4. Probably not. Even here in the forums "pixelserv" is used about as often as "pixelserv-tls"
5. Maybe as part of 2.x. The idea was for this to be a set-and-forget installer. It's doable, but seems like it might be a bit of work.
6. Ditto. This would be a wholesale backup though, I wouldn't try to figure out what's "mine" and what's someone else's.

I dunno. Probably because that's one of the first parts I coded. This has been quite a learning experience for me. I don't think any one way is "better". Today, I'd probably write that as:

[ -n "$(pidof syslogd)" ] && killall syslogd

Darn it, now I'm probably going to have to re-write that!

 

[elorimer]

2. The pixelserv-tls boffins have cooked up a script that does a daily (weekly, whatever) report identifying which hosts are trying to go to which blocked site. Using it requires setting the log to 2, and then it grows pretty fast. Left in messages it is distracting. It's one of the bigger reasons I went the syslog-ng route.
3. Yes, and I actually only pay attention to the time stamp on those logs to check activity.

Looking forward now to seeing how @Butterfly Bones does. Then @L&LD is going to jump in, and then this should head to amtm. Nice job!

 

[cmkelley]

2. The pixelserv-tls boffins have cooked up a script that does a daily (weekly, whatever) report identifying which hosts are trying to go to which blocked site. Using it requires setting the log to 2, and then it grows pretty fast. Left in messages it is distracting. It's one of the bigger reasons I went the syslog-ng route.
3. Yes, and I actually only pay attention to the time stamp on those logs to check activity.

Looking forward now to seeing how @Butterfly Bones does. Then @L&LD is going to jump in, and then this should head to amtm. Nice job!

The menu interface required for amtm is a ways off. Note my user icon, my bicycle has been sorely neglected while I've worked on this.

 

[Butterfly Bones]

The menu interface required for amtm is a ways off. Note my user icon, my bicycle has been sorely neglected while I've worked on this.

Yes, mine too. Too nice, I'm going for a ride today!

How the @#$%& does one attach a debug log tar.gz to a PM?

EDIT - I figured it out, I have to upload to a site and use the share URL

 

[cmkelley]

v0.7.0 (Beta) pushed.

Thanks to a lot of patience from elorimer and Butterfly Bones, scribe is now in beta testing. The first post in this thread gives some general information, I know I need to update and expand both it and the GitHub page.

The major motivation for creating this was dealing with the large amount of log entries from the dcd crashes and the iptables logs used by Skynet.

Remember, the good bits of code I stole from others, all the crap stuff is mine.

 

[Butterfly Bones]

Ok, I have a working syslog-ng and logrotate installation! @L&LD @skeal - TAG you're it!

My last glitch (that was tripping me up) is my ethernet filter.

Here are sample entries of what I see in the log. These are Sony smart TV and Chromecast or a computer reboot on the network.

Apr  5 22:15:26 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) Link DOWN.
Apr  5 22:15:26 kernel: br0: port 4(eth4) entered disabled state
Apr  5 22:15:29 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) Link UP 10 mbps full duplex
Apr  5 22:15:29 kernel: br0: port 4(eth4) entered listening state
Apr  5 22:15:29 kernel: br0: port 4(eth4) entered listening state
Apr  5 22:15:31 kernel: br0: port 4(eth4) entered learning state
Apr  5 22:15:32 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) Link DOWN.
Apr  5 22:15:32 kernel: br0: port 4(eth4) entered disabled state
Apr  5 22:15:35 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) Link UP 1000 mbps full duplex
Apr  5 22:15:35 kernel: br0: port 4(eth4) entered listening state
Apr  5 22:15:35 kernel: br0: port 4(eth4) entered listening state
Apr  5 22:15:37 kernel: br0: port 4(eth4) entered learning state
Apr  5 22:15:39 kernel: br0: topology change detected, propagating
Apr  5 22:15:39 kernel: br0: port 4(eth4) entered forwarding state

Here is the filter I made, using a similar one from cmkelley Git. I used the openvpn, though I checked pixelserv and skynet as well.
https://github.com/cynicastic/scribe/tree/master/syslog-ng.share

Here is my filter that kills syslog-ng

# log ethernet change to /opt/var/log/ethernet.log only
destination d_ethernt {
    file("/opt/var/log/ethernet.log");
};
filter f_ethernet {
    message("eth1") or
    message("eth2") or
    message("eth3") or
    message("eth4") or
    message("eth5") or
    message("eth6") or
    message("eth7") or
    message("eth8") or
    message("br0:");
};
log {
    source(src);
    source(kernel);
    filter(f_ethernet);
    destination(d_ethernet);
    flags(final);
};
#eof

Seeking more eyeballs to find my fubar.

EDIT - Ok, @cmkelley found it. One of those duh, oh chilins moments.

# log ethernet change to /opt/var/log/ethernet.log only
destination d_ethernt {             <<======= ethernet NOT ethernt

 

[skeal]

Ok, I have a working syslog-ng and logrotate installation! @L&LD @skeal - TAG you're it!

My last glitch (that was tripping me up) is my ethernet filter.

Here are sample entries of what I see in the log. These are Sony smart TV and Chromecast or a computer reboot on the network.

Apr  5 22:15:26 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) Link DOWN.
Apr  5 22:15:26 kernel: br0: port 4(eth4) entered disabled state
Apr  5 22:15:29 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) Link UP 10 mbps full duplex
Apr  5 22:15:29 kernel: br0: port 4(eth4) entered listening state
Apr  5 22:15:29 kernel: br0: port 4(eth4) entered listening state
Apr  5 22:15:31 kernel: br0: port 4(eth4) entered learning state
Apr  5 22:15:32 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) Link DOWN.
Apr  5 22:15:32 kernel: br0: port 4(eth4) entered disabled state
Apr  5 22:15:35 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) Link UP 1000 mbps full duplex
Apr  5 22:15:35 kernel: br0: port 4(eth4) entered listening state
Apr  5 22:15:35 kernel: br0: port 4(eth4) entered listening state
Apr  5 22:15:37 kernel: br0: port 4(eth4) entered learning state
Apr  5 22:15:39 kernel: br0: topology change detected, propagating
Apr  5 22:15:39 kernel: br0: port 4(eth4) entered forwarding state

Here is the filter I made, using a similar one from cmkelley Git. I used the openvpn, though I checked pixelserv and skynet as well.
https://github.com/cynicastic/scribe/tree/master/syslog-ng.share

Here is my filter that kills syslog-ng

# log ethernet change to /opt/var/log/ethernet.log only
destination d_ethernt {
    file("/opt/var/log/ethernet.log");
};
filter f_ethernet {
    message("eth1") or
    message("eth2") or
    message("eth3") or
    message("eth4") or
    message("eth5") or
    message("eth6") or
    message("eth7") or
    message("eth8") or
    message("br0:");
};
log {
    source(src);
    source(kernel);
    filter(f_ethernet);
    destination(d_ethernet);
    flags(final);
};
#eof

Seeking more eyeballs to find my fubar.

Line 2
destination d_ethernt {
should be
destination d_ethernet {

 

[Butterfly Bones]

Line 2
destination d_ethernt {
should be
destination d_ethernet {

Yes, cmkelley got me on that in a DM we were troubleshooting my bumblings.

I've copied it into the directory and restarted and it work!

 

[cmkelley]

Yes, cmkelley got me on that in a DM we were troubleshooting my bumblings.

I've copied it into the directory and restarted and it work!

You'd think the syntax checker would be better than that. It should find a reference to a non-existent filter or destination. Sounds like a separate script though; build an array of destination(d_dest) statements, and make sure there is a destination d_dest for every one. Same I suppose for filters and sources. Beyond my abilities, but given enough time I could figure it out.