Scribe scribe - syslog-ng and logrotate installer

[cmkelley]

scribe v3.0_0
2023-04-24

scribe is a script that installs syslog-ng and logrotate from Entware. Filters for various common logs are included, and logrotate files to manage those logs.

NOTE WELL:
Looking at log files to "see if anything is wrong" is counter-productive. Log files are best used to find the cause of a problem, or gain knowledge of how the system works. Just because a log entry "looks strange" or "unexplained", particularly if a new entry occurs after updating the firmware, does not at all mean that there is something wrong. Log entries occur for many reasons, including but not limited to:

1. Informational only - this is BY FAR the largest number of entries
2. The developers turned on debugging and/or turned up log verbosity for some reason and forgot (or didn't bother) to turn it back off
3. The developer thinks people will "obviously" know that certain log entries are just for informational purposes, no matter how poorly worded the entry is
4. English is not the 1st (or maybe not even 2nd) language for the developer and the wording of the log entry doesn't accurately reflect the developer's intent
5. Many "meaningful" log entries don't actually matter - a prime example of this is the well-known "dcd_tainted" crash an AC86U routers. While it does create a rather large number of log entries, it has no discernible effect at all on the router operation; if the log entries stopped with a firmware upgrade, you would not be able to tell if the developers actually fixed the problem or decided to "fix" it by simply suppressing the log entries
6. ASUS, like most routers, runs a stripped-down version of the Linux kernel, and the router doesn't have certain hardware that a normal PC does, so sometimes it may report not finding services or hardware that were never there in the first place

Again, if you have a noticeable problem, it may be helpful to look in the log files for clues to what is causing it, although sadly, more often then not there are no clues to be found in the logs. Noticing a new or unexplained log entry is not at all indicative of a problem.

Requirements
Asuswrt-Merlin version 380.68 or later running on a supported router with Entware installed. jffs scripts must be enabled, although Entware installation should have taken care of this.

IMPORTANT:
1: If you use Diversion, you must update Diversion to v4.1.0 or later for new installations.
2: If you intend to use Skynet, but have not installed it yet, it is preferred to install Skynet before installing scribe.
3: If you use Skynet, v6.9.2 or later is required.
4: It is not possible to update from scribe v1.0_0 (or earlier) without forcing a reinstall.

Installation
SSH into your router and run the following:

/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/cynicastic/scribe/master/scribe" -o "/jffs/scripts/scribe" && chmod 0755 /jffs/scripts/scribe && /jffs/scripts/scribe install

In addition to syslog-ng and logrotate, scribe also installs syslog-ng and logrotate configuration files to send syslog-ng's own logs, logrotate logs, WLCEVENTD logs, and (hopefully most) crash logs to their own separate files. All log files will be in /opt/var/log.

A single line is added to /opt/etc/init.d/S01syslog-ng that calls a helper script to kill the klogd and syslogd and make the necessary changes to the filesystem to keep syslogd from causing problems when GUI changes cause it to restart.

A single line is added to /jffs/scripts/service-event to call scribe and check for syslogd and klogd being restarted after either the "logger" or "time" services are (re-)started.

If Skynet is installed, a syslog-ng configuration file is also installed for it, and Skynet is automatically pointed at the new location for the iptables output. If you plan to install Skynet, it is highly preferable to install Skynet before scribe.

Options
A customized syslog-ng.conf file will be installed. This syslog-ng.conf file will remove sourcing a large number of functions from the /opt/share/syslog-ng/include/scl directory, change maximum log message size to 16K, and set stats frequency to every 6 hours. If installed, it copies the syslog-ng.conf file that came with the package to /opt/share/syslog-ng/examples.

Example Files
Of note, example files are placed in /opt/share/sylog-ng/examples/ and /opt/share/logrotate/examples/ for common programs (skynet, pixelserv-tls, etc). These can be copied directly to /opt/etc/syslog-ng.d/ and /opt/etc/logrotate.d/ for use. Do not blindly copy all of these files to their respective ".d" directories however, as doing so will prevent syslog-ng from running. Editing files directly here is not recommended as they will likely be overwritten if you choose to update the filters.

IMPORTANT
ALL files in /opt/etc/logrotate.d/ and /opt/etc/syslog-ng.d/ MUST NOT be writable by anyone other than root. Files that are writable by group or others will be ignored!

Usage
scribe has a number of commands:

• install - can be used to force re-installation of syslog-ng, logrotate, and scribe. Doing so will overwrite any customizations you have made to the script files included for syslog-ng and logrotate.
• uninstall (or remove) - removes syslog-ng, logrotate, and scribe from your system. Doing so will erase any files you have added to /opt/etc/syslog-ng.d and /opt/etc/logrotate.d.
• update - updates the scribe script, asks to update filters.
• show-config (or config) - shows the complete combined configuration for syslog-ng including all included files in the system list utility.
• status - checks to see if syslog-ng is running and if logrotate is in the crontab (cru). Also displays the scribe script version. Also checks the syntax of the syslog-ng configuration as it exists on the disk, which may be different than what is loaded if "restart" (see below) has not been run since changing the configuration files.
• reload - reloads syslog-ng, which is necessary when changing configuration files.
• restart - restarts syslog-ng Will start syslog-ng if it is not currently running.
• debug - create a debug file and tar.gz it to allow debug info to be sent easily.

Support
I'll try to answer questions and help troubleshoot. I expect this thread will also include discussions on syslog-ng configuration as well. scribe is hosted on GitHub. Syslog-ng Open Source Edition (OSE) is a product of One Identity (https://www.syslog-ng.com/products/open-source-log-management/)

Credits
I've borrowed quite a bit of code from @thelonelycoder and @Jack Yaz, with some from @Adamm and @Xentrk as well. All the well-written stuff is from them. All the crap bits are mine.

Special Thanks
To @elorimer and @Butterfly Bones for offering up their time and resources for alpha testing, including finding a couple really bone-headed mistakes on my part.

 

[cmkelley]

More ToDo:

• Change some defaults in the to-be-added non-scl syslog-ng.conf done, in the project on GitHub as syslog-ng.conf-small
• Add send HUP to all openvpn clients when rotating logs done, updated project
• Document the existence of /opt/share/syslog-ng/examples and /opt/share/logrotate/examples done on SNB, need to update GitHub README.md
• Figure out an openvpn function for syslog-ng instead of pushing it to its own log in the GUI custom configuration box done, added to /opt/share/syslog-ng/examples
• Add `export TZ=$(cat /etc/TZ)` to rc.func.syslog-ng done, updated project
• Add pixelserv-tls function done, added to /opt/share/syslog-ng/examples
• Create logrotate file for pixelserv-tls logs done, updated project
• Add debugging fiile creation done, updated project
• Ask to install syslog-ng.conf-small as syslog-ng.conf done, updated project
• Add logrotate files for pixelserv and openvpn to examples done, updated project
• Add redirect in cru for logrotate output, probably /opt/tmp/logrotate.daily done, updated project
• Add syslog-ng filter to grab /var/lib/logrotate.status and /opt/tmp/logrotate.daily to create logrotate.log done, updated project
• Add logrotate filter for logrotate.log done, updated project
• Check /opt/etc/init.d/S01syslog-ng for the hook to rc.func.syslog-ng - updating syslog-ng may overwrite S01syslog-ng done, updated project
• Have a think about running logrotate right after install. Probably a good idea, can expand in 1.0 to include error checking. done, updated project

ToDo after 1.0 is released:

• More graceful way to update configurations in .d directories without forcing reinstall of syslog-ng and/or logrotate work in process, currently adds new, doesn't update existing in case of editing by user done, updated project
• Ask about installing filters & logrotate files from /opt/share/*/examples files
• Handle forced reinstall better, wrt syslog-ng.conf file and others OBE, forced re-install should produce a "newly installed" system + any filters added by user
• Think about md5 comparison for upgrades done, present in v1.0_0
• Add ability to check syslog-ng with "syslog-ng -Fevd" w/reminder if syslog-ng fails to start done, updated project
• Add ability to run logrotate manually to check for errors - get conf file from cru l

2.0 will include a menu system done

Putting this here to include somewhere as a useful reference: https://github.com/balabit/syslog-ng-3.5/blob/master/contrib/syslog-ng.conf.doc

 

[Butterfly Bones]

Ok, believe it or not, I have read everything on Github, and think I understand. First, I want to make a backup and have done that in the AC86U admin section for the router and the jffs partition. I also make regular backups of the USB stick files and the jffs directory using an FTP client to my Linux desktop. Anything else that might be altered that I missed before I begin this alpha adventure?

Things have changed since I tried to use this a year or so ago, or I was using a much simpler method of setting it up using info from kvic and tomsk. I went back and looked at my saved files and notes. Mostly this is my lack of grasping scripting. I was too busy dissecting critters in lab when computers came around and people were learning programming.

I see that in the logrotate.share you mention the openvpn.log. You plan to use the method you mentioned in the Stubby thread about adding this line to the VPN configuration, correct? This message -
https://www.snbforums.com/threads/stubby-installer-asuswrt-merlin.49469/page-49#post-465173

log-append /opt/var/log/openvpn.log

Looks like I need to add it to my OpenVPN client as per that message in the Stubby thread? Anything I need to do to handle that log or does the install do the openvpn log?

I run an openvpn client and a IPSec server that has no configuration other than username, password, and shared key. It is used very rarely and only to allow me access to my router when away. It only generates one short line at connect and one at disconnect, so no need to worry about that.

The ChkWAN and VPN_Failover scripts should not be affected from what I can see, since they only write a few lines to the syslog each hour as the run, and I am fine with leaving them along for now. I'll see about moving them to their own log when I'm comfortable with syslog-ng.

Thank you, and I must say that fact that you do this in vim is mind boggling. I can handle simple config files only.

 

[cmkelley]

Ok, believe it or not, I have read everything on Github, and think I understand. First, I want to make a backup and have done that in the AC86U admin section for the router and the jffs partition. I also make regular backups of the USB stick files and the jffs directory using an FTP client to my Linux desktop. Anything else that might be altered that I missed before I begin this alpha adventure?

Things have changed since I tried to use this a year or so ago, or I was using a much simpler method of setting it up using info from kvic and tomsk. I went back and looked at my saved files and notes. Mostly this is my lack of grasping scripting. I was too busy dissecting critters in lab when computers came around and people were learning programming.

I see that in the logrotate.share you mention the openvpn.log. You plan to use the method you mentioned in the Stubby thread about adding this line to the VPN configuration, correct? This message -
https://www.snbforums.com/threads/stubby-installer-asuswrt-merlin.49469/page-49#post-465173

log-append /opt/var/log/openvpn.log

Looks like I need to add it to my OpenVPN client as per that message in the Stubby thread? Anything I need to do to handle that log or does the install do the openvpn log?

I run an openvpn client and a IPSec server that has no configuration other than username, password, and shared key. It is used very rarely and only to allow me access to my router when away. It only generates one short line at connect and one at disconnect, so no need to worry about that.

The ChkWAN and VPN_Failover scripts should not be affected from what I can see, since they only write a few lines to the syslog each hour as the run, and I am fine with leaving them along for now. I'll see about moving them to their own log when I'm comfortable with syslog-ng.

Thank you, and I must say that fact that you do this in vim is mind boggling. I can handle simple config files only.

Oh yeah, I forgot about the conversations about OpenVPN ... I should add a configuration file for VPN logs, rather than force everyone to set the log in the GUI.

For the time being, while I run a VPN server, not a client, I would think the command would be the same. In the "custom configuration" box for the client or server, enter:

log-append /opt/var/log/openvpn.log

EDIT: looking at my openvpn script for logrotate, it only sends a HUP to openvpn servers. When the client is running, is there a process like 'vpnclient1' or something similar running?

This will bypass the syslog-ng server and write directly to the logfile. My install doesn't touch the VPN configuration files, and I'd like to avoid that complexity.

I don't see any need to backup more than jffs and your USB. The install is fairly benign, it's just if/when you uninstall it you need to understand what customizations it will wipe out.

 

[Butterfly Bones]

Oh yeah, I forgot about the conversations about OpenVPN ... I should add a configuration file for VPN logs, rather than force everyone to set the log in the GUI.

For the time being, while I run a VPN server, not a client, I would think the command would be the same. In the "custom configuration" box for the client or server, enter:

log-append /opt/var/log/openvpn.log

This will bypass the syslog-ng server and write directly to the logfile. My install doesn't touch the VPN configuration files, and I'd like to avoid that complexity.

I don't see any need to backup more than jffs and your USB. The install is fairly benign, it's just if/when you uninstall it you need to understand what customizations it will wipe out.

Ok, I think I am ready then. I agree with not touching the openvpn config, let the user do that if they want with the line added to the custom configuration box.

 

[cmkelley]

Ok, I think I am ready then. I agree with not touching the openvpn config, let the user do that if they want with the line added to the custom configuration box.

See my update above, I think logrotate will not work properly for vpn clients without some additional lines.

 

[Butterfly Bones]

See my update above, I think logrotate will not work properly for vpn clients without some additional lines.

Running "ps | grep openvpn" shows this -

14323 (username)  6856 S    /etc/openvpn/vpnclient1 --cd /etc/openvpn/client1 --config config.ovpn

Is that what you seek?

 

[cmkelley]

Yep. For now just change "vpnserver1" to "vpnclient1" in the /opt/bin/logrotate/openvpn file. I'm not really sure what will happen the the vpnclient gets the HUP signal, it "should" just restart it (the VPN connection will go down and have to re-establish itself), but I have openvpn set to only rotate the logs monthly, so this should only be a monthly occurrence. I'll amend the openvpn logrotate script to restart all servers and clients.

EDIT: oops, forgot I stashed the openvpn script in the /opt/share/logrotate/examples folder. You'll need to copy /opt/share/logrotate/examples/openvpn to /opt/etc/logrotate.d/ to rotate the openvpn logs. My documentation is in a worse state than the code. :-(

Sorry for the delay, wife decided to drag me to the store. And now it's past my bedtime. Hope to have some time tomorrow night to hack on this, but probably not during the day. I'm not a coder by trade.

 

[Butterfly Bones]

Yep. For now just change "vpnserver1" to "vpnclient1" in the /opt/bin/logrotate/openvpn file. I'm not really sure what will happen the the vpnclient gets the HUP signal, it "should" just restart it (the VPN connection will go down and have to re-establish itself), but I have openvpn set to only rotate the logs monthly, so this should only be a monthly occurrence. I'll amend the openvpn logrotate script to restart all servers and clients.

Sorry for the delay, wife decided to drag me to the store. And now it's past my bedtime. Hope to have some time tomorrow night to hack on this, but probably not during the day. I'm not a coder by trade.

Thanks, I did run the install, all seemed good, but all syslog entries stopped, and I know there should be a symlink to copy back the lines that are not moved to separate logs by syslog-ng. A few scripts from martineau, ChkWAN.sh and VPN_Failover.sh did not show as running except in "cru l".

So I removed it for now. I think my issue is how to monitor those new logs. I tried multiple terminals using "tail -f xxxxxxx.log" and that did not seem to work here, though I could see the skynet, openvpn and crash logs as well as the messages.

Like you, too late to troubleshoot tonight. Still need to find a good way to monitor logs for awhile until I know all is running well.

 

[elorimer]

I think my issue is how to monitor those new logs

I use winscp to open them when I care to look, and the timestamp in the directory tells me whether they are working. My messages file is setup as fallback, so it is scraped of the special logs and then the gui works fine for looking at all the rest.

 

[Butterfly Bones]

I use winscp to open them when I care to look, and the timestamp in the directory tells me whether they are working. My messages file is setup as fallback, so it is scraped of the special logs and then the gui works fine for looking at all the rest.

Thanks, but I have been running Linux since 1995. I have an old laptop with Win 7 used to update some device software that only works in the Win. Otherwise it is in the cabinet for months between uses. I just need to find something that lets me SSH in and display the logs. Running "tail -f device.log" seems to work but stops updating except for Skynet. When I cat the same log it has more entries. I just need to track down why.

 

[Butterfly Bones]

Ok, I solved my terminal issue that was not really a terminal issue at all.
Turns out is it tail on the router just stops after a short time. All I need it so use -F (force retry) instead of lower case -f.

tail -F /opt/var/log/(logname).log

Now before I dive in again, I want to handle the lines from the VPN_Failover.sh script that I use from martineau. Here are the three lines it produces every time it runs. My VPN keeps dropping to I want to run this every 10 minutes at maximum. That gets messy fast.

Apr  1 16:43:00 (VPN_Failover.sh): 31536 VPN Client Monitor: Checking VPN Client 1 connection status....
Apr  1 16:43:00 (VPN_Failover.sh): 31536 VPN Client Monitor: VPN Client 1 status OK
Apr  1 16:43:00 (VPN_Failover.sh): 31536 VPN Client Monitor: Monitoring VPN Client 1 terminated

And here is the vpnfailover script that I *think* (hope?) is ok, based on the skynet, crash, and wcleventd from cmkelley GitHub. I want to strip these from syslog into their own vpnfailover.log.

# put VPN_Failover VPN Client Monitor: messages into /opt/var/log/vpnfailover.log
destination d_vpnfailover {
    file("/opt/var/log/vpnfailover.log");
};
filter f_vpn_failover {
    program("VPN_Failover")
and
    message("VPN Client");
};
log {
    source(src);
    source(kernel);
    filter(f_vpn_failover);
    destination(d_vpnfailover);
    flags(final);
};
#eof

Am I close or way off the mark?

Thank you!

 

[Martineau]

Ok, I solved my terminal issue that was not really a terminal issue at all.
Turns out is it tail on the router just stops after a short time. All I need it so use -F (force retry) instead of lower case -f.

tail -F /opt/var/log/(logname).log

Now before I dive in again, I want to handle the lines from the VPN_Failover.sh script that I use from martineau. Here are the three lines it produces every time it runs. My VPN keeps dropping to I want to run this every 10 minutes at maximum. That gets messy fast.

Apr  1 16:43:00 (VPN_Failover.sh): 31536 VPN Client Monitor: Checking VPN Client 1 connection status....
Apr  1 16:43:00 (VPN_Failover.sh): 31536 VPN Client Monitor: VPN Client 1 status OK
Apr  1 16:43:00 (VPN_Failover.sh): 31536 VPN Client Monitor: Monitoring VPN Client 1 terminated

And here is the vpnfailover script that I *think* (hope?) is ok, based on the skynet, crash, and wcleventd from cmkelley GitHub. I want to strip these from syslog into their own vpnfailover.log.

# put VPN_Failover VPN Client Monitor: messages into /opt/var/log/vpnfailover.log
destination d_vpnfailover {
    file("/opt/var/log/vpnfailover.log");
};
filter f_vpn_failover {
    program("VPN_Failover")
and
    message("VPN Client");
};
log {
    source(src);
    source(kernel);
    filter(f_vpn_failover);
    destination(d_vpnfailover);
    flags(final);
};
#eof

Am I close or way off the mark?

Thank you!

I use the following condensed format for ChkWAN

destination d_chkwan      { file("/opt/var/log/chkwan.log")            ; };
filter f_chkwan           { match("ChkWAN")                            ; };
log { source(src); filter(f_chkwan)      ; destination(d_chkwan)       ; };

So I don't see why replacing "ChkWAN" with "VPN_Failover" (and chkwan to say vpnfailover) wouldn't work.
I assume they are compatible with the latest version of syslog-ng installed by the scribe script.

EDIT: This works, although I don't use the scribe script.

destination d_vpnfailover { file("/opt/var/log/vpnfailover.log")       ; };
filter f_vpnfailover      { match("VPN_Failover")                      ; };
log { source(src); filter(f_vpnfailover) ; destination(d_vpnfailover)  ; };

 

[Butterfly Bones]

I use the following condensed format for ChkWAN

destination d_chkwan      { file("/opt/var/log/chkwan.log")            ; };
filter f_chkwan           { match("ChkWAN")                            ; };
log { source(src); filter(f_chkwan)      ; destination(d_chkwan)       ; };

So I don't see why replacing "ChkWAN" with "VPN Client Monitor" (and chkwan to say vpnfailover) wouldn't work.
I assume they are compatible with the latest version of syslog-ng installed by the scribe script.

Thank you, that is the format that I am more familiar with from trying a year ago to use syslog-ng from the original tomsk and kvic thread. I still have the original syslog-ng config files and the changed files I was working on with kvic help along with notes.

I know much has changed with syslog-ng reading that thread and seeing the setup cmkelley has done. I also understand that there is more than one way to do what one wants / needs. I just have trouble wrapping my head around scripting, all those symbols ( like } ; ( " ) flip my brain around, I think I'm too old or use the wrong brain hemisphere....

 

[Martineau]

Thank you, that is the format that I am more familiar with from trying a year ago to use syslog-ng from the original tomsk and kvic thread. I still have the original syslog-ng config files and the changed files I was working on with kvic help along with notes.

I know much has changed with syslog-ng reading that thread and seeing the setup cmkelley has done. I also understand that there is more than one way to do what one wants / needs. I just have trouble wrapping my head around scripting, all those symbols ( like } ; ( " ) flip my brain around, I think I'm too old or use the wrong brain hemisphere....

Ha!... but you don't suffer the embarrassment/humiliation by publicly posting manky scripts with the '};[!=]' in the wrong order!

P.S. Apologies, I can't remember how verbose/chatty my scripts are, or even if they have a 'silent' directive, but IMHO it is better to have logged messages to Syslog 'just-in-case', and hopefully with @cmkelley's scribe script, more users will now use syslog-ng.

 

[elorimer]

Apr  1 16:43:00 (VPN_Failover.sh): 31536 VPN Client Monitor: Checking VPN Client 1 connection status....

filter f_vpn_failover { program("VPN_Failover") and message("VPN Client"); };

Not sure about that part. The logged message looks odd to me, because there is no hostname.

If I follow the way syslog-ng parses this, the date stamp and had there been a hostname, the hostname is the HEADER part of the message.
All the rest is MSG part, divided into the name of the program and the message text, divided by the ":" Here, the name of the program looks like (VPN_Failover.sh). I believe program() returns the program name, so I think you may need "(VPN_Failover.sh)", and maybe some escape characters. But I think the message part of the filter would work. match() is deprecated now.

 

[Butterfly Bones]

Ha!... but you don't suffer the embarrassment/humiliation by publicly posting manky scripts with the '};[!=]' in the wrong order!

P.S. Apologies, I can't remember how verbose/chatty my scripts are, or even if they have a 'silent' directive, but IMHO it is better to have logged messages to Syslog 'just-in-case', and hopefully with @cmkelley's scribe script, more users will now use syslog-ng.

Ha, I could never get close to correct syntax with my mental block, I'd get chased out of wherever I posted them.

I agree with the more verbose output to review at ones leisure. I have the "quiet" tag in ChkWAN.sh and the "silent" tag in VPN_Failover.sh, though I might take them out withe syslog-ng moving them to their own logs. I also want to turn up the pixelsrv-tls logs to track down some weird things with my mobiles.

 

[Martineau]

I agree with the more verbose output to review at ones leisure. I have the "quiet" tag in ChkWAN.sh and the "silent" tag in VPN_Failover.sh

Consistency eh? ….. in my scripts; 'quiet' in one 'silent' in another

 

[elorimer]

I wanted to note two things.

First, I think @cmkelley is doing something smart here, by putting each destination, filter, log triplet in its own included file. That means segregating out something to its separate log, or not, is as simple as adding or deleting the relevant file.

Second, the processing style is using one of two alternatives with the "final" flag rather than the "fallback" flag. Using the final flag, if a message meets the filter, it is logged separately and then further handling of the message stops. If the final flag is omitted, the message is marked as "processed" and continues to the next filter. At the end, when you get to the /var/log/messages handler, if it has the fallback flag, it will handle everything that didn't get processed. In these examples it amounts to the same thing, but it is harder to put one message into two destinations. The manual notes another problem but I don't think it will come up in this case.

 

[Butterfly Bones]

Consistency eh? ….. in my scripts; 'quiet' in one 'silent' in another

But they do not do the same thing, as far as I can see and scanning through the script, however it is likely I am dead wrong.

In VPN_Failover the silent refers to "# Console Audible ERRORS may be suppressed by using 'silent' directive". I get three syslog lines whether I use it or not.

In ChkWAN the quiet is used in so many places I cannot grasp exactly what it does, except I only see one line in the syslog.