Scribe scribe 3.x_y - syslog-ng and logrotate installer

[cmkelley]

scribe v3.2_0
2023-06-11

scribe is a script that installs syslog-ng and logrotate from Entware. Filters for various common logs are included, and logrotate files to manage those logs.

NOTE WELL:
Looking at log files to "see if anything is wrong" is counter-productive. Log files are best used to find the cause of a problem, or gain knowledge of how the system works. Just because a log entry "looks strange" or "unexplained", particularly if a new entry occurs after updating the firmware, does not at all mean that there is something wrong. Log entries occur for many reasons, including but not limited to:

1. Informational only - this is BY FAR the largest number of entries
2. The developers turned on debugging and/or turned up log verbosity for some reason and forgot (or didn't bother) to turn it back off
3. The developer thinks people will "obviously" know that certain log entries are just for informational purposes, no matter how poorly worded the entry is
4. English is not the 1st (or maybe not even 2nd) language for the developer and the wording of the log entry doesn't accurately reflect the developer's intent
5. Many "meaningful" log entries don't actually matter - a prime example of this is the well-known "dcd_tainted" crash. While it does create a rather large number of log entries, it has no discernible effect at all on the router operation; if the log entries stopped with a firmware upgrade, you would not be able to tell if the developers actually fixed the problem or decided to "fix" it by simply suppressing the log entries
6. ASUS routers, like most routers, use a stripped-down version of the Linux kernel, and since the router doesn't have certain hardware that a normal PC does, sometimes the kernel may report not finding services or hardware that were never there in the first place

Again, if you have a noticeable problem, it might be helpful to look in the log files for clues to what is causing it, although sadly, more often then not there are no clues to be found in the logs. Noticing a new or unexplained log entry is not at all indicative of a problem.

Requirements
Any version of Asuswrt-Merlin with AMTM built in, running on a supported router with Entware installed. jffs scripts must be enabled, although Entware installation should have taken care of this.

IMPORTANT:
1: If you use Diversion, you must update Diversion to v4.1.0 or later for new installations.
2: If you intend to use Skynet, but have not installed it yet, it is preferred to install Skynet before installing scribe.
3: If you use Skynet, v6.9.2 or later is required.
4: It is not possible to update from scribe v1.0_0 (or earlier) without forcing a reinstall.

A note on flash drives:
PLEASE do not use scribe if you are running Entware from a flash drive. Flash drives are not built to take the high number of write cycles that syslog-ng will put on them. Additionally, many flash drives run "hot" to begin with, and when plugged into a router, they soak up some heat from the router as well. If you are using a flash drive, expect to have to replace the flash drive periodically. Your best option is an SSD drive in an external enclosure. Even a SATA III SSD can saturate a 5 GBps USB connection (USB 3.0 aka USB 3.1 Gen 1 aka USB 3.2 Gen 1 aka USB 3.2 Gen 1x1 - that's right, four different names for the same speed), any speed difference will be in the controller.

Installation:
SSH into your router and run the following:

/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/cynicastic/scribe/master/scribe" -o "/jffs/scripts/scribe" && chmod 0755 /jffs/scripts/scribe && /jffs/scripts/scribe install

In addition to syslog-ng and logrotate, scribe also installs syslog-ng and logrotate configuration files to send syslog-ng's own logs, logrotate logs, WLCEVENTD logs, and (hopefully most) crash logs to their own separate files. All log files will be in /opt/var/log.

A single line is added to /opt/etc/init.d/S01syslog-ng that calls a helper script to kill the klogd and syslogd and make the necessary changes to the filesystem to keep syslogd from causing problems when GUI changes cause it to restart.

A single line is added to /jffs/scripts/service-event to call scribe and check for syslogd and klogd being restarted whenever service-event runs.

If Skynet is installed, a syslog-ng configuration file is also installed for it, and Skynet is automatically pointed at the new location for the iptables output. If you plan to install Skynet, it is highly preferable to install Skynet before scribe.

Options
A customized syslog-ng.conf file will be installed. This syslog-ng.conf file will remove sourcing a large number of functions from the /opt/share/syslog-ng/include/scl directory, change maximum log message size to 16K, and set stats frequency to every 6 hours. If installed, it copies the syslog-ng.conf file that came with the package to /opt/share/syslog-ng/examples.

Example Files
Of note, example files are placed in /opt/share/sylog-ng/examples/ and /opt/share/logrotate/examples/ for common programs (skynet, pixelserv-tls, etc). These can be copied directly to /opt/etc/syslog-ng.d/ and /opt/etc/logrotate.d/ for use. Do not blindly copy all of these files to their respective ".d" directories however, as doing so will prevent syslog-ng from running. Editing files directly here is not recommended as they will likely be overwritten if you choose to update the filters.

IMPORTANT
ALL files in /opt/etc/logrotate.d/ and /opt/etc/syslog-ng.d/ MUST NOT be writable by anyone other than root. Files that are writable by group or others will be ignored!

Usage
scribe has a number of commands:

• about - shows a brief description of scribe, its license, etc.
• install - can be used to force re-installation of syslog-ng, logrotate, and scribe. Doing so will overwrite any customizations you have made to the script files included for syslog-ng and logrotate.
• uninstall (or remove) - removes syslog-ng, logrotate, and scribe from your system. Doing so will erase any files you have added to /opt/etc/syslog-ng.d and /opt/etc/logrotate.d.
• update - updates the scribe script, asks to update filters.
• show-config (or config) - shows the complete combined configuration for syslog-ng including all included files in the system list utility.
• status - checks to see if syslog-ng is running and if logrotate is in the crontab (cru). Also displays the scribe script version. Also checks the syntax of the syslog-ng configuration as it exists on the disk, which may be different than what is loaded if "restart" (see below) has not been run since changing the configuration files.
• reload - reloads syslog-ng, which is necessary when changing configuration files.
• restart (or start) - restarts syslog-ng Will start syslog-ng if it is not currently running.
• debug - create a debug file and tar.gz it to allow debug info to be sent easily.
• help - shows a concise version of this section

Support
I'll try to answer questions and help troubleshoot. I expect this thread will also include discussions on syslog-ng configuration as well. scribe is hosted on GitHub. Syslog-ng Open Source Edition (OSE) is a product of One Identity (https://www.syslog-ng.com/products/open-source-log-management/). I have no connection to either syslog-ng or logrotate, I simply wrote scribe to facilitate installing these programs.

Credits
I've borrowed quite a bit of code from @thelonelycoder and @Jack Yaz, with some from @dave14305, @Adamm, and @Xentrk as well. All the well-written stuff is from them. All the crap bits are mine.

 

[Stephen Harrington]

@cmkelley I've updated to 3.0_0 on my RT-AX86U and all appears to be working Ok from a quick check.
Not sure if my model was one of the affected ones or not.

 

[JGrana]

Updated to 3.0 on an AX88U Pro (one of the -> jffs models). Looks good!

 

[zer0bitz]

Update done and everything working.

 

[cmkelley]

@cmkelley I've updated to 3.0_0 on my RT-AX86U and all appears to be working Ok from a quick check.
Not sure if my model was one of the affected ones or not.

Heh, I honestly don't know either. The only model I specifically search for in the code is the AC86, since as far as I can know, it's the only model that throws tons of "DCD Tainted" errors, so I add a filter for those for the AC86.

The hope is that nobody will need to know which models are affected, the code should detect it and adjust automatically.

 

[visortgw]

Heh, I honestly don't know either. The only model I specifically search for in the code is the AC86, since as far as I can know, it's the only model that throws tons of "DCD Tainted" errors, so I add a filter for those for the AC86.

The hope is that nobody will need to know which models are affected, the code should detect it and adjust automatically.

Actually, the tons of "DCD Tainted" errors occur on all (or most?) models when users enable pixelserv-tls or exclude devices from ad-blocking within Diversion.

 

[matthew_eli]

I've just updated to Scribe 3.0.0 and if I restart it I get this warning in the startup log:

checking directory links ...cat: read error: Is a directory
rm: '/jffs/syslog.log' is a directory
rm: '/jffs/syslog.log-1' is a directory
/jffs/scripts/scribe: line 95: can't create /jffs/syslog.log-1: Is a directory
 updated.

At the first start, all is ok: I mean, at the first start syslog.log-1 file is created normally; after the restart, it seems the 2 directories would be created in the JFFS folder and then the errors/warnings come up. I'm on GT-AX6000 with 388.2 FW.

 

[ColDen]

I've just updated to Scribe 3.0.0 and if I restart it I get this warning in the startup log:

checking directory links ...cat: read error: Is a directory
rm: '/jffs/syslog.log' is a directory
rm: '/jffs/syslog.log-1' is a directory
/jffs/scripts/scribe: line 95: can't create /jffs/syslog.log-1: Is a directory
 updated.

At the first start, all is ok: I mean, at the first start syslog.log-1 file is created normally; after the restart, it seems the 2 directories would be created in the JFFS folder and then the errors/warnings come up. I'm on GT-AX6000 with 388.2 FW.

I also have a GT-AX6000 but did not have that warning:

 

[matthew_eli]

Try to restart Scribe by 'rs' command

 

[ColDen]

Try to restart Scribe by 'rs' command

OH, I see it now:

Good catch... I only updated then performed several standard status checks thereafter.

 

[cmkelley]

Actually, the tons of "DCD Tainted" errors occur on all (or most?) models when users enable pixelserv-tls or exclude devices from ad-blocking within Diversion.

OH. Well I might take the auto-install code out then and update the README with that bit of information. Didn't know a cause had been nailed down. Thanks!

 

[jsbeddow]

OH, I see it now:
View attachment 49652

Good catch... I only updated then performed several standard status checks thereafter.

Oh no, I have the same model and updated this morning, ran the same quick tests, but haven't done a complete reboot yet. Ugh...hopefully we have a hotfix coming soon.

 

[cmkelley]

OH, I see it now:
View attachment 49652

Good catch... I only updated then performed several standard status checks thereafter.

Phooey. Clearly a logic error there somewhere. I'll have to hunt around tonight after work (U.S. west coast time).

 

[cmkelley]

Oh no, I have the same model and updated this morning, ran the same quick tests, but haven't done a complete reboot yet. Ugh...hopefully we have a hotfix coming soon.

Not sure it would be a problem on reboot. If the logic error is in the restart code it might be okay. It'll be some hours before I'm home from work and can look at it, I'm in California.

 

[cmkelley]

OH, I see it now:
View attachment 49652

Good catch... I only updated then performed several standard status checks thereafter.

Okay, if you get that error, and then exit scribe, are /jffs/syslog.log and /jffs/syslog.log-1 directories or links? If they are links then it's not a big deal other than being annoying (I'll fix it either way). If they are directories then it will affect some logging.

 

[ColDen]

Phooey. Clearly a logic error there somewhere. I'll have to hunt around tonight after work (U.S. west coast time).

There's no emergency, we know that we are in good hands...

 

[ColDen]

Okay, if you get that error, and then exit scribe, are /jffs/syslog.log and /jffs/syslog.log-1 directories or links? If they are links then it's not a big deal other than being annoying (I'll fix it either way). If they are directories then it will affect some logging.

I am not very good with Unix but I think that this is what you are looking for?

If not please guide me for the right code...

 

[visortgw]

Not sure it would be a problem on reboot. If the logic error is in the restart code it might be okay. It'll be some hours before I'm home from work and can look at it, I'm in California.

Not a problem on reboot — only when executing "rs." command from menu. Interim work-around is to uninstall and reinstall scribe.

 

[cmkelley]

I am not very good with Unix but I think that this is what you are looking for?
View attachment 49656
If not please guide me for the right code...

No, that's exactly what I was looking for. So, that's no bueno, they're directories. :-(

Well, at least that helps me find the logic error.

 

[ColDen]

No, that's exactly what I was looking for. So, that's no bueno, they're directories. :-(

Well, at least that helps me find the logic error.

If I can be of further assistance, just let me know and I'll do my best...