Scribe scribe - syslog-ng and logrotate installer

[elorimer]

Hmm. The messages filter in syslog-ng is pretty powerful but slow, and the more specific you make it, the slower I suspect it is. These routers don't generate enough messages to be a problem, I suspect. But you might try to strip out some of the specifics to skinny down the filter to the minimum. Like, does anything else in the messages.log file use the phrase "CDB"? or "Sense"? Maybe that's all you need.

But I'm wondering if you should explore why these messages are being generated. I say this because I don't recall any other posts in this forum like this. I'm not sure they are "spam"; This drive is generating so many error messages that you want to make them stop. But maybe you have a fire alarm going on that you should investigate first.

 

[Megahurtz]

I haven't researched it extensively, but I recall reading in a thread somewhere that other people said it was a common occurrence with the 4TB WD portable hard drives.

Regarding slimming down my message filter, I initially tried with just "sd 0:0:0:0: [sda]", but that didn't catch any of the four messages I repeatedly have that start with it. Is there a reason all four of those messages would slip past that message filter?

 

[elorimer]

I'll think on the filtering some. There must be something, maybe the colons, maybe the brackets, that is throwing off the match.

On the idea of a 4tb portable hard drive, this is pushing a router a bit, and can I say, good on you for scoring such a thing. I had for some time a 4tb seagate desktop drive hooked up without issue, and I've had some 1 and 2tb portables, and I have a 4tb 2.5" inch drive in my tivo, with some shoehorning. But a 4tb portable external is a rare goose to start with, more so if driven off the router. Why I'm thinking these error messages are something to attend to. We might start with asking what router you are using, off which port.

Not being critical here. My first thought was that the router is on 24/7, so there would be a power savings if I hung a big disk off the router. When I got to 4Tb, I started to do some thinking. So now I am what "datahoarders" call a "babyhoarder", and I will not rely on a biggly amount of data hung off the router, ever again.

 

[cmkelley]

I haven't researched it extensively, but I recall reading in a thread somewhere that other people said it was a common occurrence with the 4TB WD portable hard drives.

Regarding slimming down my message filter, I initially tried with just "sd 0:0:0:0: [sda]", but that didn't catch any of the four messages I repeatedly have that start with it. Is there a reason all four of those messages would slip past that message filter?

So, the way I would approach it is to copy expandlog from /opt/share/syslog-ng/examples to /opt/etc/syslog-ng.d and restart syslog-ng from scribe. This will create a very large log pretty quickly, but with 4 TB I'm guessing you can stand to create a big file. You can then search the resulting expanded-syslog.log file for, say "0:0:0:0:" and see if that gives you hint why it's not working. Perhaps the message isn't quite what it seems? Don't forget to delete the filter file and restart syslog-ng again after you find the culprit. Also, don't go into the UIScribe GUI page with the expandlog filter running. The UI can't handle such large files (this is not UIScribe's fault, it's probably to do with the webserver used in the firmware).

Also RMerlin had this to say previously about running 4TB drives off the router (https://www.snbforums.com/threads/w...d-drive-that-ac68u-can-run.35763/#post-291173):

This is a bad idea. First, the router doesn't even have enough RAM to properly cache the metadata from such a large volume, or run any filesystem check in case of a problem.

True, that message is from 4 years ago, and I don't know what router you're using, but unless you're running a really recent router, I'll bet the same applies today.

My (unasked for) $.02 - Your messages may be indicative that the router or hard drive are not happy with the set up. A "common occurrence" doesn't mean it's harmless. I'd think a 4GB Raspberry Pi 4 would make a MUCH better NAS than the router, and the RPi will have nice GUI to make it easier. There are plenty of articles on how to do this on the web.

 

[cmkelley]

Im stuck in a loop. Getting the following

Updated list of available packages in /opt/var/opkg-lists/entware

Installing syslog-ng (3.27.1-1) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/syslog-ng_3.27.1-1_armv7-2.6.ipk
Configuring syslog-ng.
syslog-ng: error while loading shared libraries: /opt/lib/librt.so.1: invalid ELF header

syslog-ng version 3.19 or higher required!
Please update your Entware packages and run scribe install again.

Removing package syslog-ng from root...

Any suggestions what to do? Updating Entware says all upto date.

syslog-ng is now at 3.27.1-2 in entware, maybe try installing syslog-ng again?

 

[Megahurtz]

So, the way I would approach it is to copy expandlog from /opt/share/syslog-ng/examples to /opt/etc/syslog-ng.d and restart syslog-ng from scribe. This will create a very large log pretty quickly, but with 4 TB I'm guessing you can stand to create a big file. You can then search the resulting expanded-syslog.log file for, say "0:0:0:0:" and see if that gives you hint why it's not working. Perhaps the message isn't quite what it seems? Don't forget to delete the filter file and restart syslog-ng again after you find the culprit. Also, don't go into the UIScribe GUI page with the expandlog filter running. The UI can't handle such large files (this is not UIScribe's fault, it's probably to do with the webserver used in the firmware).

Also RMerlin had this to say previously about running 4TB drives off the router (https://www.snbforums.com/threads/w...d-drive-that-ac68u-can-run.35763/#post-291173):


True, that message is from 4 years ago, and I don't know what router you're using, but unless you're running a really recent router, I'll bet the same applies today.

My (unasked for) $.02 - Your messages may be indicative that the router or hard drive are not happy with the set up. A "common occurrence" doesn't mean it's harmless. I'd think a 4GB Raspberry Pi 4 would make a MUCH better NAS than the router, and the RPi will have nice GUI to make it easier. There are plenty of articles on how to do this on the web.

I had an extra 4TB WD My Passport drive laying around and decided to throw it on the router when I installed Merlin. My primary purpose was to set up a 1TB partition to use as a secondary Time Machine backup location for my MacBook, but figured while I had it, I may as well use it to store some video files. If the whole thing craps itself, I'd be disappointed but not terribly upset. It was more just something to tinker with given hardware I already had at hand.

I did figure out how to get all of those sda messages filtered out. This is finally what worked for me:

filter f_sdaspam{
message("Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE") or
message("Sense Key : Illegal Request") or
message("Add. Sense: Invalid command operation code") or
message("CDB: Write same") or
message("end_request: I/O error") or
message("xhci_hcd 0000:00:0c.0");
};

It isn't quite as specific as I wanted, but it doesn't appear to be inadvertently trapping anything else. All 6 messages are now going to their own log file, time machine messages are sent to their own log, and transmission is separated as well. I'm very happy with it now! The main sys log is very readable. Thanks for your help and suggestions, everybody

 

[elorimer]

syslog-ng is now at 3.27.1-2 in entware, maybe try installing syslog-ng again?

We were at 3.27-1-1. weren't we? I wonder what is new. I see github is up to 3.29-1

 

[Butterfly Bones]

We were at 3.27-1-1. weren't we? I wonder what is new. I see github is up to 3.29-1

After the recent update

user@RT-AC86U-4608:/tmp/home/root# opkg list-installed | grep syslog
syslog-ng - 3.27.1-2

 

[giant46man46]

Skynet isn't necessary.

Scribe is a script that handles configuring syslog-ng on the router to replace the native syslogd. From there syslog-ng can take you in a lot of different directions, including parsing out the single system log into separate logs. But it doesn't have anything to do with anything else other than logging.

UiScribe is a script that builds on scribe by displaying in the GUI the different logs that scribe creates.

Skynet is something entirely different. That is a firewall program that denies inbound and outbound traffic to addresses it thinks are nasty. It generates log messages each time it does that. If scribe/syslog-ng isn't active, those messages go to the system log. If scribe/syslog-ng is active, it will send those messages to its own log. Hourly it purges those messages from the log destination (whichever one) and logs a summary. There was a bit of interaction there, because the purge process had the unintended effect of stopping syslog-ng and restarting syslogd. That has long since been fixed, and that is the only interaction between scribe and skynet.

As to this:
you're going to have to finish that sentence.

I do apologize for not finishing my thought. The only log that works is the system messages. The rest of the logs do not work. They do not display anything on the screen. Can you please help me?

 

[giant46man46]

I'm not quite sure I understand your problem, but neither scribe nor Skynet rely on each other work. If you have, or are going to use Skynet, you should install Skynet first to ensure scribe properly handles Skynet's logs, but that's it.

I'm sorry, I can't understand the second part of your question. Is there a problem with the other logs?

Forget the part about Skynet. I realize they are two different things. The only reason I even mentioned the two things in the same sentence is because during the install of Scribe, I was asked if I wanted to install Skynet with scribe. Please forget the Skynet. My problem is the only log that works in Scribe is the System Messages log. The other logs are blank. I have attached a screenshot to this post just in case my problem is still unclear.

 

[elorimer]

From the screenshot, it looks like nothing is working; the last system message is the built-in log daemons exiting, and there should be stuff after that.

Login from a terminal and run "syslog-ng -Fevd" to see why syslog-ng isn't starting.

 

[giant46man46]

From the screenshot, it looks like nothing is working; the last system message is the built-in log daemons exiting, and there should be stuff after that.

Login from a terminal and run "syslog-ng -Fevd" to see why syslog-ng isn't starting.

Thanks elorimer. I have attached a screenshot of the result of the syslog-ng -Fevd command. Please let me know what my next steps are. Thank you so much.

 

[elorimer]

Thanks elorimer. I have attached a screenshot of the result of the syslog-ng -Fevd command. Please let me know what my next steps are. Thank you so much.

Can't read it unfortunately. But there should be a place where it indicates what character of what line of what file it is choking on.

 

[giant46man46]

Can't read it unfortunately. But there should be a place where it indicates what character of what line of what file it is choking on.

I just attached a text file containing the log. The name of the file is Log.txt. I didn't see any line item that it is choking on, but mine are the eyes of inexperience. Are you able to read my attachment?

Ken

 

[elorimer]

Yes, that worked to read, but it doesn't seem to be loading.

This is new to me. It seems to be loading fine, but it ends wrong, because it should end with a message that says syslog-ng is starting.

So, what model router do you have, what version of Merlin do you have?

Also, what files, if any, do you see in /opt/var/log?

EDIT: So could others check their log files? The recent Entware update borked my unbound, but looking at my files I can see some odd things. (So thanks@giant46man46 if you've picked up on something.)

 

[giant46man46]

Yes, that worked to read, but it doesn't seem to be loading.

This is new to me. It seems to be loading fine, but it ends wrong, because it should end with a message that says syslog-ng is starting.

So, what model router do you have, what version of Merlin do you have?

Also, what files, if any, do you see in /opt/var/log?

I only see one file and the name of that file is "messages". I have an ASUS AC1900, RT-AC68U, Merlin version 384.19

 

[Butterfly Bones]

I update Entware five days ago and all good here.

user@RT-AC86U-4608:/tmp/home/root# ls -alh /opt/var/log/
drwxr-xr-x    2 mtn_danc root        4.0K Sep 19 17:00 .
drwxr-xr-x    8 mtn_danc root        4.0K Sep 19 05:40 ..
-rw-------    1 mtn_danc root      131.0K Sep 19 04:26 afpd.log
-rw-------    1 mtn_danc root         188 Aug 27 18:27 afpd.log-20200830.gz
-rw-------    1 mtn_danc root         612 Sep 11 16:30 afpd.log-20200913
-rw-------    1 mtn_danc root        1.1K Apr 15 02:00 amas_lib.log
-rw-------    1 mtn_danc root         369 Sep 15 12:24 crash.log
-rw-r-----    1 nobody   root        9.3M Sep 19 17:09 dnsmasq.log
-rw-r-----    1 nobody   root       17.9M Sep 19 05:20 dnsmasq.log1
-rw-r-----    1 nobody   root       34.3M Sep 19 05:20 dnsmasq.log2
-rw-------    1 mtn_danc root       12.4K Sep 19 16:17 ethernet.log
-rw-------    1 mtn_danc root         705 Aug 21 20:44 ethernet.log-20200823.gz
-rw-------    1 mtn_danc root         871 Aug 28 15:01 ethernet.log-20200830.gz
-rw-------    1 mtn_danc root         513 Sep  2 19:17 ethernet.log-20200906.gz
-rw-------    1 mtn_danc root        9.1K Sep 12 16:52 ethernet.log-20200913
-rw-------    1 mtn_danc root      194.9K Sep 19 00:05 logrotate.log
-rw-------    1 mtn_danc root      407.7K Sep 19 16:25 messages
-rw-------    1 mtn_danc root      166.3K May 17 00:03 messages-20200517.gz
-rw-------    1 mtn_danc root      206.2K Aug 18 00:03 messages-20200818.gz
-rw-------    1 mtn_danc root         491 Aug 19 00:03 messages-20200819.gz
-rw-------    1 mtn_danc root       16.2K Aug 21 00:03 messages-20200821.gz
-rw-------    1 mtn_danc root        6.1K Aug 23 00:03 messages-20200823
-rw-------    1 mtn_danc root      734.4K Sep 19 17:04 openvpn.log
-rw-------    1 mtn_danc root       57.7K Apr 14 18:46 openvpn.log-20200415.gz
-rw-------    1 mtn_danc root       60.6K Aug 11 12:42 openvpn.log-20200812.gz
-rw-------    1 mtn_danc root        1.0M Sep 15 12:26 openvpn.log-20200916
-rw-------    1 mtn_danc root      102.0K Sep 19 04:26 pixelserv.log
-rw-------    1 mtn_danc root       21.9K Sep 19 17:09 skynet-0.log
-rw-------    1 mtn_danc root       37.9K Sep 19 17:00 syslog-ng.log
-rw-------    1 mtn_danc root        2.5K Aug 23 00:05 syslog-ng.log-20200823.gz
-rw-------    1 mtn_danc root        1.8K Aug 30 00:05 syslog-ng.log-20200830.gz
-rw-------    1 mtn_danc root        1.9K Sep  6 00:00 syslog-ng.log-20200906.gz
-rw-------    1 mtn_danc root       43.9K Sep 13 00:05 syslog-ng.log-20200913
-rw-------    1 mtn_danc root       84.9K Sep 19 17:04 wlceventd.log
-rw-------    1 mtn_danc root        3.0K Aug 22 20:21 wlceventd.log-20200823.gz
-rw-------    1 mtn_danc root        5.9K Aug 29 20:53 wlceventd.log-20200830.gz
-rw-------    1 mtn_danc root        4.4K Sep  6 00:04 wlceventd.log-20200906.gz
-rw-------    1 mtn_danc root      106.5K Sep 13 00:00 wlceventd.log-20200913

 

[elorimer]

I update Entware five days ago and all good here.

Have you rebooted? I did and syslog-ng isnt working. Haven't had time to puzzle at it.

 

[Butterfly Bones]

Have you rebooted? I did and syslog-ng isnt working. Haven't had time to puzzle at it.

Yes, at least twice in that time. Moving furniture around.

 

[elorimer]

Have you rebooted? I did and syslog-ng isnt working. Haven't had time to puzzle at it.

NVM. I had plugged a hard drive into the USB3 port to test the smb sharing, and that was preventing syslog-ng from starting on reboot correctly. When I removed it, everything worked again. So it was something entirely different, which I am not going to bother with.