uiScribe uiScribe - Custom System Log page for "scribed" logs

[cmkelley]

So based on this picture I believe it is safe to assume that is my issue. It does look as though it is rotating the logs based on the name of the .log1, .log2 file sizes and such for dnsmasq alone. Not sure what "messages" does as its over 10mb or if the other dnsmasq.log1 or log2 (84mb size) are still needed or would affect the router speed in their current state.
View attachment 18362

Ignore any files named 'dnsmasq*' they're not syslog-ng files and not shown in uiScribe. Log files rotated by logrotate will have the date code appended, like 'syslog-ng.log-20190616'

A 10Gig messages file is no bueno. 'messages' is the default log file, /tmp/syslog.log is symlinked to it.

Do you have an /opt/etc/logrotate.d/messages file? That should be limiting messages to 1024k. That limit has always been there since without Jack's uiScribe, that is the file that gets shown (because syslog.log is symlinked to it).

 

[QuikSilver]

Mine shows what your suggesting now, but will all the info and changes I've done can't say it wasn't before.

 

[no_name]

@QuikSilver, I see from your sig you have and 86U, @no_name, how about you?

I ask because I suspect the guilty logfile is crash. Unless you installed scribe very recently, /opt/etc/logrotate.d/crash probably looks like:

/opt/var/log/crash.log {
    minsize 4096k
    daily
    postrotate
        /usr/bin/killall -HUP syslog-ng
    endscript
}

The culprit is the minsize line. Change the minsize line to

    minsize 1024k

Then you should be able to select lr from the scribe menu, which should detect the logfile is too big and rotate it. Lots of 'should's, I know, I've admittedly not tested this yet. Forcing a log to rotate regardless of if all criteria are met is on my to-do list, but it's a bit more difficult that I originally thought, it's not quite as simple as passing '--force' to logrotate.

I’m using an RT-AC86U as well


Sent from my iPad using Tapatalk

 

[QuikSilver]

@cmkelley, Awesome! Did those changes, ran "rs" and then refresh to see "messages" file size went from 10mb to 1kb. Gui much faster loading now! Thanks! Now to make sure I got it configured to run automatically.

 

[cmkelley]

@cmkelley, Awesome! Did those changes, ran "rs" and then refresh to see "messages" file size went from 10mb to 1kb. Gui much faster loading now! Thanks! Now to make sure I got it configured to run automatically.

Did you run 'rs' or 'lr'? I'm not quite sure I understand why 'rs' would have done anything, it shouldn't force a logrotate.

logrotate should be running every night. If you type 'cru l' you should see something like:

5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#

If you don't, check your /jffs/scripts/post-mount, it should have in it somewhere:

cru a logrotate "5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1" # added by scribe

 

[QuikSilver]

Did you run 'rs' or 'lr'? I'm not quite sure I understand why 'rs' would have done anything, it shouldn't force a logrotate.

logrotate should be running every night. If you type 'cru l' you should see something like:

5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#

If you don't, check your /jffs/scripts/post-mount, it should have in it somewhere:

cru a logrotate "5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1" # added by scribe

Sorry, mixed up my commands. I did "lr" to run the logrotate now. As for the other sections I see the daily log rotate cron job in postmount. Thanks for your help @cmkelley , router is much faster when viewing log page now.

 

[no_name]

I’ve been following along and learned some more [emoji4] my minsize was already set to 1024k

/opt/var/log/crash.log {
minsize 1024k
daily
postrotate
/usr/bin/killall -HUP syslog-ng
endscript


I will come back to it when it’s quieter

Thanks all


Sent from my iPad using Tapatalk

 

[elorimer]

Just to be hypertechnical--minsize of 1024k or 1M means that when the daily logrotate chron job runs, it doesn't rotate the log unless it is larger than that. It doesn't limit the message size between chron jobs.

@cmkelley, I wonder if it would be useful to incorporate a check that there is nothing in syslog-ng.d or logrotate.d that relates to dnsmasq. This comes up from time to time, and it is logical to think it has something to dowith syslog-ng rather than being a diversion-specific thing. Alternatively, maybe we should relocate everything to /opt/var/log/syslog-ng.logs to avoid a collision. Or maybe /opt/var/syslog-ng.logs.

 

[cmkelley]

Just to be hypertechnical--minsize of 1024k or 1M means that when the daily logrotate chron job runs, it doesn't rotate the log unless it is larger than that. It doesn't limit the message size between chron jobs.

Right, I am aware of that, but IIRC even 2Mb files don't cause any noticeable slowness. I don't know where it starts to become noticeably slow, other than it's somewhere between 2Mb and 8Mb (based on experience, not controlled testing). So a minsize to rotate of 1Mb is pretty safe.

If someone has a log file growing at over 1Mb/day, then they can run logrotate hourly and write a 'size' rule for that file (IIRC FreeBSD takes this approach with their log rotation facility). Also logrotate supports an "hourly" time period now as well for situations where it's run hourly.

@cmkelley, I wonder if it would be useful to incorporate a check that there is nothing in syslog-ng.d or logrotate.d that relates to dnsmasq. This comes up from time to time, and it is logical to think it has something to dowith syslog-ng rather than being a diversion-specific thing. Alternatively, maybe we should relocate everything to /opt/var/log/syslog-ng.logs to avoid a collision. Or maybe /opt/var/syslog-ng.logs.

Assuming you're still talking about the webui slowness, it has nothing to do with Diversion at all. Before uiScribe, the only filesize I cared about was messages, because that was the only one displayed in the webui. So some were only rotated weekly regardless of size (i.e. they could be huge).

uiScribe will NOT show dnsmasq files if there is no syslog-ng filter that write to it. There should never be filter rules that write to dnsmasq because those files are created and used by Skynet. If someone creates a syslog-ng rule that messes with dnsmasq while Skynet is running, that's on them. Doing so _will_ cause problems with Skynet, 'grep dnsmasq firewall' will show you that.

EDIT: My bad, Diversion creates/uses the dnsmasq files as well. So it would probably break both of them if someone messes with them.

EDIT2: Had typed Gb when I meant Mb. Geh. Fixed for clarity, thanks Elorimer!

/opt/var/log is the default place Entware's syslog-ng puts the messages file, I didn't chose it. It's possible other Entware apps rely on or expect logs to be there.

 

[Swistheater]

Right, I am aware of that, but IIRC even 2Gb files don't cause any noticeable slowness. I don't know where it starts to become noticeably slow, other than it's somewhere between 2Gb and 8Gb (based on experience, not controlled testing). So a minsize to rotate of 1Gb is pretty safe.

If someone has a log file growing at over 1Gb/day, then they can run logrotate hourly and write a 'size' rule for that file (IIRC FreeBSD takes this approach with their log rotation facility). Also logrotate supports an "hourly" time period now as well for situations where it's run hourly.

Assuming you're still talking about the webui slowness, it has nothing to do with Diversion at all. Before uiScribe, the only filesize I cared about was messages, because that was the only one displayed in the webui. So some were only rotated weekly regardless of size (i.e. they could be huge).

uiScribe will NOT show dnsmasq files if there is no syslog-ng filter that write to it. There should never be filter rules that write to dnsmasq because those files are created and used by Skynet. If someone creates a syslog-ng rule that messes with dnsmasq while Skynet is running, that's on them. Doing so _will_ cause problems with Skynet, 'grep dnsmasq firewall' will show you that.

/opt/var/log is the default place Entware's syslog-ng puts the messages file, I didn't chose it. It's possible other Entware apps rely on or expect logs to be there.

could it be in relation to the auto scroll features and auto refresh options.

 

[cmkelley]

could it be in relation to the auto scroll features and auto refresh options.

Unknown. It happens in the stock webui when the syslog.log file (or messages when syslog.log is symlinked to it) gets too big.

 

[elorimer]

1Gb=1024M

My pixelserv log generates about 60M a day, so it really bogs down the webui, and I never want to see it in the first place. For that reason I'm not using uiScribe (which I really, really like) until I can figure out how to blacklist that log so it doesn't show. --preprocess is so wickedly comprehensive I haven't figured out how to do that in the syslog-ng end of things.

Rotating it more frequently defeats the purpose of the log, which, like diversion and skynet, I use to generate statistics, in this case daily, and then discard through the logrotate cycle.

 

[cmkelley]

1Gb=1024M

Man, I knew I was off my game yesterday, but that's WAY off. LOL! I've no idea why I typed Gb instead of Mb. Fixed.

My pixelserv log generates about 60M a day, so it really bogs down the webui, and I never want to see it in the first place. For that reason I'm not using uiScribe (which I really, really like) until I can figure out how to blacklist that log so it doesn't show. --preprocess is so wickedly comprehensive I haven't figured out how to do that in the syslog-ng end of things.

Keeping it out of uiScribe is a feature you'll have to work out with Jack. But it can't be a blanket keep pixelserv logs out of uiScribe, others may want to see that file.

Rotating it more frequently defeats the purpose of the log, which, like diversion and skynet, I use to generate statistics, in this case daily, and then discard through the logrotate cycle.

There's more than one way to skin that cat, you could generate the statistics hourly as well. Or have an hourly cron job to scrape the pixelserv log off to another file for your nightly statistics processing. Just make sure you send syslog-ng a HUP if you change the file inode.[/QUOTE]

 

[no_name]

Pixelserv was the cause of the webui slowing down as well, I removed it for now and will play again later. I do like to tinker. The openvpn log is a favourite of mine

Sent from my iPad using Tapatalk

 

[elorimer]

Pixelserv was the cause of the webui slowing down as well, I removed it for now

How?

 

[elorimer]

I've no idea why I typed Gb instead of Mb.

Because you are a thousand times more experienced with syslog-ng, naturally.

 

[no_name]

How?

I use WinSCP and followed these instructions

Just for fun I added these from the examples file

Sent from my iPad using Tapatalk

 

[no_name]

To remove pixelserv I went to this file and deleted pixelserv

I then went to this file and deleted the pixelserv file from there as well

I then run functions lr and rs in scribe


Ignore my method of deleting it and do as L&LD suggests [emoji4]

 

[L&LD]

To remove pixelserv I went to this file and deleted pixelserv

I then went to this file and deleted the pixelserv file from there as well

I then run functions lr and rs in scribe

Am I the only one cringing here?

Why not just disable it via the Diversion menu?

 

[no_name]

Am I the only one cringing here?

Why not just disable it via the Diversion menu?

I’m cringing now you mentioned that, it didn’t occur to me [emoji85]Thank you for picking me up that one [emoji4]


Sent from my iPad using Tapatalk