Scribe scribe - syslog-ng and logrotate installer

[cmkelley]

I know this is a completely unrelated question but it is semi related to entware and how it is running packages, does any one know how to see how entware is reading the date and time vs the routers date and time?

You might try starting a thread for that. I'm pretty sure this thread is not heavily trafficked.

 

[Jumpstarter]

You might try starting a thread for that. I'm pretty sure this thread is not heavily trafficked.

will do - the only reason I asked was I was wondering out of all the scripts amtm, skynet , diversion, and any of the other major ones, does any of these make a symlink with the routers clock for entware automatically.

Thank you guys for all of your help.
it has really made this script worth while.

 

[cmkelley]

will do - the only reason I asked was I was wondering out of all the scripts amtm, skynet , diversion, and any of the other major ones, does any of these make a symlink with the routers clock for entware automatically.

Thank you guys for all of your help.
it has really made this script worth while.

I don't think the clock is something you symlink to. Time is a kernel process, with some defined interface to get the time, it's not a file.

 

[Butterfly Bones]

will do - the only reason I asked was I was wondering out of all the scripts amtm, skynet , diversion, and any of the other major ones, does any of these make a symlink with the routers clock for entware automatically.

Thank you guys for all of your help.
it has really made this script worth while.

I don't think the clock is something you symlink to. Time is a kernel process, with some defined interface to get the time, it's not a file.

Yup, our ole buddy and nemesis, service event. It calls ntpd just before clock changes from default date to actual date. So no scripts, though some have built in "wait" for time synch before they run.

May  4 22:05:22 custom_script: Running /jffs/scripts/service-event (args: start ntpd)
May  4 22:05:22 ntpd: Started ntpd
May  4 22:05:22 kernel: br0: topology change detected, propagating
May  4 22:05:22 kernel: br0: port 4(eth4) entered forwarding state
May  4 22:05:22 nat: apply nat rules (/tmp/nat_rules_eth0_eth0)
May  4 22:05:22 custom_script: Running /jffs/scripts/nat-start
May  4 22:05:22 custom_script: Running /jffs/scripts/pre-mount (args: /dev/sda1 ext4)
Jun  7 08:02:30 ntp: Initial clock set

 

[Jumpstarter]

I don't think the clock is something you symlink to. Time is a kernel process, with some defined interface to get the time, it's not a file.

you are right -- wasn't meaning to imply clock , but more so imply the timezone.

 

[cmkelley]

you are right -- wasn't meaning to imply clock , but more so imply the timezone.

Oh, the only timezone file on the router that I know of is /etc/TZ. I grab that as part of getting syslog-ng running, or some people have weird timezone issues.

# export timezone if not already set
[ -z "$TZ" ] && export TZ=$( cat /etc/TZ )

 

[Butterfly Bones]

you are right -- wasn't meaning to imply clock , but more so imply the timezone.

Here is a thread that might help.
https://www.snbforums.com/threads/entware-tz-issue.57006/#post-496781

 

[Swistheater]

I know you can set this up to see pixelserv logs, but is there away to set this up to fetch what pixelserv has blocked via dnsmasq blocked domain filter

for example

 

[elorimer]

This seems to me to be unnecessary within scribe. If you want to see pixelserv in real time, use diversion. If you want to fetch what pixelserv has blocked, then you want something to compile stats from the pixelserv log, the way diversion parses the dnsmasq log. My router emails such a compilation each night.

 

[cmkelley]

I know you can set this up to see pixelserv logs, but is there away to set this up to fetch what pixelserv has blocked via dnsmasq blocked domain filter
View attachment 18160

for example

View attachment 18161

Well, you could set up a syslog-ng filter to watch dnsmasq.log, and create another copy of it with a different name. The problem is, you'd soon find that since dnsmasq.log gets really big, really quick, and with log file sizes somewhere between 2Mb and 20Mb (likely dependent upon the processor in your router), the webgui really bogs down. You'd probably need to modify the logrotate cron to run hourly just to keep that file in check.

 

[cmkelley]

scribe v2.1_0 is out.

- The big thing (that I thought was a little thing until it turned out to affect most of the functions!) is the ability to stop syslog-ng and logrotate, and start the firmware syslogd, without uninstalling scribe/syslog-ng/logrotate. Note that if you reboot with syslog-ng & logrotate stopped, they will re-start at bootup and shut down syslogd. Also, it's all or nothing, there is no option for instance to shut down syslog-ng and leave logrotate running or vice versa.

- scribe now adds a line to the unmount script in /jffs/scripts (or creates it if unmount isn't present) that runs the above stop routine when unmounting the usb with scribe on it. The code to accomplish this was stolen blatantly and without remorse from @thelonelycoder. Hey, if it works, don't screw with it.

- There is now a command to reload the syslog-ng configuration. After you copy files into the /opt/etc/syslog-ng.d/ directory, you need to tell scribe to reload its configuration for the new files to be included.

- Added some things to the status page. The status page tries to ensure the correct links, the correct lines in script files, and the like are present. It attempts to fix any issues it finds. Also condensed the status page, removing a lot of blank lines.

- The scribe menu will show a (very) limited set of options if the scribe script is present but it is not (or it thinks it is not) installed. If syslog-ng & logrotate have been stopped, the menu is somewhat limited to remove things you shouldn't do when scribe isn't running.

- Reduced the maximum log sizes for pixelserve and crash to keep the webui from choking if you're using @Jack Yaz's uiScribe (which is very cool, btw). Note that currently scribe doesn't update anything in the syslog-ng.d or logrotate.d directories in case the user has customized those scripts. The new pixelserv logrotate file will be copied into /opt/share/logrotate/examples, you can then copy it into logrotoate.d. You can force updating crash by, ironically, deleting the crash filter from logrotate.d before updating scribe. It will decide that crash is a new file and copy the updated version to logrotate.d


My todo list now includes
- Add a backup and restore option for the syslog-ng & logrotate configuration files
- Add ability to reset syslog-ng & logrotate configurations to be the same as a new installation
- Gracefully (i.e. ask) update files in syslog-ng.d & logrotate.d
- Have a way in the menu to select what files to copy from the share directories to the .d directories.
- Write some documentation
- Update teh GitHub page

 

[Jack Yaz]

I know you can set this up to see pixelserv logs, but is there away to set this up to fetch what pixelserv has blocked via dnsmasq blocked domain filter
View attachment 18160

for example

View attachment 18161

I've got something like this working locally, effectively filtering dnsmasq.log by the pixelserv-tls ip

Here's a project for someone! Dnsmasq filters in syslog-ng to replicate the stats generated by Diversion, so that I can hook uiDivStats to them to have realtime graphs

 

[Swistheater]

I've got something like this working locally, effectively filtering dnsmasq.log by the pixelserv-tls ip

Here's a project for someone! Dnsmasq filters in syslog-ng to replicate the stats generated by Diversion, so that I can hook uiDivStats to them to have realtime graphs

graphs that populate every 3 second depending on the interval chosen lol. kind of like the QoS graphs.

 

[elorimer]

There is every reason to filter pixelserv messages out of the messages file if pixelserv is logging at level 2 or more. There is zero point to doing anything with the result that diversion and uiDiversion don't already do.

 

[elorimer]

@cmkelley: something to consider for your todo list would be a way to clear the messages file gracefully. I sometimes want to start a clean messages file after rebooting after some change. If you just delete the file with syslog-ng running you get busted inodes. Perhaps something that would stop syslog-ng, delete the file, restart it or reboot.

 

[cmkelley]

@cmkelley: something to consider for your todo list would be a way to clear the messages file gracefully. I sometimes want to start a clean messages file after rebooting after some change. If you just delete the file with syslog-ng running you get busted inodes. Perhaps something that would stop syslog-ng, delete the file, restart it or reboot.

Clearing it is no problem, delete it and send a SIGHUP to syslog-ng. But you'd lose it all rather than have it rotated into a log. Forcing a logrotate on a single log requires every parameter you care about to be in the logrotate file for that log, none of the defaults (including those in the logrotate.conf file) get read in, and the program's internal defaults don't match those in the default logrotate.conf.

 

[cmkelley]

There is every reason to filter pixelserv messages out of the messages file if pixelserv is logging at level 2 or more. There is zero point to doing anything with the result that diversion and uiDiversion don't already do.

For you maybe. Consider some people may have use cases you haven't thought of.

 

[Swistheater]

For you maybe. Consider some people may have use cases you haven't thought of.

some people may track things more often in real time, but it is hard to think such a thing would be possible without a more daemon style approach.

 

[elorimer]

Fair enough. but how is this not what diversion does in following the log by blocked domain?

 

[cmkelley]

v2.1_1 is up

- restarts uiScribe (if installed) to recheck configuration any time the syslog-ng configuration might have changed.

That's it.