Scribe scribe - syslog-ng and logrotate installer

[Butterfly Bones]

Skynet doesn't delete syslog-0.og. Add a comment line to it. The next time skynet scrubs it, all that will be left is that comment line.

If syslog.log is still symlinked to messages, it can't miss the GUI. The GUI is literally reading the messages file through the symlink.

I see this line every hour, so I should have said "replace" and not "delete".

tail: /opt/var/log/skynet-0.log has been replaced; following end of new file

That does make sense, but that does not seem to be what my AC86U does. Last night I changed the filter section of my skynet file and added a line "message("Skynet: [#]"). It looks like this now.

# logs everything from Skynet to /opt/var/log/skynet-0.log
filter f_skynet {
    program("Skynet") or
    message("BLOCKED -") or
    message("DROP IN=") or
    message("Skynet: [#]");
};

And today only two lines of the hourly reports show in the log.

Apr 13 17:59:28 RT-AC86U-4608 kernel: [BLOCKED - INBOUND] IN=eth0 OUT= MAC=88:d7:f6:1d:46:08:00:01:5c:6d:22:46:08:00 SRC=185.53.88.2 DST=[redacted] LEN=402 TOS=0x00 PREC=0x00 TTL=42 ID=46296 DF PROTO=UDP SPT=5252 DPT=5060 LEN=382 MARK=0x8000000
tail: /opt/var/log/skynet-0.log has been replaced; following end of new file
Apr 13 17:00:03 RT-AC86U-4608 Skynet: [#] 140170 IPs (+0) -- 1672 Ranges Banned (+0) || 834 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 13 18:00:03 RT-AC86U-4608 Skynet: [#] 140170 IPs (+0) -- 1672 Ranges Banned (+0) || 958 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 13 18:00:42 RT-AC86U-4608 kernel: [BLOCKED - INBOUND] IN=eth0 OUT= MAC=88:d7:f6:1d:46:08:00:01:5c:6d:22:46:08:00 SRC=81.22.45.193 DST=[redacted] LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=15966 PROTO=TCP SPT=40664 DPT=12521 SEQ=3729800024 ACK=0 WINDOW=1024 RES=0x00 SYN URGP=0 MARK=0x8000000

Those two lines show every hour, but only those two. I miss being able to check the Skynet hourly logs since I have had a couple strange occurrences with probes. I have solved some with checking the hourly report, along with the reports menu in Skynet. I miss not having that granular control. Now that syslog-ng has cleaned up the syslog, I may revert back with Skynet and let it wright to syslog, since it cleans up every hour. At least until we can get a handle on how to keep about 48 hours of Skynet hourly reports for review. Now that is not possible.

 

[cmkelley]

I see this line every hour, so I should have said "replace" and not "delete".

tail: /opt/var/log/skynet-0.log has been replaced; following end of new file

That does make sense, but that does not seem to be what my AC86U does. Last night I changed the filter section of my skynet file and added a line "message("Skynet: [#]"). It looks like this now.

# logs everything from Skynet to /opt/var/log/skynet-0.log
filter f_skynet {
    program("Skynet") or
    message("BLOCKED -") or
    message("DROP IN=") or
    message("Skynet: [#]");
};

And today only two lines of the hourly reports show in the log.

Apr 13 17:59:28 RT-AC86U-4608 kernel: [BLOCKED - INBOUND] IN=eth0 OUT= MAC=88:d7:f6:1d:46:08:00:01:5c:6d:22:46:08:00 SRC=185.53.88.2 DST=[redacted] LEN=402 TOS=0x00 PREC=0x00 TTL=42 ID=46296 DF PROTO=UDP SPT=5252 DPT=5060 LEN=382 MARK=0x8000000
tail: /opt/var/log/skynet-0.log has been replaced; following end of new file
Apr 13 17:00:03 RT-AC86U-4608 Skynet: [#] 140170 IPs (+0) -- 1672 Ranges Banned (+0) || 834 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 13 18:00:03 RT-AC86U-4608 Skynet: [#] 140170 IPs (+0) -- 1672 Ranges Banned (+0) || 958 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 13 18:00:42 RT-AC86U-4608 kernel: [BLOCKED - INBOUND] IN=eth0 OUT= MAC=88:d7:f6:1d:46:08:00:01:5c:6d:22:46:08:00 SRC=81.22.45.193 DST=[redacted] LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=15966 PROTO=TCP SPT=40664 DPT=12521 SEQ=3729800024 ACK=0 WINDOW=1024 RES=0x00 SYN URGP=0 MARK=0x8000000

Those two lines show every hour, but only those two. I miss being able to check the Skynet hourly logs since I have had a couple strange occurrences with probes. I have solved some with checking the hourly report, along with the reports menu in Skynet. I miss not having that granular control. Now that syslog-ng has cleaned up the syslog, I may revert back with Skynet and let it wright to syslog, since it cleans up every hour. At least until we can get a handle on how to keep about 48 hours of Skynet hourly reports for review. Now that is not possible.

Actually, skynet already does that for you.

The Skynet logs get moved to a file named skynet.log in the skynet directory on the root of your usb device. Other skynet "events" are moved to a file named events.log in the same location.

I symlinked those to /opt/var/log to those so I can read all the logs in one place

ln -s /mnt/[usb name]/skynet/skynet.log /opt/var/log/skynet.log
ln -s /mnt/[usb name]/skynet/events.log /opt/var/log/events_skynet.log

 

[Cam]

Can you do me a favor? Leave source kernel commented out, and change source src to:

source src {
    system();
    internal();
};

I had already tried that and it didn't help unfortunately.
But will give another go as I was changing lots of things 1 by 1 and could have had something else conflicting...

PS. A quick look at the source code (line 3852) for klog start, and it seems it is called as /sbin/klogd, so a null file wouldn't probably help.

 

[cmkelley]

I had already tried that and it didn't help unfortunately.
But will give another go as I was changing lots of things 1 by 1 and could have had something else conflicting...

PS. A quick look at the source code (line 3852) for klog start, and it seems it is called as /sbin/klogd, so a null file wouldn't probably help.

When I look at the combined config, system() sources /dev/kmsg instead of /proc/kmsg ... I don't know if doing that will prevent garbling or not. So far I haven't seen any garbling with that (and source kernel commented out), but I'd like a few more eyes.

 

[Cam]

So no luck with the system() option as source.
Still drops/mangles some logs in skynet-0.log and some cutoff logs in messages, such as..

/opt/var/log/messages
Apr 14 13:34:05 RT-AC68U kernel[137]: RES=0x00 SYN URGP=0 OPT (0204054801010402) MARK=0x83c40070

Only difference to having the kernel source active is that kernel now has the PID (137 above, which is klogd) attached.

 

[cmkelley]

So no luck with the system() option as source.
Still drops/mangles some logs in skynet-0.log and some cutoff logs in messages, such as..

/opt/var/log/messages
Apr 14 13:34:05 RT-AC68U kernel[137]: RES=0x00 SYN URGP=0 OPT (0204054801010402) MARK=0x83c40070

Only difference to having the kernel source active is that kernel now has the PID (137 above, which is klogd) attached.

Okay, thanks, I'll just leave the unix-dgram("/dev/log") in there instead then. I was hoping syslog-ng had found a more graceful way.

 

[Butterfly Bones]

Actually, skynet already does that for you.

The Skynet logs get moved to a file named skynet.log in the skynet directory on the root of your usb device. Other skynet "events" are moved to a file named events.log in the same location.

I symlinked those to /opt/var/log to those so I can read all the logs in one place

ln -s /mnt/[usb name]/skynet/skynet.log /opt/var/log/skynet.log
ln -s /mnt/[usb name]/skynet/events.log /opt/var/log/events_skynet.log

(smacking self in forehead over and over)

Thank you! I knew about the skynet.log, but somehow never paid attention and saw the events.log when I looked that directory. This solves my efforts to resolve this for days. Ugh. Now to figure out what /jffs/scripts file to place this in to it happens after any reboot.

 

[cmkelley]

(smacking self in forehead over and over)

Thank you! I knew about the skynet.log, but somehow never paid attention and saw the events.log when I looked that directory. This solves my efforts to resolve this for days. Ugh. Now to figure out what /jffs/scripts file to place this in to it happens after any reboot. I decided to use /jffs/configs/firewall-start.add to it symlinks recreated when firewall restarts.

EDIT - Correction, it works in /jffs/scripts/firewall-start.add so that symlinks are recreated when firewall restarts.

Ahh, I think you don't have to keep re-creating it. I created it once and it sticks.

As far as I know, firewall-start.add is not a file that is sourced, but it's late and I'm going from memory.

 

[thelonelycoder]

Think about md5 comparison for upgrades

I have the same dilemma with Diversion for md5 comparison. Since Diversion is not a single-file script and each installation might have Diversion files present or not depending on which edition is installed and what services are enabled.
I have solved this by only comparing the main file (diversion) to the installed file. Since I change the release date in the file every time I upload it, the md5 hash will be different and trigger the update function built into Diversion.

For the internal check to trigger the update, I use this in function state_check. That only works because my install command does not directly move the file to its final location as you and others do.

if ! cmp -s "$0" "/opt/bin/$SELF"; then
...
fi

The date change trick works for the local-remote check as well which is coming in the next Diversion update and will also be used in amtm.
Implementing an md5 check is really simple. Look at Adamms or Jacks code, or the su code for each script in amtm.

In Diversion the code looks thus:

localmd5="$(md5sum "/opt/bin/$SELF" | awk '{print $1}')"
remotemd5="$(curl -fsL --retry 3 "$DIVERSION_URL/$VERSION/$SELF" | md5sum | awk '{print $1}')"
if [ "$localmd5" != "$remotemd5" ]; then
    echo "$INFO A minor update is available for $NAME,"
    echo "$SPACE no version change."
    while true;do
        printf "\\n Do you want to update now? [1=Yes e=Exit] ";read -r continue
        case "$continue" in
            1)    VERSION=$S_VERSION;action=update;break;;
         [Ee])    exit_message;reload_menu;break;;
            *)    printf "\\n input is not an option\\n";;
        esac
    done
else
    echo "$INFO No update found for $NAME"
    while true;do
        printf "\\n Do you want to force update? [1=Yes e=Exit] ";read -r continue
        case "$continue" in
            1)    VERSION=$S_VERSION;action=update;break;;
         [Ee])    exit_message;reload_menu;break;;
            *)    printf "\\n input is not an option\\n";;
        esac
    done
fi

That's after I check for a version number change BTW.

2.0 will include a menu system

That's when it's time to add scribe to amtm.

 

[elorimer]

I'm pretty torn between letting klogd run and removing the kernel source from syslog-ng.conf or just killing klogd.

I've changed to src{system();}, and I've changed my rc.syslog-ng.func to:

 # kill any/all running syslogd
    [ -n "$(pidof syslogd)" ] && killall syslogd
    [ -n "$(pidof klogd)" ] && killall klogd

I've run this almost 24 hours without erratic output. In that time klogd hasn't restarted. In my experiments leaving the src to internal and the unix stream/dgram was essentially the same.

So I think this way would work. @Cam has run for a while the other way. I think it could be either, possibly dependent on how @cmkelley wants to handle updates to scribe and syslog-ng. Syslog-ng is now at 3.20 so an update in the entware channel will happen sooner or later.

 

[cmkelley]

I've changed to src{system();}, and I've changed my rc.syslog-ng.func to:

 # kill any/all running syslogd
    [ -n "$(pidof syslogd)" ] && killall syslogd
    [ -n "$(pidof klogd)" ] && killall klogd

I've run this almost 24 hours without erratic output. In that time klogd hasn't restarted. In my experiments leaving the src to internal and the unix stream/dgram was essentially the same.

So I think this way would work. @Cam has run for a while the other way. I think it could be either, possibly dependent on how @cmkelley wants to handle updates to scribe and syslog-ng. Syslog-ng is now at 3.20 so an update in the entware channel will happen sooner or later.

This would appease my tendency for cleaner code, and fewer things running, but @Cam reported something restarted klogd. I've added a thread asking people who know a thing or two about the firmware code what might restart syslogd and/or klogd.

Entware seems to be running ~4 months between large-scale updates.

 

[elorimer]

@Cam helpfully posted a link to the source code:

start_logger(void)
{
_dprintf("%s:\n", __FUNCTION__);
start_syslogd();
start_klogd();

So I'm thinking whatever starts klogd will also start syslogd.

 

[cmkelley]

@Cam helpfully posted a link to the source code:

start_logger(void)
{
_dprintf("%s:\n", __FUNCTION__);
start_syslogd();
start_klogd();

So I'm thinking whatever starts klogd will also start syslogd.

Yeah, RMerlin confirmed it and apparently there's a fair amount that will. Lemme dig into it, I would rather use the system() call in the end.

 

[cmkelley]

I have the same dilemma with Diversion for md5 comparison. Since Diversion is not a single-file script and each installation might have Diversion files present or not depending on which edition is installed and what services are enabled.
I have solved this by only comparing the main file (diversion) to the installed file. Since I change the release date in the file every time I upload it, the md5 hash will be different and trigger the update function built into Diversion.

Actually, I was thinking more md5 for updating the helper files for syslog-ng and logrotate. I haven't really decided on that though.

For the internal check to trigger the update, I use this in function state_check. That only works because my install command does not directly move the file to its final location as you and others do.

if ! cmp -s "$0" "/opt/bin/$SELF"; then
...
fi

The date change trick works for the local-remote check as well which is coming in the next Diversion update and will also be used in amtm.
Implementing an md5 check is really simple. Look at Adamms or Jacks code, or the su code for each script in amtm.

In Diversion the code looks thus:

localmd5="$(md5sum "/opt/bin/$SELF" | awk '{print $1}')"
remotemd5="$(curl -fsL --retry 3 "$DIVERSION_URL/$VERSION/$SELF" | md5sum | awk '{print $1}')"
if [ "$localmd5" != "$remotemd5" ]; then
    echo "$INFO A minor update is available for $NAME,"
    echo "$SPACE no version change."
    while true;do
        printf "\\n Do you want to update now? [1=Yes e=Exit] ";read -r continue
        case "$continue" in
            1)    VERSION=$S_VERSION;action=update;break;;
         [Ee])    exit_message;reload_menu;break;;
            *)    printf "\\n input is not an option\\n";;
        esac
    done
else
    echo "$INFO No update found for $NAME"
    while true;do
        printf "\\n Do you want to force update? [1=Yes e=Exit] ";read -r continue
        case "$continue" in
            1)    VERSION=$S_VERSION;action=update;break;;
         [Ee])    exit_message;reload_menu;break;;
            *)    printf "\\n input is not an option\\n";;
        esac
    done
fi

That's after I check for a version number change BTW.


That's when it's time to add scribe to amtm.

I've already stolen a reasonable amount of code from you, Jack, and Adamm. No sense re-inventing the wheel.

 

[Jack Yaz]

Actually, I was thinking more md5 for updating the helper files for syslog-ng and logrotate. I haven't really decided on that though.


I've already stolen a reasonable amount of code from you, Jack, and Adamm. No sense re-inventing the wheel.

As long as credit is given, carry on

 

[cmkelley]

scribe v0.9_0 pushed.

Lots of changes, unfortunately, actually uninstalling and re-installing is your best bet. Forced install should work, if force everything, but you may still end up with some side effects, such as having both a "crashes" and "crash" filter in syslog-ng.d ... "crashes" was the only plural-named filter and of course it bugged the crap out of me so I had to fix it.

Other changes;

• an untested script in the share directories for filtering the "normal" firewall output and rotating those logs when Skynet is NOT installed.
• the "non-scl" syslog-ng.conf is now always installed, due to a conflict with syslog-ng and klogd, and just because it seemed a better choice. The original conf file is moved to the share directory with -opkg appended.
• related, the source kernel directive has been removed from all syslog-ng files.
• a line is now added to /jffs/scripts/service-event that kills klogd and syslogd whenever they are restarted.
• everything in the /etc/*.d and share directories is now chmod 600. Note the logrotate won't process files that are group or other writeable (644 would work also).
• logrotate now has a log file of it's own as well, this is where errors in processing the logrotate.d/* files will show up.
• corrected typos, omissions, and other errors in /etc/*.d and share directories.
• removing more stuff in the uninstall routine

It's late Sunday on the west coast as I post this so if this breaks something I failed to test, you will have a bit of a wait before I can look at it. :-/

 

[cmkelley]

As long as credit is given, carry on

Admittedly I didn't credit specific lines/sections. I probably didn't lift much stuff verbatim, but certainly was inspired by you and others. There is a credit line in the script however.

If someone does feel slighted or that I should extend specific credit somewhere, PLEASE feel free to PM me and I'll do the best that I can to ensure proper credit.

 

[Jack Yaz]

Admittedly I didn't credit specific lines/sections. I probably didn't lift much stuff verbatim, but certainly was inspired by you and others. There is a credit line in the script however.

If someone does feel slighted or that I should extend specific credit somewhere, PLEASE feel free to PM me and I'll do the best that I can to ensure proper credit.

The credit in the readme is good enough for me

I was just flicking through your source to see if there's anything I can learn for my own scripting, which is one of the reasons that I love open source, so that everyone can benefit from the knowledge of others

 

[cmkelley]

The credit in the readme is good enough for me

I was just flicking through your source to see if there's anything I can learn for my own scripting, which is one of the reasons that I love open source, so that everyone can benefit from the knowledge of others

I'm quite certain you'll recognize some stylistic elements of yours I've adopted.

 

[cmkelley]

V0.9_1 now ... I outsmarted myself with sed and broke something (adding the line to S01syslog-ng) that had been working. Fixed now. If you installed 0.9_0, you'll need to force install to get it correct.