Scribe scribe - syslog-ng and logrotate installer

[cmkelley]

Okay, I said I wasn't going to do it tonight, but I did it anyway ...

V0.8_3 is available

• /opt/etc/init.d/rc.func.syslog-ng updated to remove ARGS="-v" - unfortunately, this file is not updated during a regular update. If you haven't modified any of the files in the /opt/etc/*.d/ directories, you can safely do an install, and force install. Only those base files copied by scribe will be over-written, any files moved into those directories will be untouched. Otherwise, after you have updated to v0.8_3, at the terminal prompt, you can cut and paste the below:

cp /opt/tmp/scribe-master/init.d/rc.func.syslog-ng /opt/etc/init.d  && scribe restart

which will copy the new rc.func.syslog-ng file and restart syslog-ng to stop verbose logging of syslog-ng stuff

• After running "scribe install" or "scribe restart", scribe will verify that syslog-ng is actually running. Thanks to @elorimer for the suggestion.
• Despite previous assertions by me to the contrary, update does not write anything to the /opt/etc/*.d/ directories. That is only done on install, or a forced re-install.

The files that are copied into the /opt/etc/*.d/ directories are:

syslog-ng.d/
crashes
syslogng
wlceventd

logrotate.d/
A00global
crash (I should probably rename either this one or "crashes" above to consistent with each other)
messages
syslogng

Happy logging!

 

[cmkelley]

This may be asking a lot but it would be cool to have the skynet color palette for blocked traffic. Is this out to lunch or...?

Skynet would have to output it's logging messages with the ansi color tags embedded. The problem with that is that people not viewing it on an ansi-capable device, or with a pager that doesn't support ansi, would find it difficult to read with all the color tags.

 

[Butterfly Bones]

I have a few niggles with Skynet, nothing really wrong with it or with scribe, I just am looking for a "tweak" if possible.

In Loggy reports when I drill down, I see the raw data coming in from syslog-ng. All Skynet events are attributed to "kernel" understandably with the way he logging works and the skyner filter for syslog-ng.

2019-04-08 21:31:59.000
[BLOCKED - INBOUND] IN=eth0 OUT= MAC=88:d7:f6:1d:46:08:00:01:5c:6d:22:46:08:00 SRC=213.14.186.248 DST=[abc.de.fg.hij] LEN=40 TOS=0x00 PREC=0x00 TTL=42 ID=33757 PROTO=TCP SPT=12201 DPT=23 SEQ=1197290991 ACK=0 WINDOW=55873 RES=0x00 SYN URGP=0 MARK=0x8000000

{ syslog: { severity: "Warning", appName: "kernel", host: "RT-AC86U-xxxx", priority: "12", facility: "user-level messages", timestamp: "2019-04-08T21:31:59-07:00" } }]/code]

Is there any way to filter in the skynet filter to have the name skynet used as the appName instead of kernel?
Here is what the log entry looks like on the router (truncated since my x-term is not set for word wrap).

Apr  8 21:06:05 RT-AC86U-4608 kernel: [BLOCKED - INBOUND] IN=eth0 OUT= MAC=88:d7:f6:1d:46:08:00:01:5c:6d:22:46:08:00 SRC=74.82.47.35 DST=71.93.53.239 LEN=32 TOS=0x00 PREC=0x00 TTL=55 ID=22250 DF PROTO=UDP SPT=24583 DPT=10001

I know this is a picky little cosmetic thing, but I have seen the (almost) magic that a script guru can accomplish.

 

[elorimer]

Is there any way to filter in the skynet filter to have the name skynet used as the appName instead of kernel?

Syslog-ng has a bunch of other functions, including rewrite rules, parsers and templates, to restructure messages.

There are a couple of possibilities. First, if you just want to change the name in what goes to the syslog-0 file, you can use a rewrite rule in your syslog configuration file:

rewrite r_Skynet{
    set("Skynet", value("PROGRAM"));
};
destination d_skynet {
    file("/opt/var/log/skynet-0.log");
 
};
# logs everything from Skynet to /opt/var/log/skynet-0.log
filter f_skynet {
    program("Skynet") or
    message("BLOCKED -") or
    message("DROP IN=");
};
# final flag stops processing of messages matching the f_skynet filter
log {
    source(src);
    source(kernel);
    rewrite(r_Skynet);
    filter(f_skynet);
    destination(d_skynet);
    flags(final);
};

EDIT: The above doesn't work. The rewrite(f_Skynet): needs to be after the filter message. Otherwise it rewrites all messages as Skynet and then discards them.

Each log message that meets the filter will have "Skynet" put in place of the program when the message is written to the destination. Since the filter ORs the two message matches, it will replace "kernel" with Skynet there too. I don't know what happens when Skynet cleans this file, so we will have to see.

If you want to change the name for what goes to loggly, then I think you need an if-then, or break the loggly messages into two parts, one that meets the skynet filter and one that doesn't, because you don't want every message going to loggly to be rewritten with "Skynet" as the program name. I don't send my Skynet messages to loggly. Note that the loggly destination rewrites the message according to a template, and I'm not sure, if you set the PROGRAM field to "Skynet", whether the template will continue to see $PROGRAM as "kernel". You might fool around with it, or maybe @cmkelley's favorite manual has something.

 

[Butterfly Bones]

Thank you for that suggestion to filter Skynet, @elorimer. I'll try it later.

I have another thing going on with Skynet, I don't see it as a problem with scribe or Skynet, it is more education / information / wtf kind of thing. I'm going to tag @Adamm for feedback, since I think this is related to syslog-ng filtering and not a Skynet issue (why I don't post this in that thread), but Adamm's expertise is very much appreciated.

Why again does syslog-ng need to have the Skynet log redirected to /opt/var/log/skynet-0.log instead of skimming the regular syslog.log like it does for other filters? I've reread the conversation there on the destructive sed between cmkelley and Adamm, but it is a little over my script dunce head.


I love the stats search reports function and have always used this one almost daily -

sh /jffs/scripts/firewall stats search reports 50

Now when I run it either in a terminal of from within Skynet, I find many lines, hours missing from multiple days since setting up syslog-ng with scribe. Here is the one I ran at just before 5 am PDT this morning (yeah, sometimes I get up stupid early!).

Apr  5 17:00:03 Skynet: [#] 158940 IPs (+0) -- 1515 Ranges Banned (+0) || 56 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  5 18:00:03 Skynet: [#] 158940 IPs (+0) -- 1515 Ranges Banned (+0) || 167 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  5 19:00:03 Skynet: [#] 158940 IPs (+0) -- 1515 Ranges Banned (+0) || 275 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  6 08:32:46 Skynet: [#] 156033 IPs (+0) -- 1539 Ranges Banned (+0) || 0 Inbound -- 0 Outbound Connections Blocked! [start] [26s]
Apr  6 09:00:03 Skynet: [#] 156033 IPs (+0) -- 1539 Ranges Banned (+0) || 81 Inbound -- 0 Outbound Connections Blocked! [save] [2s]
Apr  6 10:00:02 Skynet: [#] 156033 IPs (+0) -- 1539 Ranges Banned (+0) || 173 Inbound -- 0 Outbound Connections Blocked! [save] [2s]
Apr  6 11:00:03 Skynet: [#] 156033 IPs (+0) -- 1539 Ranges Banned (+0) || 273 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  6 21:00:03 RT-AC86U-4608 Skynet: [#] 156033 IPs (+0) -- 1539 Ranges Banned (+0) || 88 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  6 22:00:03 RT-AC86U-4608 Skynet: [#] 156033 IPs (+0) -- 1539 Ranges Banned (+0) || 199 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 05:00:03 RT-AC86U-4608 Skynet: [#] 156033 IPs (+0) -- 1539 Ranges Banned (+0) || 965 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 06:00:03 RT-AC86U-4608 Skynet: [#] 156033 IPs (+0) -- 1539 Ranges Banned (+0) || 1074 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 06:25:28 RT-AC86U-4608 Skynet: [#] 154273 IPs (-1760) -- 1548 Ranges Banned (+9) || 1110 Inbound -- 0 Outbound Connections Blocked! [banmalware] [28s]
Apr  7 08:00:03 RT-AC86U-4608 Skynet: [#] 154273 IPs (+0) -- 1548 Ranges Banned (+0) || 1258 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 09:00:03 RT-AC86U-4608 Skynet: [#] 154273 IPs (+0) -- 1548 Ranges Banned (+0) || 1353 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 11:00:03 RT-AC86U-4608 Skynet: [#] 154273 IPs (+0) -- 1548 Ranges Banned (+0) || 134 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 14:00:03 RT-AC86U-4608 Skynet: [#] 154273 IPs (+0) -- 1548 Ranges Banned (+0) || 423 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 15:00:03 RT-AC86U-4608 Skynet: [#] 154273 IPs (+0) -- 1548 Ranges Banned (+0) || 543 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 17:00:03 RT-AC86U-4608 Skynet: [#] 154273 IPs (+0) -- 1548 Ranges Banned (+0) || 732 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 20:00:03 RT-AC86U-4608 Skynet: [#] 154273 IPs (+0) -- 1548 Ranges Banned (+0) || 1053 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  7 21:11:42 RT-AC86U-4608 Skynet: [#] 154273 IPs (+0) -- 1548 Ranges Banned (+0) || 1171 Inbound -- 0 Outbound Connections Blocked! [debug] [19s]
Apr  8 08:00:04 RT-AC86U-4608 Skynet: [#] 153702 IPs (+0) -- 1558 Ranges Banned (+0) || 2369 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  8 11:00:03 RT-AC86U-4608 Skynet: [#] 153702 IPs (+0) -- 1558 Ranges Banned (+0) || 2694 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  8 12:00:05 RT-AC86U-4608 Skynet: [#] 153702 IPs (+0) -- 1558 Ranges Banned (+0) || 2781 Inbound -- 0 Outbound Connections Blocked! [save] [5s]
Apr  8 16:00:03 RT-AC86U-4608 Skynet: [#] 153702 IPs (+0) -- 1558 Ranges Banned (+0) || 3190 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  8 17:00:03 RT-AC86U-4608 Skynet: [#] 153702 IPs (+0) -- 1558 Ranges Banned (+0) || 3277 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  8 19:24:02 RT-AC86U-4608 Skynet: [#] 153702 IPs (+0) -- 1558 Ranges Banned (+0) || 3507 Inbound -- 0 Outbound Connections Blocked! [stats] [2s]
Apr  8 19:25:25 RT-AC86U-4608 Skynet: [#] 154705 IPs (+1003) -- 1660 Ranges Banned (+102) || 3509 Inbound -- 0 Outbound Connections Blocked! [banmalware] [24s]
Apr  9 02:00:03 RT-AC86U-4608 Skynet: [#] 154705 IPs (+0) -- 1660 Ranges Banned (+0) || 4170 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr  9 04:49:00 RT-AC86U-4608 Skynet: [#] 154538 IPs (+0) -- 1634 Ranges Banned (+0) || 4550 Inbound -- 0 Outbound Connections Blocked! [stats] [3s]

Again, I have no reason to think Skynet is misbehaving in any way, since when I drill down to raw data in Loggly, I see eah BLOCKED or DROP lines (attributed to "kernel:" as per my post above).

Thank you everyone!

 

[elorimer]

Why again does syslog-ng need to have the Skynet log redirected to /opt/var/log/skynet-0.log instead of skimming the regular syslog.log like it does for other filters?

Skynet runs an hourly sed -i command on the file it is pointed to. If it points to /tmp/syslog.log when that is a symlink to the messages file, it doesn't follow the symlink. It reads the contents of that symlink file, does its work on it, and then overwrites it. Overwriting it destroys the link and replaces it with a text file. Two things happen then. First, the webgui is pointed to what is now an empty file, rather than following the link to the symlink. Second, syslogd isn't running, so that file never gets any new log messages, but syslog-ng is still running and still sending messages to the messages log file. (Somewhere in there, though, it might recover from memory a portion of the original syslog file. I haven't figured that out.)

The work around is to point skynet to the skynet-0 file that syslog-ng is creating. The hourly sed command works on that and scrubs out the BLOCKED and DROP messages and writes the stat line. You could set things up so that syslog-ng drops the BLOCKED and DROP messages, but then the stat line will be off.

Again, I have no reason to think Skynet is misbehaving in any way, since when I drill down to raw data in Loggly, I see eah BLOCKED or DROP lines (attributed to "kernel:" as per my post above).

So this is different. Remember your 0loggly config file is the first one processed. Syslog-ng takes each message and runs that config first, sending the message to loggly. Because there is no final flag, the message is then passed along to each of the following configs (with an added "Processed" tag we aren't making use of). When the syslog messages hits the syslog filter, it sends that to syslog-0 and because of the final flag stops processing it anymore. Hourly, skynet comes along and does its scrub, deleting all the BLOCKED and DROP lines. That's why those lines are in Loggly, and only the last hour's worth of those lines remain in the log file.

Because pixelserv-tls and skynet give me all the log information I need, and throw off a lot of messages, I filter those out before sending anything to loggly. (PS: who are calling me to upgrade my trial).

Apr 8 19:25:25 RT-AC86U-4608 Skynet: [#] 154705 IPs (+1003) -- 1660 Ranges Banned (+102) || 3509 Inbound -- 0 Outbound Connections Blocked! [banmalware] [24s]

Not sure what is going on here, since I don't use firewall stats command. I notice at some point you might have changed the name of the router (showing now a HOST field in the message), and I think stats runs the stat again (giving the non-hourly time). Not sure why you are missing other hourly reports.

 

[cmkelley]

Skynet runs an hourly sed -i command on the file it is pointed to. If it points to /tmp/syslog.log when that is a symlink to the messages file, it doesn't follow the symlink. It reads the contents of that symlink file, does its work on it, and then overwrites it. Overwriting it destroys the link and replaces it with a text file. Two things happen then. First, the webgui is pointed to what is now an empty file, rather than following the link to the symlink. Second, syslogd isn't running, so that file never gets any new log messages, but syslog-ng is still running and still sending messages to the messages log file. (Somewhere in there, though, it might recover from memory a portion of the original syslog file. I haven't figured that out.)

The work around is to point skynet to the skynet-0 file that syslog-ng is creating. The hourly sed command works on that and scrubs out the BLOCKED and DROP messages and writes the stat line. You could set things up so that syslog-ng drops the BLOCKED and DROP messages, but then the stat line will be off.

If you're going to drop them, you may as well turn debug mode off in Skynet so they aren't generated in the first place. Either way, Skynet won't generate stats. Pushing them to skynet-0 keeps the BLOCKED/DROPPED messages from clogging the messages, because you're more likely to go looking in message for something that happened in the last few minutes, not a few hours ago (where they would have been stripped).

Not sure what is going on here, since I don't use firewall stats command. I notice at some point you might have changed the name of the router (showing now a HOST field in the message), and I think stats runs the stat again (giving the non-hourly time). Not sure why you are missing other hourly reports.

The HOST field tells me when he started using syslog-ng. Syslog-ng prepends the host name to the logs so that if you send multiple systems' logs to a central server you can tell which system they came from. There's an option to turn that off, but I don't remember off the top of my head.

I see the same thing he does, and I can't figure it out, I get some of the "Ranges Banned" logs in my messages, some in skynet-0. They all should go to skynet-0 and stop, I don't understand why some are missed. Right now I don't mind seeing a few of the Ranges Banned entries in messages so I haven't spent any time looking into it. Pushing them to skynet-0 does cause the scrubbed skynet-0 file to slowly grow, since skynet doesn't scrub those messages. Maybe I should push those off into another file or just let the hourly messages go to the regular messages file.

 

[elorimer]

If you're going to drop them, you may as well turn debug mode off in Skynet so they aren't generated in the first place.

I just drop them from what gets sent to loggly. They get sent later to syslog-0.log.

Pushing them to skynet-0 does cause the scrubbed skynet-0 file to slowly grow, since skynet doesn't scrub those messages.

Logrotate can handle that.

 

[Butterfly Bones]

Thank you @cmkelley and @elorimer, your discussion if over my head right now, but the more I immerse myself, the more I learn. I'm gaining, I think.

Digging back into Skynet and especially Adamm's GitHub, I found this command:

Skynets own log file to reduce spam (cat /tmp/mnt/<USB_drive>/skynet/skynet.log | more)

and running that in a terminal, I see all the lines back into Apr. 2, but without the [save]. [banmalware] [debug] [whitelist ] [stats] summary lines. That confirms that Skynet is working as designed / expected, it is what syslog-ng filtering is effecting with the "stats search reports" output. Now, what to do with that, if anything.

 

[cmkelley]

Thank you @cmkelley and @elorimer, your discussion if over my head right now, but the more I immerse myself, the more I learn. I'm gaining, I think.

Digging back into Skynet and especially Adamm's GitHub, I found this command:

Skynets own log file to reduce spam (cat /tmp/mnt/<USB_drive>/skynet/skynet.log | more)

and running that in a terminal, I see all the lines back into Apr. 2, but without the [save]. [banmalware] [debug] [whitelist ] [stats] summary lines. That confirms that Skynet is working as designed / expected, it is what syslog-ng filtering is effecting with the "stats search reports" output. Now, what to do with that, if anything.

I'm not quite sure I follow what you're saying, but here's what happens:

Before syslog-ng, all of the Skynet (iptables, really) output (BLOCKED, DROPPED, and the stats lines) go into /tmp/syslog.log. Every hour (?) Skynet scrapes that log and moves the BLOCKED and DROPPED messages to /tmp/mnt/<USB_drive>/skynet/skynet.log which has the effect of making the last hour's bit of system logging rather difficult because of the number of BLOCKED/DROPPED messages.

Enter syslog-ng with the appropriate filter; all of the Skynet output (is supposed to*) go into /opt/var/log/skynet-0.log. Skynet scrapes that hourly and moves the BLOCKED and DROPPED messages to /tmp/mnt/<USB_drive>/skynet/skynet.log for Skynet's processing. This leaves behind the stats lines, but those are small and don't fill the log very fast. /opt/var/log/messages (i.e. the default log files for everything that isn't filtered out with a "final" flag) doesn't get any of the Skynet logs.

* I say "supposed to" because as noted above, for some reason not all of the stats lines go into skynet-0.log. Some go into messages. The only explanation I can think of is that perhaps syslog-ng is trying to write the skynet stats line to skynet-0.log at the same time that Skynet is scrubbing skynet-0.log and instead of waiting patiently or losing the log, it just writes it to messages and goes on it's merry way.

What do you do with that information? You look at the number of ads blocked occasionally and thank Adamm for saving you from all those ads.

 

[skeal]

Hey fellas I need a filter for "dropbear" and "adaptive qos" any Chance someone could coach me? Or if you have examples that show how to make on up? Thanks guys!!

 

[Butterfly Bones]

I'm not quite sure I follow what you're saying, but here's what happens:

Yes, I understand the mechanics of the process by syslog-ng, I'm just trying to figure out a way to get complete stats. Not worth anyone chasing their tail like I am.

* I say "supposed to" because as noted above, for some reason not all of the stats lines go into skynet-0.log. Some go into messages. The only explanation I can think of is that perhaps syslog-ng is trying to write the skynet stats line to skynet-0.log at the same time that Skynet is scrubbing skynet-0.log and instead of waiting patiently or losing the log, it just writes it to messages and goes on it's merry way.

What do you do with that information? You look at the number of ads blocked occasionally and thank Adamm for saving you from all those ads.

I see that the /opt/var/log/skynet-0.log is purged often, I need to remind myself to check at a minute before and a minute after to see Skynet scrubbing and running the [save] summary is what does that. I know if I go into Skynet via the amtm menu, and run and debug or stats commands, the log is wiped but quickly regenerates.

/tmp/home/root# ls -al /opt/var/log/
-rw-------    1 username root         32153 Apr  9 10:56 skynet-0.log
-rw-------    1 username root           900 Apr  9 12:00 skynet-0.log
-rw-------    1 username root          9564 Apr  9 13:16 skynet-0.log

You mean "bot probes" not "ads", right?

 

[L&LD]

Hey fellas I need a filter for "dropbear" and "adaptive qos" any Chance someone could coach me? Or if you have examples that show how to make on up? Thanks guys!!

https://www.snbforums.com/threads/s...nstaller-v0-8-3-beta.55853/page-6#post-478916

The post above by the wonderfully brave and experimental Butterfly Bones should be all you need.

 

[cmkelley]

Hey fellas I need a filter for "dropbear" and "adaptive qos" any Chance someone could coach me? Or if you have examples that show how to make on up? Thanks guys!!

First thing I'd try for dropbear is:

# log all dropbear logs to /opt/var/log/dropbear.log and stop processing dropbear logs

destination d_dropbear {
   file("/opt/var/log/dropbear.log");
};

filter f_dropbear {
   program("dropbear");
};

log {
   source(src);
   source(kernel);
   filter(f_dropbear);
   destination(d_dropbear);
   flags(final);
};

#eof

I don't know how adaptive qos logs its stuff, but some variation on the above should work for it I'd think.

 

[cmkelley]

You mean "bot probes" not "ads", right?

Yeah, that.

 

[Butterfly Bones]

Hey fellas I need a filter for "dropbear" and "adaptive qos" any Chance someone could coach me? Or if you have examples that show how to make on up? Thanks guys!!

I use an existing script from cmkelley or elorimer and then use my editor to find and replace; then compare the two side by side. Except for a spelling error, that has worked for me. I could help with the "dropbear" one (though I leave those along in case of intrusion attempts, and there are so few).

I don't use "adaptive qos" I could try, but with my scripting "skills" probably best to confirm with a pro.

 

[Butterfly Bones]

https://www.snbforums.com/threads/s...nstaller-v0-8-3-beta.55853/page-6#post-478916

The post above by the wonderfully brave and experimental Butterfly Bones should be all you need.

Do you have any idea how many times I have shot myself in the foot with syslog-ng bumbling??

 

[L&LD]

Do you have any idea how many times I have shot myself in the foot with syslog-ng bumbling??

And that is why you're so far ahead than me!

I have to admit I have installed the scribe script for a few days now, but really, I don't know what to do with it. I fire it up in PuTTY and see the options (I've even tried them all) and I still am stuck.

Do I have to do anything past installing and seeing it is 'alive'?

 

[cmkelley]

And that is why you're so far ahead than me!

I have to admit I have installed the scribe script for a few days now, but really, I don't know what to do with it. I fire it up in PuTTY and see the options (I've even tried them all) and I still am stuck.

Do I have to do anything past installing and seeing it is 'alive'?

No, except copy the filters that are relevant to you from /opt/share/*/examples to their appropriate directories. Once it's installed you only need look at the logs, until you decide you want to filter something else out of the main "messages" log.

 

[L&LD]

No, except copy the filters that are relevant to you from /opt/share/*/examples to their appropriate directories. Once it's installed you only need look at the logs, until you decide you want to filter something else out of the main "messages" log.

Copy? I can do that! Thank you.