Scribe scribe - syslog-ng and logrotate installer

[elorimer]

From my skynet-0 log:

Apr 10 11:40:06 RT-AC56R kernel: DROP IN=eth0 OUT= MAC=2c:56:dc:54:63:00:00:7e:95:41:90:19:08:00 SRC=104.154.189.224 DST=96.58.128.193 LEN=85 TOS=0x00 PREC=0x00 TTL=53 ID=744 DF PROTO=TCP SPT=50672 DPT=45588 SEQ=1981578023 ACK=2819228164 WINDOW=553 RES=0x00 ACK PSH URGP=0 OPT (0101080A77EA37630BD56CD7) 
Apr 10 11:40:12 RT-AC56R Skynet: [#] 204517 IPs (+0) -- 2004 Ranges Banned (+0) || 0 Inbound -- 0 Outbound Connections Blocked! [start] [119s]
Apr 10 11:40:50 RT-AC56R kernel: DROP IN=eth0 OUT= MAC=2c:56:dc:54:63:00:00:7e:95:41:90:19:08:00 SRC=173.194.208.125 DST=96.58.128.193 LEN=70 TOS=0x00 PREC=0x00 TTL=118 ID=54256 PROTO=TCP SPT=5222 DPT=61676 SEQ=3450784330 ACK=3179351133 <4>DROP IN=eth0 OUT= MAC=2c:56:dc:54:63:00:00:7e:95:41:90:19:08:00 SRC=173.194.208.125 DST=96.58.128.193 LEN=70 TOS=0x00 PREC=0x00 TTL=118 ID=54768 <4>[BLOCKED - INBOUND] IN=eth0 OUT= MAC=2c:56:dc:54:63:00:00:7e:95:41:90:19:08:00 SRC=89.248.174.3 DST=96.58.128.193 LEN=40 TOS=0x00 PREC=0x00 TTL=236 ID=54321 PROTO=TCP SPT=55625 DPT=502 SEQ=1967937840 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 MARK=0x81800000 
Apr 10 11:42:38 RT-AC56R kernel: [BLOCKED - INBOUND] IN=eth0 OUT= MAC=2c:56:dc:54:63:00:00:7e:95:41:9

So there are the three things the filter looks for (any of which return true for the filter).

 

[Butterfly Bones]

Can we use such a thing as program ("Skynet") ... From what i understand Skynet is script rather than a daemon. The skynet logs are mostly kernel messages from what i can see.

That is what I have been trying to figure out, how to get more accurate Skynet reporting, especially for Loggly. See a couple posts above from last night. And yes, Skynet is a script that enhances firewall use, so they are kernel logs via iptables rules.

@elorimer posted an addition to the skynet filter file above, but it did not change the logging that I am seeking, and I have a mental block understanding script language (wrong side brain maybe). If you come up with anything, please post it.

 

[elorimer]

That is what I have been trying to figure out, how to get more accurate Skynet reporting, especially for Loggly. See a couple posts above from last night. And yes, Skynet is a script that enhances firewall use, so they are kernel logs via iptables rules.

Two separate things are at work, I think. The first is output from the firewall script (Skynet). It is sprinkled with logger instructions with option "-st Skynet" or "-t Skynet", depending on whether it is intending to log an error or not. It looks like the system logger is sending such a message with the tag where syslog-ng interprets the name of the program to be.

The second is the output of iptables, set up by Skynet to send LOG messages. My understanding is the system logger sends these to the kmsg file. Syslog-ng reads those messages from the file, and sticks in "kernel" as the program.

 

[tomsk]

The second is the output of iptables, set up by Skynet to send LOG messages. My understanding is the system logger sends these to the kmsg file. Syslog-ng reads those messages from the file, and sticks in "kernel" as the program.

yes that's set up in the syslog-ng.conf

source kernel {
        file("/proc/kmsg" program_override("kernel"));
};

program-override()
Description: Replaces the ${PROGRAM} part of the message with the parameter string. For example, to mark every message coming from the kernel, include the program-override("kernel") option in the source containing /proc/kmsg.
/QUOTE]

 

[Butterfly Bones]

Any idea why this don't get filtered by syslog-ng?

Apr 10 12:50:56 RT-AC86U-4608 (ChkWAN.sh): 19642 v1.11 Monitoring WAN connection using PING method to 1.1.1.1 check FAILED
Apr 10 12:51:11 RT-AC86U-4608 (ChkWAN.sh): 19642 ***ERROR WGET 'http://proof.ovh.net/files/100Mb.dat' transfer FAILED RC=7

These do get filtered.

Apr 10 12:51:26 RT-AC86U-4608 (ChkWAN.sh): 19642 cURL 12MByte transfer took: 00:00.00 secs @2514584 B/sec

Here is the chkwan file for syslog-ng.

# put ChkWAN messages into /opt/var/log/chkwan.log
destination d_chkwan {
    file("/opt/var/log/chkwan.log");
};
filter f_chkwan {
    program("ChkWAN") and
    message("transfer took:");
};
log {
    source(src);
    source(kernel);
    filter(f_chkwan);
    destination(d_chkwan);
    flags(final);
};
#eof
[/code

 

[tomsk]

Any idea why this don't get filtered by syslog-ng?

Apr 10 12:50:56 RT-AC86U-4608 (ChkWAN.sh): 19642 v1.11 Monitoring WAN connection using PING method to 1.1.1.1 check FAILED
Apr 10 12:51:11 RT-AC86U-4608 (ChkWAN.sh): 19642 ***ERROR WGET 'http://proof.ovh.net/files/100Mb.dat' transfer FAILED RC=7

These do get filtered.

Apr 10 12:51:26 RT-AC86U-4608 (ChkWAN.sh): 19642 cURL 12MByte transfer took: 00:00.00 secs @2514584 B/sec

Here is the chkwan file for syslog-ng.

# put ChkWAN messages into /opt/var/log/chkwan.log
destination d_chkwan {
    file("/opt/var/log/chkwan.log");
};
filter f_chkwan {
    program("ChkWAN") and
    message("transfer took:");
};
log {
    source(src);
    source(kernel);
    filter(f_chkwan);
    destination(d_chkwan);
    flags(final);
};
#eof
[/code

The filter is an AND statement... so both conditions have to be met.....

 

[Butterfly Bones]

The filter is an AND statement... so both conditions have to be met.....

So this should work?

# put ChkWAN messages into /opt/var/log/chkwan.log
destination d_chkwan {
    file("/opt/var/log/chkwan.log");
};
filter f_chkwan {
    program("ChkWAN") and
    message("check FAILED") or
    message("transfer FAILED") or
    message("transfer took:");
};
log {
    source(src);
    source(kernel);
    filter(f_chkwan);
    destination(d_chkwan);
    flags(final);
};
#eof

 

[tomsk]

So this should work?

# put ChkWAN messages into /opt/var/log/chkwan.log
destination d_chkwan {
    file("/opt/var/log/chkwan.log");
};
filter f_chkwan {
    program("ChkWAN") and
    message("check FAILED") or
    message("transfer FAILED") or
    message("transfer took:");
};
log {
    source(src);
    source(kernel);
    filter(f_chkwan);
    destination(d_chkwan);
    flags(final);
};
#eof

I think this should work

filter f_chkwan {
   program("ChkWAN") and
   (message("check FAILED") or
   message("transfer FAILED") or
   message("transfer took:"));
};

 

[Butterfly Bones]

I think this should work

filter f_chkwan {
   program("ChkWAN") and
   (message("check FAILED") or
   message("transfer FAILED") or
   message("transfer took:"));
};

Thank you, I wish I could wrap my head around this scripting thing...

 

[elorimer]

Thank you, I wish I could wrap my head around this scripting thing...

I thought we had worked out that program() here had to be "(ChkWAN.sh)"?

 

[Cam]

I'm on the trail of a puzzle about the skynet logging. With the customary filter:[....
Anyone else seeing this?

Same issue here with Skynet logs.
I'm guessing the "<4>" part is the syslog facility number, but as to why some long lines are truncated, I can't seem to get to bottom of it either. I tried dumping all kernel mssages separately into a log as per below, but still see the same issue of multiple lines with "<4>" in between, in that log also.

destination d_kernel {
  file("/opt/var/log/kernel.log");
};
log {
    source(kernel);
    destination(d_kernel);
};
#eof

 

[elorimer]

@Cam, I think you are right about the <4>. Here is a sequence from syslog-ng -Fevd:

[2019-04-10T22:46:48.753535] Incoming log entry; line='<4>[BLOCKED - INBOUND] IN=eth0 OUT= MAC=2c:56:dc:54:63:00:00:95:41:90'
[2019-04-10T22:46:48.754430] Initializing destination file writer; template='/opt/var/log/messages', filename='/opt/var/log/messages'
[2019-04-10T22:46:48.755171] Outgoing message; message='Apr 10 22:46:48 RT-AC56R kernel: [BLOCKED - INBOUND] IN=eth0 OUT= MAC=2c:56:dc:54:63:00:00:95:41:90\x0a'

So the input going to the syslog-ng parser is having added to it the time, the host, and the program (based on the syslog facility). I'm not following the addition of the "\xoa" though, or why this template is referenced.

Also, if 4 is the facility, I'm not following that either:

            0             kernel messages
             1             user-level messages
             2             mail system
             3             system daemons
             4             security/authorization messages
             5             messages generated internally by syslogd
             6             line printer subsystem
             7             network news subsystem
             8             UUCP subsystem
             9             clock daemon
            10             security/authorization messages
            11             FTP daemon
            12             NTP subsystem

 

[Cam]

Well I've noticed the if I kill klogd then the messages all seem to settle. Perhaps a race condition with klog and syslog-ng reading /proc/kmsg at same time.

Then I found that Syslog-ng recommend not having klog running, though not sure what the impact of that is on an embedded system.

In any case, for the past 30 min or so, my system seems ok with no issues in the messages or skynet-0.log as before. It remains to be seen as to whether I miss some other logs in "messages", but time will tell I guess.

 

[Butterfly Bones]

I thought we had worked out that program() here had to be "(ChkWAN.sh)"?

I tried both ways and either one works the same. I took the .sh out hoping to change the display at Loggy, but it is the same.

 

[Cam]

I tried both ways and either one works the same. I took the .sh out hoping to change the display at Loggy, but it is the same.

Have you tried with just the basic filter?

filter f_chkwan {
program("ChkWAN");
};

Sent from my Nokia 7 plus using Tapatalk

 

[Butterfly Bones]

Have you tried with just the basic filter?

filter f_chkwan {
program("ChkWAN");
};

Sent from my Nokia 7 plus using Tapatalk

No, but I will. If you have not seen by now, thank you. Since I just can't grasp scripting language, these suggestions really help.

 

[Butterfly Bones]

Have you tried with just the basic filter?

filter f_chkwan {
program("ChkWAN");
};

Sent from my Nokia 7 plus using Tapatalk

That much simpler filter works very well, but does not change what Loggy displays. I suspect this is because the ChkWAN script uses logger to display the lines in syslog.

Apr 10 18:25:15 RT-AC86U-4608 (ChkWAN.sh): 20003 ***ERROR WGET 'http://proof.ovh.net/files/100Mb.dat' transfer FAILED RC=7
Apr 10 18:25:15 RT-AC86U-4608 (ChkWAN.sh): 20003 v1.11 Monitoring WAN connection using cURL data IP retrieval method check FAILED
Apr 10 18:25:33 RT-AC86U-4608 (ChkWAN.sh): 20003 cURL 12MByte transfer took: 00:00.00 secs @726195 B/sec
Apr 10 18:30:05 RT-AC86U-4608 (ChkWAN.sh): 20595 cURL 12MByte transfer took: 00:00.00 secs @2497003 B/sec

 

[skeal]

Hey guys what log file should have the hourly skynet entry? When I look at my skynet-0.log they are not there. It just shows the log drops and startup stuff.

 

[Butterfly Bones]

Hey guys what log file should have the hourly skynet entry? When I look at my skynet-0.log they are not there. It just shows the log drops and startup stuff.

Did you redirect Skynet to place logs into /opt/var/log?
One easy way to see it run "scribe debug" and look at that file. I have found if you go into Skynet settings and check it does not show.

Oops, I totally screwed that up!

The don't show in the logs, only in Skynet stats search. I posted earlier about that, and no one seems to really know the answer. I see parts of them in my /opt/var/log/skynet-0.log, but it skips many. I don't know at this point how to get and show them using syslog-ng.

 

[Butterfly Bones]

continuing...

The problem is that Skynet deletes the skynet-0.log each hour and starts a new one. I posted some examples of this yesterday. I just ran a grep on that log and see this.

Apr 10 18:00:03 RT-AC86U-4608 Skynet: [#] 154538 IPs (+0) -- 1634 Ranges Banned (+0) || 196 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 10 19:00:03 RT-AC86U-4608 Skynet: [#] 154538 IPs (+0) -- 1634 Ranges Banned (+0) || 286 Inbound -- 0 Outbound Connections Blocked! [save] [3s]
Apr 10 20:00:03 RT-AC86U-4608 Skynet: [#] 154538 IPs (+0) -- 1634 Ranges Banned (+0) || 390 Inbound -- 0 Outbound Connections Blocked! [save] [3s]

I tried for "Skynet: [#]" but grep failed, so I just did "cat /opt/var/log/skynet-0.log | grep [#]' and got the above. I've been trying to make a filter that will pull those as they appear, but my script skills are not working, duh!

We need one that will work with "Skynet: [#]" before the skynet filter and then process the existing skynet filter that has the final flag and stops processing.