Scribe scribe - syslog-ng and logrotate installer

[elorimer]

@Butterfly Bones, in looking at your graphs, I decided not to send pixelserv-tls logs to loggly. I keep track of them separately, and they distort the graphs in loggly. Same for skynet.

So I created filters for the 0loggly file:

filter f_loggly1{ not program("pixelserv-tls"); };
filter f_loggly2{ not program("Skynet"); };

And added it to the log statement:

log {
    source(src);
    filter(f_loggly1);
    filter(f_loggly2);
    destination(d_loggly);
};

The free tier in loggly has a size limit, and that might help there, too.

 

[elorimer]

Something new here, @cmkelley. You've added the -v argument to the S01syslog-ng file (through the rc.func.syslog-ng). I'm curious about your thinking.

Also, one of the things we found over the weekend was that scribe restart could look like it started okay, but scribe status would show that syslog-ng failed if it encountered a config error. Perhaps you should have scribe restart include a call to scribe status to show everything is ok.

 

[cmkelley]

Something new here, @cmkelley. You've added the -v argument to the S01syslog-ng file (through the rc.func.syslog-ng). I'm curious about your thinking.

I think errors that are not caught by the -s parameter show up the the -v. It was late when I added that, maybe there's a better choice. It does log a lot though.

Also, one of the things we found over the weekend was that scribe restart could look like it started okay, but scribe status would show that syslog-ng failed if it encountered a config error. Perhaps you should have scribe restart include a call to scribe status to show everything is ok.

Good Idea. I'll add that.

 

[elorimer]

I think errors that are not caught by the -s parameter show up the the -v.

If a good configuration is working, I'm not sure it is necessary. Also, since this will occur when syslogd is stopped, and syslog-ng isn't starting, where does the log go? If the status returns a bad start, then it is better to just run a trial of syslog-ng with the -d command. Maybe that could be an option ("trial-start").

 

[cmkelley]

If a good configuration is working, I'm not sure it is necessary. Also, since this will occur when syslogd is stopped, and syslog-ng isn't starting, where does the log go? If the status returns a bad start, then it is better to just run a trial of syslog-ng with the -d command. Maybe that could be an option ("trial-start").

I wonder if trial-start return an exit code?

 

[Martineau]

If a good configuration is working, I'm not sure it is necessary. Also, since this will occur when syslogd is stopped, and syslog-ng isn't starting, where does the log go? If the status returns a bad start, then it is better to just run a trial of syslog-ng with the -d command. Maybe that could be an option ("trial-start").

When users choose to restart scribe and it doesn't actually start, then I use

syslog-ng -Fevd

which then shows the failing syslog log to the terminal...also very useful for debugging the rules as should in my screenshot

This could be an option.

 

[cmkelley]

When users choose to restart scribe and it doesn't actually start, then I use

syslog-ng -Fevd

which then shows the failing syslog log to the terminal...also very useful for debugging the rules as should in my screenshot

This could be an option.

That sounds better. It's going to be a couple days before I can do any serious work on it though.

 

[skeal]

Ok here I go then using AX88U on the new alpha.

 

[skeal]

Ok here I go then using AX88U on the new alpha.

Ok everything is configured and is running. Why doesn't the command:

tail /opt/var/log/skynet-0.log

Continue mine shows the last few entries and then exits and back to command prompt, shouldn't tail keep running?

 

[skeal]

How do I get the putty display to keep scrolling?

 

[skeal]

It's not putty because I tried following logs in skynet ad Diversion and it scrolls.

 

[Butterfly Bones]

Ok everything is configured and is running. Why doesn't the command:

tail /opt/var/log/skynet-0.log

Continue mine shows the last few entries and then exits and back to command prompt, shouldn't tail keep running?

You need to use

tail -f /opt/var/log/skynet-0.log

Mine will stop after a few hours, but I found using -F instead of -f keeps it going.

 

[skeal]

You need to use

tail -f /opt/var/log/skynet-0.log

Mine will stop after a few hours, but I found using -F instead of -f keeps it going.

Bingo. I did this with a stuck "n" key on my keyboard....man what a drag!! This is awesome guys!! Great work!

 

[skeal]

This may be asking a lot but it would be cool to have the skynet color palette for blocked traffic. Is this out to lunch or...?

 

[elorimer]

Here's a new thing.

We're now starting syslog-ng with the -verbose option, so it kicks out informational messages and sends them to the log. I'm finding I get this message frequently repeated for two files (my pixelserv-tls and skynet files, both getting hit multiple times a second but with significant gaps in between):

"Destination timed out, reaping; "

If I am following, this means that no new message has been written to a destination file in 60 seconds, so it is closing the file. I assume that will happen for every log file from time to time. Purely informational, but I don't want it clogging up my messages file, so for the moment I've added a filter to send them to the syslog-ng log file.

 

[cmkelley]

Here's a new thing.

We're now starting syslog-ng with the -verbose option, so it kicks out informational messages and sends them to the log. I'm finding I get this message frequently repeated for two files (my pixelserv-tls and skynet files, both getting hit multiple times a second but with significant gaps in between):

"Destination timed out, reaping; "

If I am following, this means that no new message has been written to a destination file in 60 seconds, so it is closing the file. I assume that will happen for every log file from time to time. Purely informational, but I don't want it clogging up my messages file, so for the moment I've added a filter to send them to the syslog-ng log file.

I'll remove the -v from next 8_3, it was a late addition I probably should have thought that thru more. If you want to change it now, just edit rc.func.syslog-ng to remove the option and then scribe restart.

 

[Butterfly Bones]

Here's a new thing.

We're now starting syslog-ng with the -verbose option, so it kicks out informational messages and sends them to the log. I'm finding I get this message frequently repeated for two files (my pixelserv-tls and skynet files, both getting hit multiple times a second but with significant gaps in between):

"Destination timed out, reaping; "

If I am following, this means that no new message has been written to a destination file in 60 seconds, so it is closing the file. I assume that will happen for every log file from time to time. Purely informational, but I don't want it clogging up my messages file, so for the moment I've added a filter to send them to the syslog-ng log file.

Ah ha!! That explains what I have seen. I was trying to dig and figure it out. I also often assume I have something fubar in one of my files. I'll use the temp fix that @cmkelley posted above.

 

[elorimer]

I'll remove the -v from next 8_3, it was a late addition I probably should have thought that thru more. If you want to change it now, just edit rc.func.syslog-ng to remove the option and then scribe restart.

I think that is best. There are a bunch of other messages -v kicks off, almost more than pixelserv!

 

[cmkelley]

If you haven't already, I recommend integrating Codacy and/or TravisCI to monitor your script using shellcheck. They can help catch potential errors before a user does!

Heh, I do all my coding on the router in a putty session. I try to remember to run the script through shellcheck before I upload it, but sometimes I forget. I do upload it through Git Desktop on my windows machine, but that's reading right off a "master" copy on my router that I maintain by hand. While I know better tools would help me in the long run, learning how to use them is another chunk of time. And I'm not sure I'll do much script writing after this.

Heck, I even ran into a problem with Git Desktop last night merging from my development branch (used to install on my test router).

 

[skeal]

Hey @cmkelley this is an amazing effort. Love the script and installer, filters are great too. Bravo mate!!