Scribe scribe - syslog-ng and logrotate installer

[elorimer]

I actually just looked at the commit history, and I have zero idea anymore why I did that.

I do, except it might have been better the first way:

When using multiple filters one after the other, or connecting filters with the logical AND/OR operators, the order of filters has a significant impact on performance. We recommend to put those filters in front that are the most likely to match the incoming log messages (if known).

Quoting from https://support.oneidentity.com/tec...mance-guideline-for-syslog-ng-premium-edition

Which leads to a different thought about performance. Syslog-ng processes the files in alphabetic order, so anything that hits the first filter doesn't get processed by any other filter, and anything that goes to messages has been processed by every other filter. I don't know that the rate of logging and the power of the router are that mismatched, but syslog-ng would be fastest if we ordered the filters so that, say, pixelserv was processed before openvpn, and maybe skynet or unbound before everything.

EDIT: Nope, not as simple as renaiing the config files without renaming the log files the way uiScribe is setup, and then with skynet. Something relating to the scripts keeps restoring the skynet config file, and then syslog-ng fails because there are two config files with the same definition. I don't want to allow duplicates.

 

[skeal]

Give me a few and I can do Cake...I'll see if I can get it over before the next publish...

Or share what you built...might just need a tweak or two. One key thing with the config files for both logrotate and syslog-ng is perms....0600 - else if never reads the config.

My hostapd scripts are below: I get this error when running logrotate.

==> /opt/tmp/logrotate.daily <==
error: hostpad:1 lines must begin with a keyword or a filename (possibly in double quotes)
error: hostpad:6, unexpected text after }

# put hostapd messages into /opt/var/log/hostapd.log

destination d_hostapd {
    file("/opt/var/log/hostapd.log");
};

filter f_hostapd {
    ( program("hostapd") or
    program("hostapd") ) and
    ( message("eth7") or
    message("eth7") ) or
    ( program("syslog") and
    message("hostapd") );
};

log {
    source(src);
    filter(f_hostapd);
    destination(d_hostapd);
    flags(final);
};

#eof

/opt/var/log/hostapd.log {
    rotate 4
    postrotate
        /usr/bin/killall -HUP syslog-ng
    endscript
}

Thanks for the help. I don't know why the error shows it as hostpad. The above does filter the hostapd content, but throws the error from the scribe menu when selecting "lr" logrotate.

 

[elorimer]

My hostapd scripts are below: I get this error when running logrotate.

==> /opt/tmp/logrotate.daily <==
error: hostpad:1 lines must begin with a keyword or a filename (possibly in double quotes)
error: hostpad:6, unexpected text after }

I created "hostapd" files in both syslog-ng.d and logrotate.d directories cutting and pasting your text, then I did from the scribe menu rl, rs, lr without issue (but I didn't have any messages in that log).

Perhaps the file is not in Unix format? That is a logrotate error, and I'm guessing there is a typo in the name of the config file?

 

[skeal]

I created "hostapd" files in both syslog-ng.d and logrotate.d directories cutting and pasting your text, then I did from the scribe menu rl, rs, lr without issue (but I didn't have any messages in that log).

Perhaps the file is not in Unix format? That is a logrotate error, and I'm guessing there is a typo in the name of the config file?

I created both files with notepad++ on my Ubuntu desktop. I don't think that's the problem but a simple dos2unix command would fix that.
You mention a typo in the config file. What config file are you referring too?

 

[skeal]

I'm guessing there is a typo in the name of the config file?

Yup typo was it, now it runs without error. Anyone got clues on how to do Cake yet? Everything I try fails.

 

[skeal]

I'm trying to filter these:

Jun 24 17:30:00 RT-AX88U-9D28 CakeQOS-Merlin: Checking if running...
Jun 24 17:30:00 RT-AX88U-9D28 CakeQOS-Merlin: Running successfully

 

[elorimer]

Using "CakeQOS-Merlin" with a program filter doesn't work?

As an aside, I'm not really a fan of filtering everything out of messages, just the somewhat noisy loggers, like pixelserv, skynet and spdMerlin that obscure other useful messages.

 

[skeal]

Using "CakeQOS-Merlin" with a program filter doesn't work?

Not in the efforts I tried...lol

 

[skeal]

As an aside, I'm not really a fan of filtering everything out of messages, just the somewhat noisy loggers, like pixelserv, skynet and spdMerlin that obscure other useful messages.

It looks like withe release of 1.0.0 the log spam has stopped or at least reduced.

 

[dsancho]

Hi,

Sorry for my stupid question, but what does "run logrotate now" menu option ?
And how can I clean all the logs on demand ?

Thanks

 

[elorimer]

It's not stupid because there is a hidden surprise. lr runs logrotate as currently configured with the files in /opt/etc/logrotate.d and /opt/etc/logrotate.conf. Because those are usually configured for daily rotation, that means you can rotate the logs in /opt/var/logs once. If you run it again, it will see that they have been rotate already and not rotate them again.

Logrotate leaves you with a current log, a plain text rotated log, and several compressed earlier rotated logs. You can just delete the rotated log and the compressed logs if you want.

If you want to "clean", that is edit, the current log, it is best to stop syslog-ng first. You might do that, for example, if you've gotten some runaway logging to pixelserv.log. Otherwise your editor and syslog-ng will both be operating on the same file and you will get an inode mismatch.

If you want to force a rotation of a logs, you need to manually run it:

logrotate -f logrotate.conf

That will force (-f) a rotation of the logs without regard to age (the daily part) or size.

 

[dsancho]

It's not stupid because there is a hidden surprise. lr runs logrotate as currently configured with the files in /opt/etc/logrotate.d and /opt/etc/logrotate.conf. Because those are usually configured for daily rotation, that means you can rotate the logs in /opt/var/logs once. If you run it again, it will see that they have been rotate already and not rotate them again.

Logrotate leaves you with a current log, a plain text rotated log, and several compressed earlier rotated logs. You can just delete the rotated log and the compressed logs if you want.

If you want to "clean", that is edit, the current log, it is best to stop syslog-ng first. You might do that, for example, if you've gotten some runaway logging to pixelserv.log. Otherwise your editor and syslog-ng will both be operating on the same file and you will get an inode mismatch.

If you want to force a rotation of a logs, you need to manually run it:

logrotate -f logrotate.conf

That will force (-f) a rotation of the logs without regard to age (the daily part) or size.

Thanks !

 

[CaptnDanLKW]

I suspect this is outside the current capabilities and maybe even the roadmap of scribe, but I'd like to pull in the logs for my other Mesh nodes and have them presented in the main router's UI. Pre scribe and syslog-ng there was a open to forward log levels > 'x' to a syslog server. Can syslog on the mesh node be configured to forward to the main router, where syslog-ng is listening? I guess it would need to write it to a newfile, and have scribe configured to manage them?

Not a lot of interesting stuff happens on nodes buta few do. Things like roam assist, acsd, kernel bugchecks, etc would still be useful.

Just wondering...

 

[elorimer]

Can syslog on the mesh node be configured to forward to the main router, where syslog-ng is listening?

Should be, although I don't know anything about syslog on a mesh node. Presumably you don't need scribe and syslog-ng on the mesh node, so you can uninstall uiScribe on the mesh node, if it is installed and then you should get the fields to send log messages to a remote server. Remote logging on Asus-AC88U with Merlin 384.16
So send them to the main router at port 514.

Then, configure syslog-ng on the main router to receive those messages; checkout the A00Remote configuration file and the main syslog-ng.conf file for how to do that. Messages from the mesh node will be tagged with a different Host field and you can figure out how you want to distribute them.

I pull in messages from my FreeNAS servers and send them to their own file, which shows up in the GUI.

 

[skeal]

I removed Suricata and still have the suricata.log entry on the scribe syslog page. How do I get rid of it? It's the only lingering part left of manual uninstall.

 

[Butterfly Bones]

I removed Suricata and still have the suricata.log entry on the scribe syslog page. How do I get rid of it? It's the only lingering part left of manual uninstall.

Delete the suricata log from /opt/var/log/ and then open uiScribe and use menu #1 to hide it from the uiScribe group of logs. Next time you reboot, uiScribe will not have anything about it.

 

[elorimer]

I removed Suricata and still have the suricata.log entry on the scribe syslog page. How do I get rid of it? It's the only lingering part left of manual uninstall.

Or, I think better, delete the configuration file from /opt/etc/syslog-ng.d (which I'm guessing you did), and then open uiScribe and use rf to rescan the /opt/etc/syslog-ng.d directory.

 

[Kingp1n]

I removed Suricata and still have the suricata.log entry on the scribe syslog page. How do I get rid of it? It's the only lingering part left of manual uninstall.

@skeal any specific reason you removed suricata? I'm thinking of installing but I'm just gathering info on pros/cons! Thanks!

 

[skeal]

Or, I think better, delete the configuration file from /opt/etc/syslog-ng.d (which I'm guessing you did), and then open uiScribe and use rf to rescan the /opt/etc/syslog-ng.d directory.

That worked. Thanks.

 

[skeal]

@skeal any specific reason you removed suricata? I'm thinking of installing but I'm just gathering info on pros/cons! Thanks!

I was only trying Suricata out because I was trying out Cake QOS. Turns out doesn't matter to me whether Trend Micro is running or Suricata. So I went with what I know.