Scribe scribe - syslog-ng and logrotate installer

[elorimer]

It doesn't look like anything in 3.24 or 3.25 will break the current configuration, but with these guys you never know.

Yes, boyo, we've seen that movie.

 

[joe68000]

My default log is "messages" and it is flooded with these after my update to 384.14_2:

Jan  4 10:30:14 xxx dnsmasq-script[5362]: connect error: No such file or directory
Jan  4 10:30:14 xxx dnsmasq-script[5362]: [SEND_AMAS_NODE_EVENT:(4684)] ERROR connecting:No such file or directory.

What is the best way to filter these out?

Thanks!

 

[elorimer]

You could create a configuration file to place in /opt/etc/syslog-ng.d that would have

destination d_dnsscripterror{ 
    file("/opt/var/log/dnsscripterror.log");
};
filter f_dnsscripterror{
    program("dnsmasq-script");
};
log {
    source(src);
    filter(f_dnsscripterror);
    destination(d_dnsscripterror);
    flags(final);
};
#eof

Or you could trash them after you have figured out what they mean:

filter f_dnsscripterror{
    program("dnsmasq-script");
};
log {
    source(src);
    filter(f_dnsscripterror);
    flags(final);
};
#eof

 

[joe68000]

You could create a configuration file to place in /opt/etc/syslog-ng.d that would have

destination d_dnsscripterror{
    file("/opt/var/log/dnsscripterror.log");
};
filter f_dnsscripterror{
    program("dnsmasq-script");
};
log {
    source(src);
    filter(f_dnsscripterror);
    destination(d_dnsscripterror);
    flags(final);
};
#eof

Or you could trash them after you have figured out what they mean:

filter f_dnsscripterror{
    program("dnsmasq-script");
};
log {
    source(src);
    filter(f_dnsscripterror);
    flags(final);
};
#eof

Thanks very much! I saw in another thread that these messages are related to closed-source code for AiMesh and can be ignored. I'm curious if anyone knows more about the messages.

 

[truglodite]

Syslog-ng noob here...

I played with installing scribe and uiscribe using amtm, and I seem to be having problems with large log files and configuration (system log webui crashes, and wlceventd.log is 150MB). I want to just delete wlcevend and reboot, lol, but I have a feeling that is not a good idea. So I figure I must configure properly, using the example files or whatever. Since I'm using lots of amtm scripts, do I just copy/paste those example files into their respective .d folders and reboot? Will this fix the 150MB file and keep things stable, or am I missing something here?

TIA,
Kevin

[edit: ... on an ac86u w/ latest merlin, amtm, diversion, skynet, freshjr qos, ntpmerlin, yazfi, uidivstats...

Nevermind, I fixed it by updating my wlceventd filter, and learned how to use the other filters. I got the ui working again by clearing out the big log files.]

 

[elorimer]

I want to just delete wlcevend and reboot,

You don't need to reboot after changing configurations, just use scribe to reload the configuration. Or skynet will do it on the hour for you.

Also, you can have syslog-ng just delete log messages by removing the destination from the log definition.

 

[cmkelley]

Syslog-ng noob here...

I played with installing scribe and uiscribe using amtm, and I seem to be having problems with large log files and configuration (system log webui crashes, and wlceventd.log is 150MB). I want to just delete wlcevend and reboot, lol, but I have a feeling that is not a good idea. So I figure I must configure properly, using the example files or whatever. Since I'm using lots of amtm scripts, do I just copy/paste those example files into their respective .d folders and reboot? Will this fix the 150MB file and keep things stable, or am I missing something here?

TIA,
Kevin

[edit: ... on an ac86u w/ latest merlin, amtm, diversion, skynet, freshjr qos, ntpmerlin, yazfi, uidivstats...

Nevermind, I fixed it by updating my wlceventd filter, and learned how to use the other filters. I got the ui working again by clearing out the big log files.]

Please do read the README.1st file in /opt/share/syslog-ng/examples. It should help you decide which files to copy.

 

[elorimer]

Thinking about this message over in the skynet thread: [Release] Skynet - Router Firewall & Security Enhancements
I have sometimes thought that it would be useful to clear all the logs and start over with a reboot. It is easy enough to do by stopping scribe and deleting the logs and rebooting. but I wonder if this might usefully be a scribe function, perhaps even a button on the web page. Of course, over time, one ends up with something close just as the logs roll off.

 

[Butterfly Bones]

The fun begins again, it appears. Entware has a list of updates, including this that seems to trip up some scribe users already.

 ep manage   Entware packages    -> upd avail
- syslog-ng              3.23.1-1  -> 3.25.1-1

Posting to warn others and alert @cmkelley and @elorimer because they (hopefully) can solve or at least workaround the twists and turns that the syslog-ng devs like to add.

 

[Butterfly Bones]

1. The tried and true reboot.
2. Stop and restart syslog-ng. Do not just restart syslog-ng, that doesn't seem to be enough. From the scribe menu, select st to stop scribe, then rs to start it. From the command line "scribe stop" and then "scribe start".

It took three tries with st and then rs (and patience) to get it logging again. I was ready to reboot too, but not until Clippers halftime.

 

[cmkelley]

It took three tries with st and then rs (and patience) to get it logging again. I was ready to reboot too, but not until Clippers halftime.

Hrm. I tried rs first, and that didn't work, so then I did st followed by rs and that worked for me first time. I'm increasingly of the mind that rebooting the router after upgrading Entware is probably a good idea, who knows what all else didn't restart properly?

 

[Butterfly Bones]

Hrm. I tried rs first, and that didn't work, so then I did st followed by rs and that worked for me first time. I'm increasingly of the mind that rebooting the router after upgrading Entware is probably a good idea, who knows what all else didn't restart properly?

I agree. Halftime.

 

[cmkelley]

I agree. Halftime.

I'll have wait longer, I'm WUW (Wife Using WiFi) - and I only single out my wife because it's just the two of us here.

 

[elorimer]

Posting to warn others and alert @cmkelley and @elorimer because they (hopefully) can solve or at least workaround the twists and turns that the syslog-ng devs like to add.

Neato! I don't see anything in the release notes for 3.24 or 3.25 that would break things. Put another way, I'm not smart enough to see something. But there is this interesting bit about visualizing the syslog-ng configuration, as well as the existing path to generating heatmaps. I'm thinking there might be a use here for one of the new webgui hooks.

I think if you update syslog-ng through amtm, you also have to rl scribe to reload the configuration. Interestingly, along the way, the version number of syslog-ng.conf was updated to 3.25. It is clearly the scribe version, but I swear I didn't do it. I issued a rs in scribe, it threw an old version error and went on its way, but with an updated .conf file.

 

[cmkelley]

Neato! I don't see anything in the release notes for 3.24 or 3.25 that would break things. Put another way, I'm not smart enough to see something. But there is this interesting bit about visualizing the syslog-ng configuration, as well as the existing path to generating heatmaps. I'm thinking there might be a use here for one of the new webgui hooks.

I think if you update syslog-ng through amtm, you also have to rl scribe to reload the configuration. Interestingly, along the way, the version number of syslog-ng.conf was updated to 3.25. It is clearly the scribe version, but I swear I didn't do it. I issued a rs in scribe, it threw an old version error and went on its way, but with an updated .conf file.

No, you didn't do it.

A simple rs didn't work for me, I had to st then rs to get it working, as you read it took Butterfly Bones a bit more to convince his system to work properly.

 

[Butterfly Bones]

No, you didn't do it.

A simple rs didn't work for me, I had to st then rs to get it working, as you read it took Butterfly Bones a bit more to convince his system to work properly.

It seemed to be working after the st, rs sequences. I did a router reboot to make sure everything was running as it should. That was a long list of Entware updates!

 

[elorimer]

But there is this interesting bit about visualizing the syslog-ng configuration,

Okay, I went down this rabbit hole and I don't recommend it. I used the syslog-ng-ctl function to export the configuration as a JSON file, then tracked down where the syslog-ng folks have a python script to convert that to a DOT file, and then opened that in graphviz. That gave me a not-very-helpful tree diagram of the configuration:

The part on the left (the "Y") is the logrotate configuration. Next to it (the "I") is the part of the configuration where I pour my starting log into syslog-ng so it comes out in the right place in messages. The thing on the right is the main scribe setup, with the flags(final) instruction giving it linear paths.

 

[cmkelley]

Okay, I went down this rabbit hole and I don't recommend it. I used the syslog-ng-ctl function to export the configuration as a JSON file, then tracked down where the syslog-ng folks have a python script to convert that to a DOT file, and then opened that in graphviz. That gave me a not-very-helpful tree diagram of the configuration:View attachment 21160

Somehow I knew you would.

The part on the left (the "Y") is the logrotate configuration. Next to it (the "I") is the part of the configuration where I pour my starting log into syslog-ng so it comes out in the right place in messages. The thing on the right is the main scribe setup, with the flags(final) instruction giving it linear paths.

I saw that a few weeks ago trying to get a heads up on what might break in the next release. I thought about it for about 30 seconds and went "Naaaahhhhhhhh". Anyone who would want that is welcome to either take over maintenance of scribe, or create their own script/fork, I ain't doing it.

 

[elorimer]

Yes, but now, let scribe take the skynet logs and make heat maps out of who is generating the denied traffic and add it to the GUI as a page! Boo, no elasticsearch in entware....

 

[markusd1984]

Can I change the logrotate for wlceventd without loosing the existing logs?

As you can imagine, my webui currenly won't load the log page anymore

Or am I better of deleting them and change the config to avoid these large sizes (don't know yet how) ?