Scribe scribe - syslog-ng and logrotate installer

[Martineau]

Heh, don't do that on my account. I know I'm a bit over-zealous at times ... just be aware that if you update syslog-ng in the future and stuff starts breaking, then the `match` function is a likely suspect.

I don't remember the () usually being there around the program name, but I haven't used this script for a while, and I'm not at home to look at the log. I only use that script periodically for debugging since it makes large logs. Are they there for other entries? If not, `program` might have to match on `(VPN_Failover.sh)`.

The parentheses not only acted as an eye-catcher in Syslog, but back when I used syslog_move.sh at the end of init-start, this used a simple grep to extract my 'signature' logger tags for boot period script execution analysis.

It appears the following works...without explicit parentheses

# Put VPN_Failover.sh script messages into /opt/var/log/VPNFailover.log

destination d_vpnfailover {
 file("/opt/var/log/VPNFailover.log");
};

filter f_vpnfailover {
 program("VPN_Failover.sh");
};

log {
 source(src);
 source(kernel);
 filter(f_vpnfailover);
 destination(d_vpnfailover);
 flags(final);
};
#eof

 

[cmkelley]

The parentheses not only acted as an eye-catcher in Syslog, but back when I used syslog_move.sh at the end of init-start, this used a simple grep to extract my 'signature' logger tags for boot period script execution analysis.

It appears the following works...without explicit parentheses

# Put VPN_Failover.sh script messages into /opt/var/log/VPNFailover.log

destination d_vpnfailover {
 file("/opt/var/log/VPNFailover.log");
};

filter f_vpnfailover {
 program("VPN_Failover.sh");
};

log {
 source(src);
 source(kernel);
 filter(f_vpnfailover);
 destination(d_vpnfailover);
 flags(final);
};
#eof

Good to know, thanks.

 

[Butterfly Bones]

You should still be getting the dropbear stuff in the GUI syslog. It sounds like /opt/etc/init.d/S01syslog-ng isn't calling the rc.func.syslog-ng script for some reason, so the links and such aren't being made correctly. Is /tmp is syslog.log symlinked to /opt/var/log/messages? That symlink allows the GUI syslog to see the messages file.

No, no entries in GUI syslog of anything after install. And looking at the /opt/var/log/messages file, not there either. Very few entries in messages, only the output of ChkWAN and VPN_Failover scripts run by cron. Other cron jobs like auto leds on and off do not show either. Just FYI here are the last few lines of that gui syslog I see all the events (too many) as I usually see until syslog-ng starts, then all displayed entries stop.

Apr  2 21:06:54 syslogd exiting
Apr  2 21:06:54 RT-AC86U-4608 syslog-ng[9995]: syslog-ng starting up; version='3.19.1'

I forget to check the symlink last night before I uninstalled, not wanting to let it run overnight without confidence all was functioning properly. I will reinstall this afternoon and check again.

Hrmmm. Skynet debug watch seems to be working for me. If you go into your usb device, in the skynet directory lives the skynet.cfg fiile. If you cat that, there should be 2 lines in there that say:

syslogloc="/opt/var/log/skynet-0.log"
syslog1loc="/tmp/syslog.log-1"

if syslogloc is still pointing at /tmp/syslog.log, then the script failed to correctly call skynet to fix the log file location.

Yes both "syloglogloc" & "syloglog1loc" lines are there exactly like that.

If the symlink isn't there and sysloglog is pointing to the wrong thing, then your installation is broken, you should try removing and re-installing, but back up anything in /opt/etc/syslog-ng.d and /opt/share/syslog-ng/examples that you've modified beforehand because those will be wiped when removing syslog-ng.

Well I have already done that 5-6 times yesterday, but I am a glutton for punishment so my friends say. I backed up the entire drive to my Linux box so I can look at it anytime for reference and to save all working (or non-working) configs.

Thank you for the ongoing troubleshooting on this. I have been trying to get syslog-ng working off and on for over a year. So close at this point!

 

[Butterfly Bones]

This really looks like it's coming along niceley. I'm a log watcher of sorts and would really like a solution to the Skynet logging to syslog. This appears to be it. I'll wait to jump in though, as I'm not well health wise right now. I've wanted this solution since Skynet came out...LOL Thank you for this outstanding effort.

First, get your health in order. Best wishes to you, I've been through hell the last five years and now all those things seem to be behind me, so I torture myself here. Get well so you can jump in the fray, misery loves company.

 

[elorimer]

check the symlink

Not sure how it works in your terminal, but doesn't ls give you symlinks in a different color?

In a terminal, do it manually:

rm /tmp/syslog.log
ln -s /opt/var/log/messages /tmp/syslog.log

That will create the symlink and should show your messages file in the WebGUI.

If that doesn't work, it may be syslogd restarted in between those two instructions, so run them again with the up arrow autoloading them for you. If that still doesn't work, something else is at fault and the scribe script won't fix it.

 

[Butterfly Bones]

In a terminal, do it manually:

rm /tmp/syslog.log
ln -s /opt/var/log/messages /tmp/syslog.log

That will create the symlink and should show your messages file in the WebGUI.

If that doesn't work, it may be syslogd restarted in between those two instructions, so run them again with the up arrow autoloading them for you. If that still doesn't work, something else is at fault and the scribe script won't fix it.

Thank you for the suggestion, I actually did that last night referencing a script I saved a year ago working in the original kvic thread. I was considering adding a cron job to run that and *maybe* solve it. What I found was the "messages" has almost all of the original syslog lines with nothing that i configured to be removed to separate logs in syslog-ng. Some things were not in "messages" that were in the gui syslog before syslog-ng start, but not all.

Obviously something is not right in my router. It was reset and reconfigured manually one merlin release back, but the USB drive has been running all the scripts for over a year, I might need to think about reformat it and reinstall everything fresh, then restore only Diversion and Skynet data. At that point I was tired and needed a break, so I removed splice, rebooted, and went to bed.

 

[elorimer]

Perhaps delete everything in the messages file, delete all the rotated messages logs and reboot. I sometimes forgot that the messages file persists from reboot to reboot, and where I was looking often was several reboots earlier.

Then you should only have one fresh messages file, displayed in the webGUI. It should have everything from the initial start, up to the syslogd exit message and the start syslog-ng, with all the logging and no messages sifted into the other buckets, and from that point on all the logging that isn't sifted into the other buckets.

It looks like from the start of syslog-ng your messages file is right, because it has nothing in it that you had configured to be removed. I'm puzzled by the messages file not having everything that had been in the webGUI syslogd, because messages starts with a cat from the syslog.log before it gets blown away.

The only thing I've found that affects the symlink is a sed done on it (skynet if it isn't set to the custom log); otherwise mine lasts weeks.

I know I am repeating what you already know; I'm just not figuring out why you've had so much trouble. Mine is an 87U with amtm/diversion/pixelserv/skynet/stubby and entware, with Openvpn servers and some openvpn-event scripts. So I'm not far off what you have.

 

[cmkelley]

No, no entries in GUI syslog of anything after install. And looking at the /opt/var/log/messages file, not there either. Very few entries in messages, only the output of ChkWAN and VPN_Failover scripts run by cron. Other cron jobs like auto leds on and off do not show either. Just FYI here are the last few lines of that gui syslog I see all the events (too many) as I usually see until syslog-ng starts, then all displayed entries stop.

Apr  2 21:06:54 syslogd exiting
Apr  2 21:06:54 RT-AC86U-4608 syslog-ng[9995]: syslog-ng starting up; version='3.19.1'

I forget to check the symlink last night before I uninstalled, not wanting to let it run overnight without confidence all was functioning properly. I will reinstall this afternoon and check again.


Yes both "syloglogloc" & "syloglog1loc" lines are there exactly like that.



Well I have already done that 5-6 times yesterday, but I am a glutton for punishment so my friends say. I backed up the entire drive to my Linux box so I can look at it anytime for reference and to save all working (or non-working) configs.

Thank you for the ongoing troubleshooting on this. I have been trying to get syslog-ng working off and on for over a year. So close at this point!

Are you customizing either syslog-ng.conf or any of the files in syslog-ng.d? If so, can you run `scribe config` and PM me the file /tmp/syslog-ng-complete.conf? I'll see if I can spot anything amiss.

 

[Butterfly Bones]

Perhaps delete everything in the messages file, delete all the rotated messages logs and reboot. I sometimes forgot that the messages file persists from reboot to reboot, and where I was looking often was several reboots earlier.

Then you should only have one fresh messages file, displayed in the webGUI. It should have everything from the initial start, up to the syslogd exit message and the start syslog-ng, with all the logging and no messages sifted into the other buckets, and from that point on all the logging that isn't sifted into the other buckets.

It looks like from the start of syslog-ng your messages file is right, because it has nothing in it that you had configured to be removed. I'm puzzled by the messages file not having everything that had been in the webGUI syslogd, because messages starts with a cat from the syslog.log before it gets blown away.

The only thing I've found that affects the symlink is a sed done on it (skynet if it isn't set to the custom log); otherwise mine lasts weeks.

I know I am repeating what you already know; I'm just not figuring out why you've had so much trouble. Mine is an 87U with amtm/diversion/pixelserv/skynet/stubby and entware, with Openvpn servers and some openvpn-event scripts. So I'm not far off what you have.

Thank you, repeating is good as I learn this and blunder my way along. I have removed all the log files from syslog-ng in my /opt/var/log/ directory, good advice. Here is a link to the last messages file that does not contain all regular webGUI syslog lines as I see with syslog-ng. https://pastebin.com/HR9BqJqV

Here is a paste from the regular webGUI syslog of a selection of typical lines. from today. https://pastebin.com/1Cygz9Fu

This has after I copied config files into the /opt/etc/syslog-ng.d directory and restated using the scribe menu. That is why there is a stop and then start at the beginning.

One specific question that you and @cmkelley might need to answer that might be the source of my issues with Skynet, is the use of a custom log for Skynet. Do I need to change Skynet log location or leave it alone as is (/tmp/syslog.log and default)?

Are you customizing either syslog-ng.conf or any of the files in syslog-ng.d? If so, can you run `scribe config` and PM me the file /tmp/syslog-ng-complete.conf? I'll see if I can spot anything amiss.

No change to the syslog-ng.conf. I am using the skynet, pixelserv, syslogng. wlceventd, crashes as you have them in GitHub, just copied from the share/examples to /opt/etc/syslog-ng.d where needed. The chkwan and vpnfailover I have created with elorimer help earlier in this thread. Those do show lines in their respective log files in /opt/var/log/ and they still show in messages as well, yet Skynet is logged in /opt/var/log/ but not in messages.

I may need to have you look at those to make sure the format is good. I used the wlceventd as a template, but because the scripting symbols are hard for me to grasp, I may have something wrong.

Here is what is / was in the directory. I copied all of my USB drive to my computer. I do not have syslog-ng installed right now, so I will get the "scribe config" to you tomorrow when I reinstall. I had a couple unexpected items that needed attention today.

drwxrwxr-x  2 user root 4096 Apr  3 05:54 .
drwxrwxr-x 11 user root 4096 Apr  3 05:54 ..
-rw-rw-r--  1 user root  283 Apr  2 15:51 chkwan
-rw-rw-r--  1 user root  574 Apr  2 21:06 crashes
-rw-r--r--  1 user root    0 Mar 20 00:45 .keep
-rw-rw-r--  1 user root  351 Apr  2 15:59 pixelserv
-rw-rw-r--  1 user root  539 Apr  2 20:59 skynet
-rw-rw-r--  1 user root  319 Apr  2 21:06 syslogng
-rw-rw-r--  1 user root  324 Apr  2 15:54 vpnfailover
-rw-rw-r--  1 user root  328 Apr  2 21:06 wlceventd

Also, I stole your quote from the ntpMerlin thread for my signature below.

 

[cmkelley]

One specific question that you and @cmkelley might need to answer that might be the source of my issues with Skynet, is the use of a custom log for Skynet. Do I need to change Skynet log location or leave it alone as is (/tmp/syslog.log and default)?

Also, I stole your quote from the ntpMerlin thread for my signature below.

The scribe installer, if it detects Skynet, will automatically change the skynet log location (syslogloc) to "/opt/var/log/skynet-0.log". This needs to be done to keep skynet from breaking the symlink from /tmp/syslog.log to /opt/var/log/messages. syslog1loc doesn't need to be changed, but I think I should a line in the installer to delete "/tmp/syslog.log-1". That might help un-confuse the GUI.

 

[Butterfly Bones]

The scribe installer, if it detects Skynet, will automatically change the skynet log location (syslogloc) to "/opt/var/log/skynet-0.log". This needs to be done to keep skynet from breaking the symlink from /tmp/syslog.log to /opt/var/log/messages. syslog1loc doesn't need to be changed, but I think I should a line in the installer to delete "/tmp/syslog.log-1". That might help un-confuse the GUI.

My reply above in post #43 is NOT correct. I was tired and not thinking clearly. This is what is in the skynet.cfg file. (saved entire USB stick to computer before syslog-ng uninstall)

syslogloc="/tmp/syslog.log"
syslog1loc="/tmp/syslog.log-1"

and not this as it should be

syslogloc="/opt/var/log/skynet-0.log"
syslog1loc="/tmp/syslog.log-1"

Does this mean the install is not changing that line? I see in the skynet.cfg states to not manually edit. If I have trouble tomorrow, I will make the change above using the Skynet menu.

When I install tomorrow, I will only use the config files in your command line install and not add my extra configs.

 

[cmkelley]

My reply above in post #43 is NOT correct. I was tired and not thinking clearly. This is what is in the skynet.cfg file. (saved entire USB stick to computer before syslog-ng uninstall)

syslogloc="/tmp/syslog.log"
syslog1loc="/tmp/syslog.log-1"

and not this as it should be

syslogloc="/opt/var/log/skynet-0.log"
syslog1loc="/tmp/syslog.log-1"

Does this mean the install is not changing that line? I see in the skynet.cfg states to not manually edit. If I have trouble tomorrow, I will make the change above using the Skynet menu.

When I install tomorrow, I will only use the config files in your command line install and not add my extra configs.

Yeah, that tells me the rc.func.syslog-ng file isn't being run when syslog-ng is started. I removed and reinstalled a couple times on my test router and it always created the symlink and pointed Skynet at the correct file. And the rc.func.syslog-ng file already deletes both /tmp/syslog.log and /tmp/syslog.log-1 when it runs.

You should be able to verify a successful install by running `scribe status` and/or `ps | grep syslog`, the latter should output something like:

 1683 hostname  9500 S    {syslog-ng} supervising syslog-ng
 1686 hostname  154m S    syslog-ng
17273 hostname  5320 S    grep syslog

If it shows syslogd still running, the rc.func.syslog-ng file isn't being run at all.

 

[cmkelley]

V 0.5_5 pushed. Cosmetic changes, no functionality change.

 

[elorimer]

Here is a paste from the regular webGUI syslog of a selection of typical lines. from today. https://pastebin.com/1Cygz9Fu

The logs tell me two things. First, skynet messages in the webGUI in the last hour, and not before, are a tell-tale sign that skynet is running its cleaning against the same file showing in the webGUI. That isn't a bad thing, if that file is the messages file, but that isn't the way scribe works. Scribe sorts the skynet messages into their own log, points skynet at it, and lets it do its cleaning there. So something about forming the symlink and redirecting skynet is not working, as @cmkelley is pointing out above.

The bigger problem is it looks like syslog-ng isn't doing anything. Your messages file has in it Chkwan and VPN_failover stuff and not much else. I thought those were being sifted out into their own records, but the fact that they are in messages means they aren't being processed at all. The fact that messages doesn't have much else (all the kernel messages, for example) that seem to be in /tmp/syslog.log suggests to me that both syslogd and syslog-ng are running and syslog-ng is getting only some things. That must mean your sources are messed up.

@cmkelley, might it be that rc.func.syslog-ng isn't being marked as executable? That would mean syslogd isn't being killed but syslog-ng is started; also explaining the symlink stuff?

EDIT: I see in scribe a chmod for scribe itself but not for rc.func.syslog-ng. (0.5.3)

 

[cmkelley]

The logs tell me two things. First, skynet messages in the webGUI in the last hour, and not before, are a tell-tale sign that skynet is running its cleaning against the same file showing in the webGUI. That isn't a bad thing, if that file is the messages file, but that isn't the way scribe works. Scribe sorts the skynet messages into their own log, points skynet at it, and lets it do its cleaning there. So something about forming the symlink and redirecting skynet is not working, as @cmkelley is pointing out above.

The bigger problem is it looks like syslog-ng isn't doing anything. Your messages file has in it Chkwan and VPN_failover stuff and not much else. I thought those were being sifted out into their own records, but the fact that they are in messages means they aren't being processed at all. The fact that messages doesn't have much else (all the kernel messages, for example) that seem to be in /tmp/syslog.log suggests to me that both syslogd and syslog-ng are running and syslog-ng is getting only some things. That must mean your sources are messed up.

@cmkelley, might it be that rc.func.syslog-ng isn't being marked as executable? That would mean syslogd isn't being killed but syslog-ng is started; also explaining the symlink stuff?

EDIT: I see in scribe a chmod for scribe itself but not for rc.func.syslog-ng. (0.5.3)

rc.func.syslog-ng doesn't need to be executable. Note that rc.func (and rc.func.div if you have Diversion) also isn't executable. Sourcing the script using ". /opt/etc/init.d/rc.func.syslog-ng" doesn't require it to be executable, you can think of it as inserting the script at that point in the running script (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#dot). I'm not even certain it needs the she-bang, but it seems to be the norm to include it.

 

[elorimer]

Well, I'm confused.

I rebooted (still 0.5.3) and sort of reproduced @Butterfly Bones situation. The symlink formed, syslogd exited and syslog-ng started, but the webGUI was blank. I can see the symlink points to messages, and I can see messages populating, but no display.

I marked rc.func.syslog-ng as executable, rebooted, and everything came up okay.

So then I went back and unmarked rc.func.syslog-ng, then edited S01syslog-ng to add a ";" after the call to rc.func.syslog-ng, rebooted and everything came up okay.

Should the call to rc.func.syslog-ng be

. ./rc.func.syslog-ng

 

[Butterfly Bones]

rc.func.syslog-ng doesn't need to be executable. Note that rc.func (and rc.func.div if you have Diversion) also isn't executable. Sourcing the script using ". /opt/etc/init.d/rc.func.syslog-ng" doesn't require it to be executable, you can think of it as inserting the script at that point in the running script (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#dot). I'm not even certain it needs the she-bang, but it seems to be the norm to include it.

Well, I'm confused.

I rebooted (still 0.5.3) and sort of reproduced @Butterfly Bones situation. The symlink formed, syslogd exited and syslog-ng started, but the webGUI was blank. I can see the symlink points to messages, and I can see messages populating, but no display.

I marked rc.func.syslog-ng as executable, rebooted, and everything came up okay.

So then I went back and unmarked rc.func.syslog-ng, then edited S01syslog-ng to add a ";" after the call to rc.func.syslog-ng, rebooted and everything came up okay.

Should the call to rc.func.syslog-ng be

. ./rc.func.syslog-ng

Ok, I'm going to hold off install until this gets resolved. Thanks both of you for the comments. I am learning, and could make these changes, but I'd think it best to make sure the script works for others.

 

[cmkelley]

Ok, I'm going to hold off install until this gets resolved. Thanks both of you for the comments. I am learning, and could make these changes, but I'd think it best to make sure the script works for others.

Now we're getting somewhere ... the system log in webgui works fine on my 86U after a reboot, but not my 3200. So, I missed something somewhere in my setup for scribe that I had done along the way. That's not going to be easy to track down.

On a "maybe easier to solve note", if logs are appear in both a dedicated log and messages, first thing to look for is a missing `flags(final);` in the log section of the misbehaving filter.

 

[Goobi]

Don’t think this is related to scribe since I noticed this before installing. I noted that when viewing the syslog via the web GUI, it takes forever to load. Anyone else notice this?

 

[cmkelley]

Now we're getting somewhere ... the system log in webgui works fine on my 86U after a reboot, but not my 3200. So, I missed something somewhere in my setup for scribe that I had done along the way. That's not going to be easy to track down.

On a "maybe easier to solve note", if logs are appear in both a dedicated log and messages, first thing to look for is a missing `flags(final);` in the log section of the misbehaving filter.

Oh, goody. Went away for a couple hours, came back, and now magically system log in webgui is working on my 3200. I completely wiped the 3200 before I started using it as a test router, all I did was install amtm, Entware, a swap file, and scribe, so it seems unlikely to be some weird thing that could be cleared by wiping everything and starting over again.

And now after another reboot of the 3200, the system log is being updated in the webgui immediately.