Scribe Logging stops after changing Router System settings

[CaptnDanLKW]

I've noticed that logging is stopping after I sometimes change/save a setting in the System section. For example, I turned off DNS monitoring in the System. I know this bounces a few services, etc. However I know I have an issue when I see this as the last item logged.

Mar 23 11:46:40 RT-AC86U rc_service: httpd 1167:notify_rc restart_time;restart_leds;restart_usb_idle;restart_firewall;
Mar 23 11:46:40 RT-AC86U custom_script: Running /jffs/scripts/service-event (args: restart time)
Mar 23 11:46:40 RT-AC86U ntpd: Stopped ntpd
Mar 23 11:46:40 RT-AC86U dropbear[1120]: Early exit: Terminated by signal
Mar 23 11:46:40 RT-AC86U ntpd: Started ntpd
Mar 23 11:46:40 syslogd started: BusyBox v1.25.1
Mar 23 11:46:40 kernel: klogd started: BusyBox v1.25.1 (2021-02-12 17:48:18 EST)
Mar 23 11:46:40 dropbear[25587]: Running in background
Mar 23 11:46:40 custom_script: Running /jffs/scripts/service-event (args: restart leds)
Mar 23 11:46:40 custom_script: Running /jffs/scripts/service-event (args: restart usb_idle)
Mar 23 11:46:40 custom_script: Running /jffs/scripts/service-event (args: restart firewall)
Mar 23 11:46:40 sd-idle-2.6[25676]: initialized
Mar 23 11:46:41 kernel: klogd: exiting
Mar 23 11:46:41 syslogd exiting

At thing point nothing else is logged though syslog-ng is still running

n@RT-AC86U:/tmp/home/root# ps ax |grep -i syslog
2905 ? S 0:00 supervising syslog-ng
2906 ? Ss 0:11 syslog-ng

checking scribe, it says everything looks good. Restarting scribe from the menu doesnt fix. the only thing that works is a reboot.

Looking for some help... Thanks.

 

[elorimer]

The last lines are the kill_logger function called when S01syslog-ng is started. That's good. It also sounds like syslog-ng is running but not logging, perhaps trying to log to a file location that no longer exists, as if along the way there should be a hangup signal.

You might:

1. start syslog-ng with the -Fevd switch. That more detailed logging might tell you something.
2. See in the syslog-ng.log if there is a hint.
3. Wait until skynet does its hourly rollup and issues the sighup message to cause syslog-ng to restart.
4. See if you stop syslog-ng with scribe, and then restart it.

 

[skeal]

I have the same issue. Not sure what setting or change I made in the settings that stopped scribe from logging. Weird thing is you restart syslog-ng and all the missed logs are dumped without filtering into the syslog portion of the log page. Skynet stuff and all of it in a dump. Then it stops working again. I had to restart syslog-ng and then reboot to get logging working again.

 

[skeal]

For me on my AX88U on this beta 2, all I need to do to recreate this, is run the scribe check button to see if there is an update. If the update logging shows your fine if not you are not logging anything to the syslog.

 

[skeal]

I think I know what happens here. Something that uses entware isn't working right since I updated all the entware packages in AMTM. Since then I've had this log stopping issue.

 

[bluzfanmr1]

I think I know what happens here. Something that uses entware isn't working right since I updated all the entware packages in AMTM. Since then I've had this log stopping issue.

Ditto here. I had to restart scribe, reinstall unbound and then reinstall ubound adblocking.

 

[elorimer]

Weird thing is you restart syslog-ng and all the missed logs are dumped without filtering into the syslog portion of the log page. Skynet stuff and all of it in a dump.

This means nothing in opt/etc/syslog.d is being reloaded on restart.

I see that among the packages being updated with amtm in the last bit is syslog-ng itself, which goes from 3.29 to 3.30. But I don't see anything in the changelog that should affect scribe.

Also: see the unbound thread for the entware update problem.

 

[CaptnDanLKW]

So, two days ago I started all my addons over again. Not that I had very many. scMerlin, conmon and scribe (and uiScribe).

checked and manually cleaned all references in /jffs/scripts and /jffs/addons, etc
checked to make sure files /opt/etc/syslog-ng.conf and /opt/etc/logrotate.conf were gone
checked to make sure directories /opt/etc/syslog-ng.d and /opt/etc/logrotate.d were gone

I reinstalled only conmon and scribe (and uiScribe).

Today I wanted to make a change, and enable HTTPS & generate a new UI cert using a Lets Encrypt service.

Same thing happened as I highlighted below. Noteworthy is that the though it says kernel logging daemon exited, (along with syslogd - which I assume is really syslog-ng's daemon) kernel logging still happened. And only kernel logging.

As @skeal mentioned above, when we try to restart syslog-ng (I do though the scribe menu), it drops the logs without processing or even retaining them and it still refuses to log anything new. A reboot is the only solution. What's interesting is that if you DON'T force restart syslog-ng and go directly to a reboot, the logs are NOT dropped.

@elorimer I noticed the version bump too, but IIRC I've had this issue since 3.27 and for at least a year. For the most part once I'm done on the Advanced --> Administration --> System tab, I rarely have to change anything so this issue, while annoying, doesn't bother me much.

My problem lately, is that after I started a v386.1_2 fresh full reconfig - starting with M&M - I've had frequent random WAN disconnect events. I've troubleshot down to the Networking Monitor 'DNS' test failing (dns requests to router's resolver sometimes times out - something else outside the scope of this issue). So I've had to enable/disable that test from time to time, which I notice broke syslog-ng, hence my initial post about issue.


Mar 25 11:17:32 RT-AC86U inadyn[14220]: In-a-dyn version 2.8.1 -- Dynamic DNS update client.
Mar 25 11:17:32 RT-AC86U inadyn[14220]: Failed resolving hostname ***************: Name or service not known
Mar 25 11:17:32 RT-AC86U inadyn[14220]: Update forced for alias ***************, new IP# xxx.yyy.zzz.aaaa
Mar 25 11:17:45 RT-AC86U inadyn[14220]: alias address=<xxx.yyy.zzz.aaaa>
Mar 25 11:17:45 RT-AC86U inadyn[14220]: Remove old cache file /var/cache/inadyn for ***************
Mar 25 11:17:45 RT-AC86U inadyn[14220]: Updating cache for **************
Mar 25 11:18:51 RT-AC86U rc_service: httpd 1257:notify_rc restart_time;restart_leds;restart_usb_idle;restart_httpd;restart_firewall;
Mar 25 11:18:51 RT-AC86U custom_script: Running /jffs/scripts/service-event (args: restart time)
Mar 25 11:18:51 RT-AC86U ntpd: Stopped ntpd
Mar 25 11:18:51 RT-AC86U dropbear[23039]: Exit (****) from <192.168.1.2:56324>: Terminated by signal
Mar 25 11:18:51 RT-AC86U dropbear[23602]: Early exit: Terminated by signal
Mar 25 11:18:51 RT-AC86U ntpd: Started ntpd
Mar 25 11:18:51 syslogd started: BusyBox v1.25.1
Mar 25 11:18:51 kernel: klogd started: BusyBox v1.25.1 (2021-03-20 01:29:20 EDT)
Mar 25 11:18:51 dropbear[15297]: Running in background
Mar 25 11:18:51 custom_script: Running /jffs/scripts/service-event (args: restart leds)
Mar 25 11:18:51 custom_script: Running /jffs/scripts/service-event (args: restart usb_idle)
Mar 25 11:18:51 custom_script: Running /jffs/scripts/service-event (args: restart httpd)
Mar 25 11:18:51 sd-idle-2.6[15361]: initialized
Mar 25 11:18:51 RT-AC86U: start https:8443
Mar 25 11:18:51 RT-AC86U: start httpd:80
Mar 25 11:18:51 custom_script: Running /jffs/scripts/service-event (args: restart firewall)
Mar 25 11:18:51 httpd: Save SSL certificate...80
Mar 25 11:18:52 httpd: Generating SSL certificate...80
Mar 25 11:18:52 kernel: klogd: exiting
Mar 25 11:18:52 syslogd exiting
Mar 25 11:21:10 RT-AC86U kernel[Thu]: Mar 25 11:21:10 DST 2021]
Mar 25 11:21:10 RT-AC86U kernel: account key ok.
Mar 25 11:21:10 RT-AC86U kernel[Thu]: Mar 25 11:21:10 DST 2021]
Mar 25 11:21:10 RT-AC86U kernel: account
Mar 25 11:21:11 RT-AC86U kernel[Thu]: Mar 25 11:21:11 DST 2021]
Mar 25 11:21:12 RT-AC86U kernel[Thu]: Mar 25 11:21:12 DST 2021]
Mar 25 11:21:12 RT-AC86U kernel[Thu]: Mar 25 11:21:12 DST 2021]
Mar 25 11:21:12 RT-AC86U kernel: domain key
Mar 25 11:21:12 RT-AC86U kernel[Thu]: Mar 25 11:21:12 DST 2021]
Mar 25 11:21:12 RT-AC86U kernel: domain key is here: /jffs/.le/************/************.key
Mar 25 11:21:12 RT-AC86U kernel[Thu]: Mar 25 11:21:12 DST 2021]

 

[elorimer]

syslogd is the firmware's internal logging daemon, nothing to do with syslog-ng.

Immediately after syslogd exits (because the startup script for syslog-ng kills it), syslog-ng is starting and taking over the log function. You can tell because the logs up to the syslogd/klogd startup and from the exit message onward have the host added. (the two with the host in the middle I think are taking their own sweet time from the before-part. But I guess the specific config files aren't being loaded.

I can't explore this myself because I'm a few thousand klics away from my router.

If you start syslog-ng in the foreground with -Fevd you should see it load the files from syslog-ng.d.