Scribe Strange Issue with Scribe / Syslog-NG after Reboot

[ScottW]

About 3 months ago, I rebooted my AX86U and noticed afterwards that syslog-ng was not logging correctly. The files were simply not being updated. Scribe status showed everything as working. Rather than investigate, I just did a scribe restart and the logging began working fine again.

The next time I rebooted (about 2 months ago), I again noticed syslog-ng was not updating files after the reboot. As before, Scribe status showed everything was fine, and a scribe restart fixed the problem.

I noticed it again this week, so I started investigating. I *think* I found the problem -- which is that (2) instances of syslog-ng are running after boot (I assume they are fighting over the message sources). But I can't figure out how it is getting started twice.

I added some ps | grep syslog-ng >>/jffs/testlog.log statements to services-start and firewall-start. At services-start time, there was no instance of syslog-ng. At firewall-start, there was still no syslog-ng process running. About 25 seconds later, there was a second firewall-start event -- and at that point, there were TWO syslog-ng processes (plus the two service supervisor processes) running.

 3393 admin    13356 S    {syslog-ng} supervising syslog-ng
 3394 admin     374m S    syslog-ng
 3448 admin    13356 S    {syslog-ng} supervising syslog-ng
 3450 admin     344m S    syslog-ng

Executing a scribe restart command killed both of those and created a single new syslog-ng process (plus the supervising process). After that, everything worked correctly (until the next reboot).

I uninstalled scribe and syslog-ng, using the uninstall command on the scribe menu. Rebooted, and there was no syslog-ng process running. Used AMTM to reinstall Scribe, with the default configuration. Rebooted, and the duplicate syslog-ng is back! As before, scribe restart kills both of them, starts up a new one, and everything works fine again.

What could be causing syslog-ng to be started twice? I have looked through all files in /jffs/scripts, looking for any reference to scribe or syslog-ng. The only relevant call is in service-event, where it belongs.

There's nothing in /etc/init.d calling scribe or syslog-ng. In /opt/init.d, I have rc.func.syslog-ng and S01syslog-ng files, both of which are freshly replaced by today's re-install of Scribe. Nothing else there seems to reference syslog-ng.

Any ideas what is happening here, or how to track it down?

I also see multiple instances of these processes. Is that normal?

 7084    admin     3424     S N      {taildnstotal-ca} /bin/sh /jffs/addons/uiDivStats.d/taildns.d/taildnstotal-cache
7105    admin     3424     S N      {taildnstotal-ca} /bin/sh /jffs/addons/uiDivStats.d/taildns.d/taildnstotal-cache
9529    admin     3556     S        {uiDivStats} /bin/sh /jffs/scripts/uiDivStats querylog
16155    admin     3556     S        {uiDivStats} /bin/sh /jffs/scripts/uiDivStats querylog
19540    admin     3556     S        {uiDivStats} /bin/sh /jffs/scripts/uiDivStats querylog
29162    admin     3556     S        {uiDivStats} /bin/sh /jffs/scripts/uiDivStats querylog

 

[Ripshod]

You could try the gamma, but check the prerequisites first regarding syslog.log location and symlink.

Scribe - Scribe 2.5.0 Testers Needed

NB: I MAKE NO PROMISES!!! Don't get mad if you lose your log files, you have been warned! Since ASUS is now putting the default syslog.log on /jffs instead of /tmp, this is causing problems for scribe. I don't have one of those new-fangled routers, so I can only test that it doesn't...

www.snbforums.com

 

[ScottW]

Update...

I did a backup of the USB drive and set it up again with amtm. Before, it was two partitions: partition-1 was 80gb ext4, and partition-2 was unformatted. This time I set it up as one ext 4 partition using the entire drive. Used amtm to create the 4gb swap file, as before, then restored the backup.

After a reboot... Only a single syslog-ng process, and syslog-ng is logging properly. Not really sure if it was something about the empty partition, or maybe something else amiss with the formatting -- but it seems to be operating much better now!

 

[Ripshod]

It's strange though, that I have two syslog-NG processes with one sticking on zero runtime, yet no such errors.
Glad you found your solution

 

[ScottW]

It's strange though, that I have two syslog-NG processes with one sticking on zero time, yet no such errors.
Glad you found your solution
View attachment 55189

Well... My linux knowledge is really limited, so I could be off base here but...

I *think* the "white" line in your 'htop' snapshot is the main process -- and the "green" line(s) are worker threads. So, you have only (1) main process, and (1) worker thread. If you run "ps | grep "syslog-ng", it will only show the main process (pid=10356).

The problem I had was (2) main syslog-ng processes, i.e., (2) of the "white" syslog-ng lines. When I ran ps | grep "syslog-ng", it returned (2) main syslog-ng processes (and two of the supervisors).

I still can't figure out what was causing that -- or why a backup/reformat/restore would stop it, when uninstalling/reinstalling scribe and syslog-ng had no effect.

 

[Ripshod]

Thanks for the clarity. I wasn't particularly worried about it as it's running fine, just pointing out the two processes as you had explained.

I still can't figure out what was causing that -- or why a backup/reformat/restore would stop it, when uninstalling/reinstalling scribe and syslog-ng had no effect.

Mysteries, gotta love them.

 

[jc259]

It's strange though, that I have two syslog-NG processes with one sticking on zero runtime, yet no such errors.
Glad you found your solution
View attachment 55189

@Ripshod What did you run to get that view of the processes?

 

[Ripshod]

I ran htop over SSH and filtered (f4) "syslog".