Strange message being spammed in system log

[Rocketboy235]

Hello,

I noticed my logs are being spammed with the following message every 4 minutes and I'm not sure why. Curious if anyone ran into a similar issue or may have any insight. Didn't see anyone else experience this issue on the forums but maybe I didn't search enough.

I have the AC68U and it is running Merlin 386.1_2

May 14 19:04:00 crond[10222]: can't change directory to '/dev/null'
May 14 20:32:00 crond[19459]: can't change directory to '/dev/null'
May 14 20:36:00 crond[19876]: can't change directory to '/dev/null'
May 14 20:40:00 crond[20297]: can't change directory to '/dev/null'
May 14 20:44:00 crond[20716]: can't change directory to '/dev/null'
May 14 20:48:00 crond[21136]: can't change directory to '/dev/null'
May 14 20:52:00 crond[21555]: can't change directory to '/dev/null'
May 14 20:56:00 crond[21971]: can't change directory to '/dev/null'
May 14 21:00:00 crond[22391]: can't change directory to '/dev/null'
May 14 21:04:00 crond[22816]: can't change directory to '/dev/null'
May 14 21:08:00 crond[23241]: can't change directory to '/dev/null'
May 14 21:12:00 crond[23657]: can't change directory to '/dev/null'
May 14 21:16:00 crond[24075]: can't change directory to '/dev/null'
May 14 21:20:00 crond[24495]: can't change directory to '/dev/null'
May 14 21:24:00 crond[24897]: can't change directory to '/dev/null'
May 14 21:28:00 crond[25317]: can't change directory to '/dev/null'
May 14 21:32:00 crond[25754]: can't change directory to '/dev/null'
May 14 21:36:00 crond[26172]: can't change directory to '/dev/null'
May 14 21:40:00 crond[26721]: can't change directory to '/dev/null'
May 14 21:44:00 crond[27139]: can't change directory to '/dev/null'
May 14 21:48:00 crond[27580]: can't change directory to '/dev/null'
May 14 21:52:00 crond[28026]: can't change directory to '/dev/null'
May 14 21:56:00 crond[28459]: can't change directory to '/dev/null'

Thanks for your time!

 

[eibgrad]

Looks like a misconfigured entry in the scehduler. If not by you, perhaps a third-party script. You can dump the scheduled jobs from an ssh session and perhaps find the offender.

cru l

 

[Rocketboy235]

Looks like a misconfigured entry in the scehduler. If not by you, perhaps a third-party script. You can dump the scheduled jobs from an ssh session and perhaps find the offender.

cru l

Thanks, eibgrad. I don't think I added any new scheduler jobs. When I ran the command, this is what was returned:

I know that I run the OpenVPN server functionality. Also, the cron job seems to be configured to execute every 2 minutes while I see an error in the log every 4 minutes. This is what is shown when I check out the .sh script.

 

[ColinTaylor]

Thanks, eibgrad. I don't think I added any new scheduler jobs. When I ran the command, this is what was returned:

View attachment 33851

Was that the only line in your crontab?

 

[Rocketboy235]

Was that the only line in your crontab?

Yes

Also, custom scripts/configs should be off.

My router has been online for 82-83 days though and I think this issue wasn't happening during the start so something must have happened in between.

 

[ColinTaylor]

It's strange that there were no messages between 19:04 and 20:32. Are the messages still continuing?

If you turn off the VPN server does cron entry disappear and the messages stop?

 

[ColinTaylor]

It would appear that there is another cron table for a user account that has /dev/null as its home directory. My first guess would be the nobody account, used by things like dnsmasq and ntpMerlin.

Can you do the following please:

ps w | grep crond

cd /etc
grep null passwd

ls -l /var/spool/cron/crontabs

cat /var/spool/cron/crontabs/nobody

 

[Rocketboy235]

It's strange that there were no messages between 19:04 and 20:32. Are the messages still continuing?

If you turn off the VPN server does cron entry disappear and the messages stop?

Just checked logs again today and it's now executing every 2 minutes though interestingly maybe there are 2 things going on now since I see it where it happens every 2 minutes and another one where it happens every 4 minutes. But I think it's because I was looking at the process list before you asked me to check and I ran the crond process and that probably explains why it's running every 2 minutes now.

I'll try to turn off the VPN server and see what happens and report back.

It would appear that there is another cron table for a user account that has /dev/null as its home directory. My first guess would be the nobody account, used by things like dnsmasq and ntpMerlin.

Can you do the following please:

ps w | grep crond

cd /etc
grep null passwd

ls -l /var/spool/cron/crontabs

cat /var/spool/cron/crontabs/nobody

Now that I think about it. I think within the past 2 months, I did change the username and password of the administrative account (and because I changed the admin login username to the new username, that would change the default OpenVPN login username as well so what I did was I added the old username to the OpenVPN user access list).

Sorry if the redaction of the username in my screenshots is a bit annoying.
So the white box is the old admin username and the gray box is the current admin username

I accidently ran crond just an hour and a half ago so might be why you see another crond process in the ps list

I wonder if it somehow did not properly remove the old user account after I changed the admin login username credentials.

 

[ColinTaylor]

Yeah it's a bit difficult trying to guess what's happened to the user accounts and whether it's intentional or a bug. I would have thought that all traces of the old admin account should have been removed. There's been reports of the accounts still persisting in things like the Samba shares.

Try looking at that old crontab, it might provide some more clues as to what's creating it.

cat /var/spool/cron/crontabs/oldadmin

Where oldadmin is the name in the white box.

 

[Rocketboy235]

Just reporting back on what happens if I turn off the OpenVPN server... appears that nothing changed and the logs are spammed at the same rate as before.

Yeah it's a bit difficult trying to guess what's happened to the user accounts and whether it's intentional or a bug. I would have thought that all traces of the old admin account should have been removed. There's been reports of the accounts still persisting in things like the Samba shares.

Try looking at that old crontab, it might provide some more clues as to what's creating it.

cat /var/spool/cron/crontabs/oldadmin

Where oldadmin is the name in the white box.

I also checked the new account and it only shows the vpn-watchdog1.sh in that account's cron table.

Also, regarding the log I posted in my first post.... sorry for the confusion but the first line should have been deleted since I copied and pasted a bunch of lines from the log but the error message was essentially the same so I deleted most of it but forgot to get rid of the first line as well. That probably led to your confusion why it seems that nothing happened for about an hour in between. 19:04 and 20:32

May 14 19:04:00 crond[10222]: can't change directory to '/dev/null'
May 14 19:08:00 crond[10645]: can't change directory to '/dev/null'
May 14 19:12:00 crond[11062]: can't change directory to '/dev/null'
May 14 19:16:00 crond[11478]: can't change directory to '/dev/null'
May 14 19:20:00 crond[11903]: can't change directory to '/dev/null'
May 14 19:24:00 crond[12318]: can't change directory to '/dev/null'
May 14 19:28:00 crond[12737]: can't change directory to '/dev/null'
May 14 19:32:00 crond[13167]: can't change directory to '/dev/null'
May 14 19:36:00 crond[13583]: can't change directory to '/dev/null'
May 14 19:40:00 crond[14002]: can't change directory to '/dev/null'
May 14 19:44:00 crond[14417]: can't change directory to '/dev/null'
May 14 19:48:00 crond[14838]: can't change directory to '/dev/null'
May 14 19:52:00 crond[15260]: can't change directory to '/dev/null'
May 14 19:56:00 crond[15676]: can't change directory to '/dev/null'
May 14 20:00:00 crond[16095]: can't change directory to '/dev/null'
May 14 20:04:00 crond[16527]: can't change directory to '/dev/null'
May 14 20:08:00 crond[16948]: can't change directory to '/dev/null'
May 14 20:12:00 crond[17367]: can't change directory to '/dev/null'
May 14 20:16:00 crond[17785]: can't change directory to '/dev/null'
May 14 20:20:00 crond[18205]: can't change directory to '/dev/null'
May 14 20:24:00 crond[18613]: can't change directory to '/dev/null'
May 14 20:28:00 crond[19033]: can't change directory to '/dev/null'
May 14 20:32:00 crond[19459]: can't change directory to '/dev/null'

 

[ColinTaylor]

Start by looking at your LetsEncrypt setup script (I don't use it myself). I suspect it has a hard-coded username in there from when it was first set up. I don't know where the CheckVPNServer script comes from.

 

[eibgrad]

I don't know where the CheckVPNServer script comes from.

That's just automatically added by the router when the VPN server is started. My router is just plain vanilla (no scripts, no nothing) and even it adds the watchdog.

 

[ColinTaylor]

That's just automatically added by the router when the VPN server is started. My router is just plain vanilla (no scripts, no nothing) and even it adds the watchdog.

Thanks for that. I can see it being added here and here by the startup process but what I don't understand is why it's doing it twice, once for the current admin account and again for the old account. That would imply that there's two instances of the VPN server 1 running.

 

[eibgrad]

Thanks for that. I can see it being added here and here by the startup process but what I don't understand is why it's doing it twice, once for the current admin account and again for the old account. That would imply that there's two instances of the VPN server 1 running.

Frankly, I don't why there's a watchdog for the server AT ALL. Since when has it become a problem having a running OpenVPN server just fail? If anything needs a watchdog, you would think it's the OpenVPN client! I've seen my share of commercial OpenVPN providers who send an AUTH FAIL to the client to kick them off the system, or even refuse to start, killing the openvpn process (since it's considered a fatal error). I didn't even know there was a watchdog until this thread!

 

[ColinTaylor]

I think it would be best if @Rocketboy235 just reset his router and set it up again from scratch using his preferred admin account name, and not try to reuse the old account name. In the long run its going to be less hassle.

 

[Rocketboy235]

I think it would be best if @Rocketboy235 just reset his router and set it up again from scratch using his preferred admin account name, and not try to reuse the old account name. In the long run its going to be less hassle.

Thanks for your help, Colin!

I ended up just renaming the old cron file to the new account name and got rid of the old account file in the cron folder. Rebooted router afterwards and seems to be okay. I should have just rebooted the router first and see if maybe the issue could have gone away by itself in case maybe the previous account username data isn't handled until after the next time the router reboots before I started renaming the files just to see if that could have just resolved it.

Unfortunately I am not on site to do a full reset at this time but I probably should consider doing a full factory reset one of these days the next time I'm on site.

Anyway, I guess this was a bug?

 

[ColinTaylor]

I ended up just renaming the old cron file to the new account name and got rid of the old account file in the cron folder. Rebooted router afterwards and seems to be okay. I should have just rebooted the router first and see if maybe the issue could have gone away by itself in case maybe the previous account username data isn't handled until after the next time the router reboots before I started renaming the files just to see if that could have just resolved it.

All along I've been assuming that you've rebooted the router at least once since you changed the account name because you said you renamed it about 2 months ago. The only thing I couldn't work out was why the old crontab was still present after a reboot as that is stored in RAM and recreated every time the router boots up.

 

[Rocketboy235]

All along I've been assuming that you've rebooted the router at least once since you changed the account name because you said you renamed it about 2 months ago. The only thing I couldn't work out was why the old crontab was still present after a reboot as that is stored in RAM and recreated every time the router boots up.

Sorry for the confusion. I did mention that it was up for 82 days earlier though I wonder if one changes the username, the router doesn't automatically reboot for those changes, right?

 

[ColinTaylor]

Sorry for the confusion. I did mention that it was up for 82 days earlier though I wonder if one changes the username, the router doesn't automatically reboot for those changes, right?

Yes I missed the bit about 82 days. I think you added that to your post after I had already replied to it. It might have saved a lot of time if I had noticed it. I haven't renamed the admin account on my router for years (so it would be a different firmware version anyway) and would also have assumed that would trigger a reboot but it sounds like it doesn't.

 

[ColinTaylor]

OK out of curiosity I've just tried renaming the admin account on my router and it doesn't do a reboot. So, mystery solved.