'nvram get' lookup causes script hang

[Viktor Jaep]

Hi everyone... I need your advice, please. I have been dealing with this strange issue ever since I got my RT-AC86U and started playing around with scripts... Every so often, randomly... could happen a few times a day, or go days and days before I ever see it happen... but a simple "nvram get <value>" can cause my script to just hang... indefinitely. Until I kill it with a CTRL-C. Lately, in trying to troubleshoot this... I will hop on "htop", sort it by 'nvram', and will find the culprit "nvram get" task just sitting there, drawing 0% cpu and no activity indicated... so when I kill that task in htop, the script will continue running without needing to restart it.

I have been able to get around this issue by putting a band-aid on the problem, but it doesn't solve the core issue... thanks to @eibgrad, he turned me onto the "timeout <sec>" command, which needs to be installed separately and be inserted before each "nvram get <value>" command... which makes it a real PITA to keep adding/removing when sharing these scripts. This does seem to help the issue, and allows the scripts to run indefinitely as it will then invoke the timeout function when the "nvram get" hangs. The other thing I have done is created another script that runs hourly through cru to check and see if the script is hung, and if so, kills it and restarts it. But you *shouldn't* have to go through all this trouble I would think - a router should just work!

I have not found any other references to this anywhere, nor do I see anyone else bringing this issue up as a flaw on their router... and my scripts seem to run flawlessly on other people's routers. So this gets me thinking -- am I the only one with this issue - and why?

1.) An "nvram get" command is pretty low-level -- could there be a hardware issue with my router? I'm not dealing with any overheating, and even have a CPU fan tied directly to it, keeping a cool 53C at all times.

2.) Could there be some sort of a conflict where possibly some other program is calling the same values at the same time, or perhaps has a lock on calling nvram variables? Like with SQL - concurrent transaction locking/hangups... I'm not sure if the router can handle concurrent calls if this might be the reason why?

3.) Could this be a sign of needing to wipe the router and start from scratch? I've had it for about 1.5yrs. Perhaps it's due time?

4.) Any other software/config-related tweaks I could look at or perform to see if it will alleviate this issue?


I really appreciate your help, input and advice on this!

 

[ColinTaylor]

Are you seeing any bcm63xx_nand errors in the syslog?

 

[dave14305]

Stuck commands

Has anyone else observed any "stuck" commands on their router since upgrading to 386.x? Below is a frequent occurrence of the wl command getting stuck being spawned by conn_diag (this process was created at 4:10 AM). But I've also seen it happen with some nvram get commands (I once had Skynet...

www.snbforums.com

I have seen this quite regularly.

 

[Viktor Jaep]

Are you seeing any bcm63xx_nand errors in the syslog?

I don't seem to have these errors... ;(

 

[Viktor Jaep]

Stuck commands

Has anyone else observed any "stuck" commands on their router since upgrading to 386.x? Below is a frequent occurrence of the wl command getting stuck being spawned by conn_diag (this process was created at 4:10 AM). But I've also seen it happen with some nvram get commands (I once had Skynet...

www.snbforums.com

I have seen this quite regularly.

OMG... this is the first I'm seeing this... so yeah - same issue! So I'm guessing it's more widespread than I thought! I'm glad I'm not the only one!

 

[Viktor Jaep]

Wondering now if this is a AC86U flaw of some kind. Just found some other references to it hanging during NVRAM backups and such on a regular basis... and the fix was to replace it with an AX86U.

 

[SomeWhereOverTheRainBow]

Wondering now if this is a AC86U flaw of some kind. Just found some other references to it hanging during NVRAM backups and such on a regular basis... and the fix was to replace it with an AX86U.

do you know at what point in your script it is hanging on the nvram get?

 

[Viktor Jaep]

do you know at what point in your script it is hanging on the nvram get?

Lol. Pick any 'nvram get' statement, and it's hung on most of them.

 

[MarkusI]

Hey,
having the same issue... occurs also randomly. Seems like the UI also stops working in that case (I suppose it uses the same mechanism to create the status overview).

Seems like until now, no FW update has fixed it. Don't know, if it is possible at all...
Is it still happening on your devices?

PS: we're having 4 of these devices in an AiMesh setup... not being able to fix it without replacing the devices would be really costly

 

[bennor]

@Markusl, you have necroposted an almost three year old thread. It would help if you included your router model(s) and firmware versions to provide more context about your setup. And you may want to include what scripting command you are running so others can review it to see if the verbiage is correct.

 

[alan6854321]

Yes, I still see this happening, pretty much at random.
The one I notice most often is the 'nvram save xxxxx' command in my backup script.
And still hoping for a fix after three years!

 

[Viktor Jaep]

There were one or two other threads opened up that discussed this issue... and I believe there were some suggestions made on changing some variable to a higher number that helped prevent this issue from happening over a longer span in time. If you can't find it, let me know, and I'll spend a few minutes looking for it after the work day ends.

But in the end, the AC86U was definitely flawed. The real fix is upgrading your router to a newer model.

 

[dave14305]

some suggestions made on changing some variable to a higher number that helped prevent this issue from happening over a longer span in time.

Asus AC86U - nvram show stops working after few hours causing router not accessible via WebUI

Hi, I came across a strange problem with my AC86U running latest merlin firmware (but it was happening on older as well). Basically router becomes unaccessible via WebUI after few hours after reboot. Just login screen shows up and after putting in login details it doesn't continue any further...

www.snbforums.com

 

[alan6854321]

But in the end, the AC86U was definitely flawed. The real fix is upgrading your router to a newer model

No it isn't - I still see nvram hangs on my AX88U Pro

 

[MarkusI]

@Markusl, you have necroposted an almost three year old thread. It would help if you included your router model(s) and firmware versions to provide more context about your setup. And you may want to include what scripting command you are running so others can review it to see if the verbiage is correct.

Yeah, true... my bad...
AC86U and I had the issue with the newest Merlin _and_ with the newest stock firmware (last one today).

We're running an AiMesh setup with 4 AC86Us and 3 AC68Us of which the latter seem to be very stable, while the AC86U central router and the AC86U nodes regularly hang up or are showing strange behavior.

I am actually not running any scripts, but I observed that when the UI of the main router stops responding, also nvram commands (e.g. nvram get cfg_device_list) will hang. If nvram get cfg_device_list works, the UI works, too.

Thus, I concluded the UI issues is maybe just another symptom of nvram get cfg_device_list hanging, which is why I replied to this thread.

 

[MarkusI]

No it isn't - I still see nvram hangs on my AX88U Pro

Do you see other issues, like e.g. the UI not responding in theses cases, too?
I actually do also have an AX88U Pro (current Merlin-FW) in another location but in a non-AiMesh-setup and I never saw any issues like the UI not responding.
Might be related to AiMesh!?

edit: just read the other thread posted above... yes, it seems to be related to AiMesh... :/

 

[alan6854321]

Do you see other issues, like e.g. the UI not responding in theses cases, too?

No, but then I don't use the UI very much.