RT-AC3100 slow GUI when 5GHZ radio is on

[Peter Dolan]

Hi all,
I'm not certain if anybody will be able to provide any advice or not about this issue (see below), but since I haven't seen my exact symptoms mentioned I thought mentioning it would serve the dual purpose of potentially providing a solution, but also alerting others who might have similar issues.

The symptoms right now:
If the 5GHZ radio is set to "on" the web-based admin becomes EXTREMELY unresponsive (it takes about 5 minutes to change pages between menu items). When the 5GHZ radio is set to "off" the page is once again responsive. No 5GHZ SSID is visible.

Another symptom-- when 5GHZ is on the 2GHZ band stops working correctly (SSID is visible but connection attempts are refused with errors like "incorrect password"-- I suspect that the 5GHZ radio's activation slows processing of various events on the router and this interferes with the router's password authentication code. Any advice on how to check for this would be greatly appreciated.

I've tried factory resets, switching to MERLIN firmware, and switching back to the latest stock (both have the slow UI issue). I have not tried down-grading to an older firmware (but see two paragraphs below).

The issue is *probably* just a bad radio, but even if that's the end of the story, I'd like to understand why this manifests as such a dramatic slow-down in the UI. There's a few other twists that make me wonder if the issue isn't related to some subtle interaction between firmware and radio:

It arrived from ebay with an older firmware which I did not write down-- I did not suffer the slow UI on initial configuration. In fact, I didn't have the issue until AFTER I tried to add the RT-AC3100 as an AI mesh node to my RT-AC68U. That failed (which would be consistent with a bad radio), however I subsequently added the AC68U as the AI mesh node using the AC3100 as the master... and that worked fine at first.... that's when I first noticed the slow UI and I lost the ability to connect via wifi-- Subsequent changes to the firmware and factory resets did not fix the issue. And it was only much later that I discovered the connection between 5GHZ and performance. That's why I'm still holding out a *sliver* of hope that it's a software/hardware/firmware interaction issue. That said, I've never seen an SSID for 5G from the AC3100.

SSHing into the router is painless and there does not seem to be any issues with high CPU or memory usage (I haven't figured out how to check for thermal throttling-- the temperatures do seem to be in line with what other people have reported so I don't think that's the issue).

So here's my wish list:
* Advice on troubleshooting a 5GHZ radio on an RT-AC3100.
* Advice/information on event handling in the linux variant used on the router and how to troubleshoot/observe that.

 

[kfp]

The “incorrect password” happens a lot when there is interference, I doubt this is software related and if I had to guess you probably have bad hardware.

Since it came from eBay I’m going to assume it’s out of warranty, maybe you can pop it open and see if there are any blown/burnt components inside.

As for logs, try looking at dmesg, it might have additional information not present in syslog. Maybe toggle 5G on and off couple times then look at dmesg.

 

[Peter Dolan]

The “incorrect password” happens a lot when there is interference, I doubt this is software related and if I had to guess you probably have bad hardware.

Since it came from eBay I’m going to assume it’s out of warranty, maybe you can pop it open and see if there are any blown/burnt components inside.

As for logs, try looking at dmesg, it might have additional information not present in syslog. Maybe toggle 5G on and off couple times then look at dmesg.

Thanks for the advice. I think you're right about the bad hardware. I've made some progress in *why* the bad hardware causes the UI issue:

I monitored CPU usage via ssh and `top`. When I toggled the 5G on and off using `radio off 1` and `radio on 1` I saw /usr/sbin/acsd usage jump up to 49%-- I'm not at all sure why I didn't see that before... but I didn't. This time it's perfectly correlated-- radio 1 on-> /usb/sbin/acsd usage high, radio 1 off-> /usb/sbin/acsd usage low (and the UI works properly again). Whatever is happening slows down commands like `wl ver`-- the command just hung until I turned the radio off again (two different ssh sessions). I'm sure it's a bad radio... but at least I have better idea of how that radio is causing the issue. Since that's a broadcom wireless driver it makes sense to me that a 5G issue could cause that process to go batty.

EDIT: It's not /usr/sbin/acsd-- killing the process doesn't fix the problem.

Thanks again for your help.

 

[indark]

you mentioned about switching between merlin and stock firmware, so here is something that you can try
try clear nvram to see if it help on the CPU load issue.
power off the router for 30 sec, hold wps button while powering on
wait till power led to start blinking/flashing, release the wps button and wait till the router finish initialize

 

[Peter Dolan]

you mentioned about switching between merlin and stock firmware, so here is something that you can try
try clear nvram to see if it help on the CPU load issue.
power off the router for 30 sec, hold wps button while powering on
wait till power led to start blinking/flashing, release the wps button and wait till the router finish initialize

No luck I'm afraid. Thanks for the suggestion.

 

[kfp]

No luck I'm afraid. Thanks for the suggestion.

Can you post the output of dmesg?

 

[Peter Dolan]

Can you post the output of dmesg?

I've attached the complete output of dmesg from after rebooting with 5G disabled.

The tail of dmesg after enabling and then disabling 5G is below ([*] marks the same location in the complete dmesg that I've attached)

[*] device wds0.1 entered promiscuous mode
br0: topology change detected, propagating
br0: port 3(wds0.1) entering forwarding state
br0: port 3(wds0.1) entering forwarding state
br0: port 3(wds0.1) entering forwarding state
device wds0.1 left promiscuous mode
br0: port 3(wds0.1) entering disabled state
br0: port 2(wl0.1) entering forwarding state
device wl0.1 left promiscuous mode
br0: port 2(wl0.1) entering disabled state
device wl0.1 entered promiscuous mode
br0: topology change detected, propagating
br0: port 2(wl0.1) entering forwarding state
br0: port 2(wl0.1) entering forwarding state​

 

[indark]

while it seems to be hardware issue, here is what I would suggest to try:
1. forgot to ask how you setup the 2.4Ghz and 5Ghz, are you using same ssid for both band? (still prefer separate SSID myself ;p)
2. and when I saw wds message, I assumed you still have ai-mesh setup while testing.
I would suggest try to test without ai-mesh.
3. it looks like there is also a usb device attached, try check without the usb as well.

basically clear nvram, and try to just keep ac3100 as close to default as possible. And see if you still experience the issue.
If you dont, you can slowly adding ai-mesh, and usb 1 at a time, to see if you can id where things might go wrong.

While it doesnt match your case, I recently had experience slow ui with ac3100. it was caused by norton security extension on chrome. Once I removed it, ui is smooth.

 

[Peter Dolan]

while it seems to be hardware issue, here is what I would suggest to try:
1. forgot to ask how you setup the 2.4Ghz and 5Ghz, are you using same ssid for both band? (still prefer separate SSID myself ;p)
2. and when I saw wds message, I assumed you still have ai-mesh setup while testing.
I would suggest try to test without ai-mesh.
3. it looks like there is also a usb device attached, try check without the usb as well.

basically clear nvram, and try to just keep ac3100 as close to default as possible. And see if you still experience the issue.
If you dont, you can slowly adding ai-mesh, and usb 1 at a time, to see if you can id where things might go wrong.

While it doesnt match your case, I recently had experience slow ui with ac3100. it was caused by norton security extension on chrome. Once I removed it, ui is smooth.

Thanks for the suggestions. I've tried them all. 2.4 Ghz with same name, without same name, with smart connect active and not active. With ai mesh, without ai mesh. With stock firmware with Merlin's firmware. With USB and without (I installed the USB to make it easier to install some packages). It is the strangest thing. If the 5G radio is activated via the GUI then the GUI slows down terribly. It's not *just* the GUI-- a number of shell commands show the same lag when I'm ssh'ed into the router (`wl status`, for example). If I subsequently turn off the radio via ssh (radio off 1), or if I turn it off via the gui then performance is restored for everything that's being laggy.

I had seen a 49% CPU usage (the acsd process) during one of my many attempts to understand what was occurring... but CPU usage doesn't necessarily go up in all the scenarios that I've tried; and killing that process doesn't fix the problem in any event. If the 5G radio is disabled during reboot, then turning the radio on via ssh (radio on 1) does not cause the problem. So it's not purely about the radio being on, but the radio being on in such a way that the OS expects to interact with it. Activating the radio via the GUI seems to do a bit more than just turn the radio on.

I've used acrylic to watch the activity on the 2.4G and 5G bands while I experiment and there's never been even a hint of 5G activity so I'm calling the 5G radio DOA. It's just a matter of curiosity about how the antenna and the OS interact at this point.

Because I'm stubborn and like to understand things I'm trying to figure out how SMI works on Linux to see if that's where the problem is arising-- perhaps some driver that's supposed to manage an interrupt is waiting for a signal to arrive and has to time out before control is returned to the main OS. I'm not seeing anything obviously connected to any of this in the logs, but this is my first time playing with an embedded distro so I might be missing something obvious.

I'll let everybody know if I find anything. Again, I appreciate everybody who take the time to offer suggestions. This is a friendly community and I appreciate that.

 

[kfp]

I've attached the complete output of dmesg from after rebooting with 5G disabled.

The tail of dmesg after enabling and then disabling 5G is below ([*] marks the same location in the complete dmesg that I've attached)

[*] device wds0.1 entered promiscuous mode
br0: topology change detected, propagating
br0: port 3(wds0.1) entering forwarding state
br0: port 3(wds0.1) entering forwarding state
br0: port 3(wds0.1) entering forwarding state
device wds0.1 left promiscuous mode
br0: port 3(wds0.1) entering disabled state
br0: port 2(wl0.1) entering forwarding state
device wl0.1 left promiscuous mode
br0: port 2(wl0.1) entering disabled state
device wl0.1 entered promiscuous mode
br0: topology change detected, propagating
br0: port 2(wl0.1) entering forwarding state
br0: port 2(wl0.1) entering forwarding state​

It's interesting that one interface is named wl0 and the other is wds0. Do you have repeater mode or WDS on? This should be cleared when you cleared NVRAM so I'm not sure this is directly the cause but thought I'd point that out.

Also, how were you clearing NVRAM? Maybe try clearing it using the WPS button first, then after it reboots "Initialize" it.
Attaching a serial cable might be another option too if you want to dive deeper.

 

[Sinner]

Id also like to suggest a full power cycle after fw reset. For whatever reason the radios can retain previous instructions if not fully power cycled after a hard reset.

 

[Sinner]

And what happens sometimes is it become incompatible with the new firmware due to changes that have bene made

 

[Sinner]

Also try clearing your browser cache ctrl or shift f5. may be trying to used an older cached version since you updated.

 

[Peter Dolan]

It's interesting that one interface is named wl0 and the other is wds0. Do you have repeater mode or WDS on? This should be cleared when you cleared NVRAM so I'm not sure this is directly the cause but thought I'd point that out.

Also, how were you clearing NVRAM? Maybe try clearing it using the WPS button first, then after it reboots "Initialize" it.
Attaching a serial cable might be another option too if you want to dive deeper.

I don't have repeater mode or WDS on (as near as I can tell... I'm still learning about network interfaces and how to manipulate/view them properly). Perhaps the wds0 interfaces is associated with the guest network? (The dmesg I included was from a reboot after I reconfigured the router for my home setup). I've ordered a new router so I'll be free to tinker with this one in a week or so (my partner's a bit upset with the constant up/down internet access). I have done both the WPS button and the initialize but not both one after the other.

I should still have an RS232 cable lying around somewhere .

 

[Peter Dolan]

Id also like to suggest a full power cycle after fw reset. For whatever reason the radios can retain previous instructions if not fully power cycled after a hard reset

And what happens sometimes is it become incompatible with the new firmware due to changes that have bene made

Also try clearing your browser cache ctrl or shift f5. may be trying to used an older cached version since you updated.

Thanks for the suggestions... I've done a few power cycles after the upgrade and even used a fresh browser in private/incognito mode after flushing DNS on the command line. It doesn't change the behavior-- there is also a lag issue tied to the status of the 5G radio that occurs via ssh when I use the `wl status` command and that's independent of the browser.

I can remove the lag (without reboot) on both the ssh command and the UI just by turning off the 5G radio `radio off 1` and then restore the lag (what a phrase!) by turning on the 5G radio `radio on 1`.

The `wl status` command is particularly interesting because it involves no network access (other than the need to ssh into the router). As a result almost all of the usual complicating factors are taken out of the equation-- it's purely the OS interacting with the hardware via the drivers... I'm looking into the source code to figure out the precise interaction, but I've not done any linux system programming so there's a learning curve, but it's a good excuse to learn something fun

 

[Sinner]

Thanks for the suggestions... I've done a few power cycles after the upgrade and even used a fresh browser in private/incognito mode after flushing DNS on the command line. It doesn't change the behavior-- there is also a lag issue tied to the status of the 5G radio that occurs via ssh when I use the `wl status` command and that's independent of the browser.

I can remove the lag (without reboot) on both the ssh command and the UI just by turning off the 5G radio `radio off 1` and then restore the lag (what a phrase!) by turning on the 5G radio `radio on 1`.

The `wl status` command is particularly interesting because it involves no network access (other than the need to ssh into the router). As a result almost all of the usual complicating factors are taken out of the equation-- it's purely the OS interacting with the hardware via the drivers... I'm looking into the source code to figure out the precise interaction, but I've not done any linux system programming so there's a learning curve, but it's a good excuse to learn something fun

Haha right on. Well good luck with that. Ive enjoyed figuring this stuff out as well. But i have no lag on my 3100. I have all the beamforming off etc as i have too many mixed devices. If u havent try turning all that off

 

[kfp]

I don't have repeater mode or WDS on (as near as I can tell... I'm still learning about network interfaces and how to manipulate/view them properly). Perhaps the wds0 interfaces is associated with the guest network? (The dmesg I included was from a reboot after I reconfigured the router for my home setup).

No, the wds0 is definitely your 5Ghz. The wl0 is the 2.4 and wl1 is usually the name for 5Ghz. Guest networks on each would then be named wl0.1, wl0.2, wl1.1 etc

 

[Peter Dolan]

No, the wds0 is definitely your 5Ghz. The wl0 is the 2.4 and wl1 is usually the name for 5Ghz. Guest networks on each would then be named wl0.1, wl0.2, wl1.1 etc

Well THAT's interesting! I'll poke at that after my new router arrives

 

[kfp]

Well THAT's interesting! I'll poke at that after my new router arrives

I just don’t know how the driver is stuck with it set to WDS, even after resets and power cycles, maybe pop the case open to see if there are any hardware defects.