AC-5300 Web GUI Will Stop Responding After Authenticating; Requires Reboot of Router to Recover

[NickBurns]

I've been chasing this issue for quite a long time now spanning several releases of ASUSWRT. I am currently on the latest - 386_2.4 and have diligently updated as time has gone by and have twice factory defaulted and configured from scratch this router. I am looking to avoid that hassle again.

Before anyone feels the need to point out the release notes and the blurb about "after you upgrade you may need to wait xx" - please don't. I have had this issue going back several releases; months to a year+ -- it is NOT that issue. I will concede, when I did upgrade to that particular release I did incur a long wait period for the GUI to be operational -- but ultimately it was operational.

The router is in its problem state now. I am accessing it via HTTP vs HTTPS for sake of troubleshooting / sniffing, etc. It is no different on HTTPS.

Now after I log in, I am seeing the following - the requests to the GUI stop... requests get held up in (pending) and the GUI is otherwise dead.

Take note of client_functions.js returning on 269 bytes. This file is 129k.

admin@RT-AC5300:/tmp# ls -al /www/client_function.js
-rw-rw-r-- 1 admin root 129409 Apr 30 17:32 /www/client_function.js

netstat shows I have requests queued

admin@RT-AC5300:/tmp# netstat -anp|grep -i ":80"
tcp 0 0 127.0.0.1:80 0.0.0.0:* LISTEN 29930/httpd
tcp 4 0 192.168.1.1:80 0.0.0.0:* LISTEN 29930/httpd
tcp 365 0 192.168.1.1:80 192.168.1.40:1024 ESTABLISHED -
tcp 396 0 192.168.1.1:80 192.168.1.40:6265 ESTABLISHED -
tcp 389 0 192.168.1.1:80 192.168.1.40:11764 ESTABLISHED 29930/httpd
tcp 375 0 192.168.1.1:80 192.168.1.40:14459 ESTABLISHED -
tcp 0 0 192.168.1.1:80 192.168.1.40:14586 ESTABLISHED 29930/httpd
tcp 371 0 192.168.1.1:80 192.168.1.40:32513 ESTABLISHED -

Now if I manually restart the GUI (service restart_httpd) the login page will come back; but once I log in I'm back at the same problem point; requests are queuing.

admin@RT-AC5300:/tmp# netstat -anp|grep -i ":80"
tcp 0 0 127.0.0.1:80 0.0.0.0:* LISTEN 31428/httpd
tcp 4 0 192.168.1.1:80 0.0.0.0:* LISTEN 31428/httpd
tcp 375 0 192.168.1.1:80 192.168.1.40:1024 ESTABLISHED -
tcp 396 0 192.168.1.1:80 192.168.1.40:24200 ESTABLISHED -
tcp 389 0 192.168.1.1:80 192.168.1.40:1028 ESTABLISHED -
tcp 0 0 192.168.1.1:80 192.168.1.40:1025 TIME_WAIT -
tcp 0 0 192.168.1.1:80 192.168.1.40:15996 TIME_WAIT -
tcp 365 0 192.168.1.1:80 192.168.1.40:11350 ESTABLISHED -
tcp 371 0 192.168.1.1:80 192.168.1.40:17038 ESTABLISHED 31428/httpd
tcp 0 0 192.168.1.1:80 192.168.1.40:2703 TIME_WAIT -
tcp 0 0 192.168.1.1:80 192.168.1.40:1026 ESTABLISHED 31428/httpd
tcp 0 0 192.168.1.1:80 192.168.1.40:33010 TIME_WAIT -
tcp 0 0 192.168.1.1:80 192.168.1.40:13728 TIME_WAIT -
tcp 0 0 192.168.1.1:80 192.168.1.40:14586 TIME_WAIT -
tcp 0 0 192.168.1.1:80 192.168.1.40:3699 TIME_WAIT -
tcp 0 0 192.168.1.1:80 192.168.1.40:4765 TIME_WAIT -

If I reboot the router things will work for several hours, days but then at some point will stop. There is nothing in the syslog to indicate a problem.

I turned on HTTPD_DEBUG flag to get /jffs/HTTPD_DEBUG.log and all it shows is it hangs on what is the last request:

Fri May 28 10:52:00 2021 [info] [do_html_post_and_get(2646)]ost_buf = , query = (null)

Fri May 28 10:52:00 2021 [info] [handle_request(1235)]:IP(192.168.1.40), file = client_function.js
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36


Fri May 28 10:52:00 2021 [info] [do_html_post_and_get(2646)]ost_buf = , query = (null)

Tail end of syslog for today's quick tests:

May 28 10:39:00 rc_service: service 27522:notify_rc restart_letsencrypt
May 28 10:39:00 custom_script: Running /jffs/scripts/service-event (args: restart letsencrypt)
May 28 10:47:20 HTTPD: waitting 10 minitues and restart
May 28 10:47:20 rc_service: httpds 10101:notify_rc restart_httpd
May 28 10:47:20 custom_script: Running /jffs/scripts/service-event (args: restart httpd)
May 28 10:47:20 RT-AC5300: start https:8443
May 28 10:47:20 RT-AC5300: start httpd:80
May 28 10:47:22 httpd: Succeed to init SSL certificate...8443
May 28 10:47:22 httpd: Succeed to init SSL certificate...80
May 28 10:49:35 dropbear[30749]: Child connection from 192.168.1.40:1041
May 28 10:49:35 dropbear[30749]: Password auth succeeded for 'admin' from 192.168.1.40:1041
May 28 10:51:43 rc_service: service 31378:notify_rc httpd_restart
May 28 10:51:43 custom_script: Running /jffs/scripts/service-event (args: httpd_restart)
May 28 10:51:48 rc_service: service 31415:notify_rc restart_httpd
May 28 10:51:48 custom_script: Running /jffs/scripts/service-event (args: restart httpd)
May 28 10:51:48 RT-AC5300: start https:8443
May 28 10:51:48 RT-AC5300: start httpd:80
May 28 10:51:51 httpd: Succeed to init SSL certificate...8443
May 28 10:51:51 httpd: Succeed to init SSL certificate...80
May 28 11:02:50 HTTPD: waitting 10 minitues and restart
May 28 11:02:50 rc_service: httpd 31428:notify_rc restart_httpd
May 28 11:02:50 custom_script: Running /jffs/scripts/service-event (args: restart httpd)
May 28 11:02:51 RT-AC5300: start https:8443
May 28 11:02:51 RT-AC5300: start httpd:80
May 28 11:02:52 httpd: Succeed to init SSL certificate...8443
May 28 11:02:52 httpd: Succeed to init SSL certificate...80


There is firewall on this machine I'm testing from right now; Windows Firewall is disabled. I also get the same experience from my phone browser as well as other machines I have / other distros.

Usually I let this stuff slide and 'live with it' but it's getting in the way now of me doing things I need to do on the router when I need to do it and not schedule reboots that interfere with the overall network in the house. I have kids... kids don't know these days what life was like prior to the internet.


I've already tried the following:

Cleared Browser Caches (tech support's #1 go-to / time waster)
Toggle /proc/sys/net/ipv4/tcp_sack 1 to 0... no change
Toggle /proc/sys/net/ipv4/tcp_tw_recycle | tcp_tw_reuse... no change
I read somewhere in these forums this could be caused by the ASUS Android App. I have since removed remote access and the app from my device... no change


So if I can please ask the community here for help, it would be appreciated. I'd like to solve this one. Thank you.

 

[NickBurns]

I left the system alone with TCP sessions in ESTABLISHED state as above... the below entries - I did not initiate a restart @ 11:16.

May 28 11:02:51 RT-AC5300: start httpd:80
May 28 11:02:52 httpd: Succeed to init SSL certificate...8443
May 28 11:02:52 httpd: Succeed to init SSL certificate...80
May 28 11:16:52 HTTPD: waitting 10 minitues and restart
May 28 11:16:52 rc_service: httpd 2148:notify_rc restart_httpd
May 28 11:16:52 custom_script: Running /jffs/scripts/service-event (args: restart httpd)
May 28 11:16:53 RT-AC5300: start https:8443
May 28 11:16:53 RT-AC5300: start httpd:80
May 28 11:16:54 httpd: Succeed to init SSL certificate...80
May 28 11:16:54 httpd: Succeed to init SSL certificate...8443

So at this point the GUI was back at the login page. However wash, rinse, repeat. Logging back in and it's hung.

 

[NickBurns]

... I have a feeling the unattended restarts are due to the /tmp/HTTPD_DEBUG flag being set.

 

[NickBurns]

nt_center

admin@RT-AC5300:/tmp# netstat -anp|grep -i nt_cent
tcp 518 0 <my_ISP_ip>:43382 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 518 0 <my_ISP_ip>:43384 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 348 0 <my_ISP_ip>:43394 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 518 0 <my_ISP_ip>:36998 75.2.7.103:443 CLOSE_WAIT 23602/nt_center
tcp 55 0 <my_ISP_ip>:37001 75.2.7.103:443 CLOSE_WAIT 23602/nt_center
tcp 83 0 <my_ISP_ip>:43393 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 1 0 <my_ISP_ip>:43379 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 518 0 <my_ISP_ip>:43385 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 5286 0 <my_ISP_ip>:37005 75.2.7.103:443 CLOSE_WAIT 23602/nt_center
tcp 518 0 <my_ISP_ip>:36993 75.2.7.103:443 CLOSE_WAIT 23602/nt_center
tcp 5286 0 <my_ISP_ip>:43396 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 55 0 <my_ISP_ip>:43390 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 518 0 <my_ISP_ip>:36996 75.2.7.103:443 CLOSE_WAIT 23602/nt_center
tcp 83 0 <my_ISP_ip>:43392 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 518 0 <my_ISP_ip>:43380 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 1 0 <my_ISP_ip>:36999 75.2.7.103:443 CLOSE_WAIT 23602/nt_center
tcp 518 0 <my_ISP_ip>:36997 75.2.7.103:443 CLOSE_WAIT 23602/nt_center
tcp 518 0 <my_ISP_ip>:43381 99.83.159.30:443 CLOSE_WAIT 23602/nt_center
tcp 1 0 <my_ISP_ip>:37007 75.2.7.103:443 CLOSE_WAIT 23602/nt_center

I read in another from from @RMerlin that this is the process that runs during the aforementioned 'database' conversion. It looks like it's still running... and spawning several children... also trying to talk to a handful of AWS endpoints over 443.

admin@RT-AC5300:/tmp# uptime
12:11:36 up 14 days, 21:20, load average: 3.77, 4.42, 5.11

 

[L&LD]

As much as you don't want to hear it, your router is not in a good/known state.

Fully Reset Router and Network

Best Practice Update/Setup Router/AiMesh Node(s) 2021

 

[ComputerSteve]

SO I have this exact issue.. Its only happening when I change settings of the WIFI settings.

 

[NickBurns]

As much as you don't want to hear it, your router is not in a good/known state.

Fully Reset Router and Network

Best Practice Update/Setup Router/AiMesh Node(s) 2021

I did the full factory reset 14 days ago as much as it was a pain. It's about the fourth time in just over a year. It's taking less time to recover/rebuild however since I'm getting much better at wash, rinse, repeat.

What bothers me is that following a reboot the router will operate fine for hours, days... and then enter this condition. So granted, it's not in a good/known state at that point, but for the previous xx hours of uptime it's just fine.

I'm interested in trying to figure out where and when the wheels fall off.

I'm currently modeling this nt_center application. I've since rebooted the router, things are back to working and there are three spawns of this nt_center task; each of them talking to some AWS service (now I'm thinking IFTTT/ALEXA link) and none of the open sockets are queueing... yet.

I'll read through the best practice guide. The Fully Reset/WPS is old hat now.

 

[NickBurns]

Working on this some more tonight. I have a feeling this is ALEXA/IFTTT.

I've never enabled IFTTT (yet GUI says I've successfully registered)... I've now disabled ASUSWRT on the alexa.amazon.com side and rebooted my router. I watched it try to reach out to the server; spawn a few nt_center processes and then stops.

I also touched /tmp/IFTTT_ALEXA to enable debug logging. I sure hope NT_CENTER (Notification Center) isn't spilling my client MAC addresses up to the cloud as I suspect it is or has been all this time.

[Notification_Center][event_handler993)][2021-05-28 21:48:36]NOTIFY_EVENT[GENERAL_ETH_DEV_OFFLINE (70004)] Action:[20] MSG:[{"from":"cfg_server","client_mac":"<one_of_many_macs>"}] Num:[54]

 

[L&LD]

Everything you're writing is saying to me that the router is not as 'fully reset' as you think.

I would nuke it as per the first link I've sent above. The details, matter.

 

[NickBurns]

So I think I've reproduced it.

I re-enabled / re-linked Alexa. In order to do so I had to change my WAN HTTPS port from x back to 443. I then asked Alexa ASUS ROUTER to simply enable a guest Wifi. The Alexa app said it created network Y with a code for 3 hours - Amazingly it worked. During this time there are TCP on 443 to/from my router to AWS services that come and go. The GUI showed the guest network. All good.

I then changed my WAN HTTPS port back from 443 to x. I then asked Alexa to create a guest Wifi. The app says it created one for 3 hours. I watched live as nt_center spawned mulitple spawns all trying to talk TCP and the queuing began. Shortly there after the GUI became inoperable.


NOTE: There is a GUI bug. On the System page I could not change the HTTPS WAN port setting because the username/password kept balking "name already exists" stating my account already existed. Duh - of course it already exists and I am not changing the password. To get around it I had to create a new user w/ password. Save. Log in with new user. Return to this page and set it back.


Now after rebooting my router and clearing this condition, only SSH to the router to start looping netstat output - with my WAN HTTPS port back to x (not 443) I simply asked Alexa to create a guest Wifi. Boom. nt_center spawned and queued.

admin@RT-AC5300:/tmp# netstat -anpt|grep -i nt_center
tcp 518 0 <my_ISP_ip>:48618 75.2.7.103:443 CLOSE_WAIT 1312/nt_center
tcp 55 0 <my_ISP_ip>:48621 75.2.7.103:443 CLOSE_WAIT 1312/nt_center
tcp 518 0 <my_ISP_ip>:48616 75.2.7.103:443 CLOSE_WAIT 1312/nt_center
tcp 55 0 <my_ISP_ip>:48620 75.2.7.103:443 CLOSE_WAIT 1312/nt_center
tcp 518 0 <my_ISP_ip>:45609 99.83.159.30:443 CLOSE_WAIT 1312/nt_center
tcp 518 0 <my_ISP_ip>:48617 75.2.7.103:443 CLOSE_WAIT 1312/nt_center
tcp 518 0 <my_ISP_ip>:48611 75.2.7.103:443 CLOSE_WAIT 1312/nt_center
tcp 1 0 <my_ISP_ip>:45613 99.83.159.30:443 CLOSE_WAIT 1312/nt_center
tcp 518 0 <my_ISP_ip>:48614 75.2.7.103:443 CLOSE_WAIT 1312/nt_center