In some dire need of assistance (AX86U)

[MatynK]

I hadn't seen any mention of this ....
Has anyone thought to check the PiHole DNS logs to see whats being blocked to the Google-Home-Mini's ?
They are expected to "phone home" once in a while for updates/syncing.
When they can't call home they think the internet is down and start reconnecting.
Think of it as Googles way to ensure they get full access to the internet.

I have the same issue with other devices, Nest, Alexa, Dyson fans, to name but a few.
Exactly the same issue and results.

 

[KsWoodsMan]

I have the same issue with other devices, Nest, Alexa, Dyson fans, to name but a few.
Exactly the same issue and results.

What did your DNS logs show for these clients ?
Any blocked domains ? especialy blocked google gomains ? or amazon-aws ?

 

[MatynK]

What did your DNS logs show for these clients ?
Any blocked domains ? especialy blocked google gomains ? or amazon-aws ?

Nothing in DNS at all to show any issues.
It was all about these devices not managing to renew their IPs from DHCP

 

[KsWoodsMan]

Nothing in DNS at all to show any issues.
It was all about these devices not managing to renew their IPs from DHCP

Was the checking done on the router ?
Or the dns logs on the pihole ?
To me this would seem more like a finicky set of client devices, rather than a flakey connection to them.
One person mentioned his devices in question will go 2 -5 days without a problem before they all seem to go down about the same time. Coincidence ?

 

[MatynK]

Was the checking done on the router ?
Or the dns logs on the pihole ?
To me this would seem more like a finicky set of client devices, rather than a flakey connection to them.
One person mentioned his devices in question will go 2 -5 days without a problem before they all seem to go down about the same time. Coincidence ?

My DNS/DHCP is on a Windows PC, DNS logs are the windows ones and my wired devices dont have the issue.
Maybe the 2~5 days could be something to do with the DHCP lease time.

 

[KsWoodsMan]

My DNS/DHCP is on a Windows PC, DNS logs are the windows ones and my wired devices dont have the issue.
Maybe the 2~5 days could be something to do with the DHCP lease time.

Or the time it takes to fill up their filesystem with conversations it's been eves dropping on.

One of the comonalities does seem to be that DNS is not being served from the router itself
and DNS filtering/protection is taking place on the local network.
I may be going in the wrong direction but thought to add it to the possibilities of things to address.

 

[MatynK]

Or the time it takes to fill up their filesystem with conversations it's been eves dropping on.

One of the comonalities does seem to be that DNS is not being served from the router itself
and DNS filtering/protection is taking place on the local network.
I may be going in the wrong direction but thought to add it to the possibilities of things to address.

Not quite sure what you mean by filling up file systems and eves dropping on what ?

Well, I can see nothing DNS here, it all seems to be DHCP and the fact that devices cannot get IPs allocated to them.

I can even see this on a phone on my guest network, it will attempt to attached to the network, but never gets an IP from the DHCP server and there is nothin in the logs showing an attempt has even been made to get an IP address.

 

[KsWoodsMan]

I missunderstood you then that

you were having the same problem with your google devices and the ones listed.

evesdropping ... if it has a microphone or a mic and camera it has the capacity to record and store/forward anything within it's range.
I might be a bit paranoid over this.
But then again I might not be paranoid enough.

I do hope someone is able to figure it.
I'm out.

 

[shadowfiber]

My devices were running over 7 days with no issues - then all of a sudden all lost connectivity around the same time. Router doesn't report any issue other than the DHCP offer's again. It's been stable for 16 days on Merlin Beta 4.
Sigh

 

[shadowfiber]

Beta 5 - even worse - disconnects every few hours again.

 

[L&LD]

No issues for me or my customers with Beta 5.

You must have a very customized setup. What network equipment are you running today?

 

[shadowfiber]

No issues for me or my customers with Beta 5.

You must have a very customized setup. What network equipment are you running today?

I don't feel like it's very customized...

Wired house - to a patch panel - goes into an unmanaged D-Link 24port switch ... Router plugged into that... along with a NAS that has a pihole docker container for DNS. Removed the pihole completely for testing - no change... So to me, its a bunch of devices and nothing special. Open to suggestions and thoughts though!!

The Router is Merlin firmware with no addons or plugins or customized scripts.

 

[L&LD]

Pihole docker sounds pretty 'advanced' to me.

The router's LAN port is plugged into the switch, or the WAN?

Have you tried powering down the entire network? All infrastructure and wired clients too, for at least 10 - 20 minutes?

I would disconnect all power cords, adaptors, Ethernet cables, USB cables and devices from all equipment after the power was off. And plug, connect and attach everything before you powered up the network again.

How much do you trust that switch? Maybe test by powering it off and on first (as quick test).

Maybe test with a couple of new cables too from the WAN and the LAN. While you're at it, try different LAN ports too (if you're only using one to the switch).

Try different WAN port too, by making a LAN port for WAN.

 

[shadowfiber]

Pihole docker sounds pretty 'advanced' to me.

The router's LAN port is plugged into the switch, or the WAN?

Have you tried powering down the entire network? All infrastructure and wired clients too, for at least 10 - 20 minutes?

I would disconnect all power cords, adaptors, Ethernet cables, USB cables and devices from all equipment after the power was off. And plug, connect and attach everything before you powered up the network again.

How much do you trust that switch? Maybe test by powering it off and on first (as quick test).

Maybe test with a couple of new cables too from the WAN and the LAN. While you're at it, try different LAN ports too (if you're only using one to the switch).

Try different WAN port too, by making a LAN port for WAN.

Great suggestions and thoughts; I'd like to pass one piece back to you though that may have been missed from the beginning. (before I do them)

If I put the AC66U (non b1) back into place. Everything works fine.

Makes me think that the switch, the pihole, the everything else is fine; and there's something with the router. Let me know if you think I may be missing something there.

 

[L&LD]

I don't think you answered my question(s)?

Nothing is fine until proven otherwise. Working 'okay' with other equipment doesn't automatically get a 'pass' with new hardware in the loop.

Can you give us a sketch/drawing of the way everything is wired? It will answer the questions I asked and be easier to visualize any other issues too.

 

[sdts]

Wireless drivers were updated in 386.

3.0.0.4.384.9318
Broadcom BCA: 17.10 RC121.11
wl0: Oct 21 2020 14:23:41 version 17.10.121.11 (r783116)

9.0.0.4.386.41157
Broadcom BCA: 17.10 RC121.37
wl0: Nov 30 2020 11:15:42 version 17.10.121.37 (r789389)

This is the essence of it.

384 has no issues with Google home minis or Sonos. 386 is broken for them.

No setting is going to fix that and as Merlin mentioned, the wifi code is closed source, so it's going to be the same regardless of whether stock firmware or merlinwrt.

Doesnt sound like Asus have had any luck in fixing it in a couple of months of people reporting it.

Plausible it could get fixed in future firmware, but no guarantees. So I guess the options are to either run old 384 firmware, get rid of the devices it can't handle, or use a different router.

 

[ColinTaylor]

@ColinTaylor Any suggestions on why the radio crash may be occurring, or is that something more for Merlin/ASUS. It didn't seem to affect the googles - at least this time.

Spoiler: crash

Dec 8 22:11:45 kernel: CONSOLE: 152722.039 wl1: PSM microcode watchdog fired at 95822 (seconds). Resetting.
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 wl1: reason = psm watchdog at 95822 seconds. corerev 129.1 ucode revision 1570.159 features 0x9106
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 psmdebug 0x01ff000d phydebug 0x1 macctl 0x4160403 maccmd 0x4
Dec 8 22:11:45 kernel: CONSOLE: psm_brc 0xa000 psm_brc_1 0x000c M_UCODE_DBGST 0x2
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 PSMR1: psmdebug 0x0000009b macctl 0x4060402 maccmd 0x0 M_UCODE_DBGST 0x2
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 brwk_0 0x0501 brwk_1 0x0010 brwk_2 0x82a0 brwk_3 0x0ff0 brwk_4 0x0006
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 ifsstat 0x9da0 ifsstat1 0xe0 txectl 0x481d txestat 0x445
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 rxestat1 0x1 rxestat2 0x500 rcv_frmcnt 0x0 rxe_rxcnt 0x68
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 wepctl 0x0
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 aqmf_ready0 0x0 aqmf_ready1 0x0 aqmf_ready2 0x0 aqmf_ready3 0x0
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 aqmf_ready4 0x2 wepstat 0x0 wep_ivloc 0x1a wep_psdulen 0x157
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 rxe_errval 0x0 rxe_errmask 0x20f rxe_status3 0x108 txe_status2 0x445
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 dbg_bmc_status 0x0 psm_chk0_err 0x0 psm_chk1_err 0x0
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 PC :
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R0: 0x24 0x24 0x24 0x24
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:45 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 R1: 0x9b 0x9b 0x9b 0x9b
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 phydebug :
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1 0x1 0x1 0x1
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 pktproc 0x101 pktproc 0x101 pktproc 0x101 pktproc 0x101
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 TxFifoStatus0 0x1c3 TxFifoStatus1 0x1c3
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 gpiosel 0x0 gpiolo 0x0 gpiohi 0x0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 gpiosel 0x1 gpiolo 0x0 gpiohi 0x0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 psm stack_status : 0x8000
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 psm stack_entries:
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x0024
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1d99
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x0629
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x1f21
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x086c
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x0848
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x0848
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0x086a
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 0) usr_count = 0, schedid = 0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 1) usr_count = 0, schedid = 0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 2) usr_count = 0, schedid = 0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 3) usr_count = 0, schedid = 0
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 wl1: wlc_check_assert_type HAMMERING: reinit_reason 2
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 wl1: fatal error, reinitializing
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 wl1: fatal error, reinitializing, total count of reinit's[6]
Dec 8 22:11:46 kernel: CONSOLE: 152722.039 wl1: 802.11 reinit reason[2], count[6]
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 wl1: wlc_bmac_suspend_mac_and_wait: waited 83000 uS and MI_MACSSPNDD is still not on.
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 wl1: reason = mac suspend failure at 95822 seconds. corerev 129.1 ucode revision 1570.159 features 0x9106
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 psmdebug 0x01ff000d phydebug 0xb macctl 0x4160402 maccmd 0x4
Dec 8 22:11:46 kernel: CONSOLE: psm_brc 0xa000 psm_brc_1 0x000c M_UCODE_DBGST 0x2
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 PSMR1: psmdebug 0x0000009b macctl 0x4060402 maccmd 0x0 M_UCODE_DBGST 0x2
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 brwk_0 0x0583 brwk_1 0x0095 brwk_2 0xaaa0 brwk_3 0x0ef0 brwk_4 0x0006
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 wlc_dump_ucode_fatal: log_done 0x2
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 HWA4a TxSTAT: hwa_txstat_reclaim: reinit<1> stall<0>
Dec 8 22:11:46 kernel: CONSOLE: 152722.123 HWA3b TxFIFO: reclaim fifo 1 pkts 1
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 HWA1b RxFILL: mac_counter_status sat<0> need_post<0> aval<256>
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 HWA1b RxFILL: reclaim core<0> 256 rxbuffers RD<218> WR<474>
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 HWA1a RxPOST: wi_cnt<126> in hwa internal memory<RD:98 WR:96>
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 wl1: wlc_bmac_wowlucode_start mctrl=0x4020402 0x4000400
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 wl1: wlc_bmac_wowlucode_start (mctrl | MCTL_PSM_JMP_0)=0x4020406 0x4020406
Dec 8 22:11:46 kernel: CONSOLE: 152722.124 wl1: wlc_bmac_wowlucode_start mctrl=0x4020402 0x4020402
Dec 8 22:11:46 kernel: CONSOLE: 152722.126 wl1: CORE INIT : mode 4 pktclassify 1 rxsplit 1 hdr conversion 1 DMA_CT Enabled

@shadowfiber Did you ever work out what was causing these crashes?

 

[SenorClean]

Glad to hear I'm not alone, but I'm pulling my hair out here too.

Have been running an RT-AC68U for about 6 years, just upgraded to an AX86U this week and of course flashed it with Merlin firmware before doing anything else.
It's generally running well, but crazy unstable with the nest minis.

I've tried every trick I could find mentioned in here (Shift to 2.4ghz, remove AImesh, manually assigned IP with unique hostname) - nothing seems to solve it. My Plan B is to plug in an old router and run a dedicated AP for them - hardly elegant, but I'm running out of options.

EDIT: I think I've solved it - I had DNS over TLS enabled with the 'Strict' profile - shifting to Opportunistic (or disabling DOT) seems to fix it. Maybe there's some sort of phoning home that was being blocked?

 

[shadowfiber]

@shadowfiber Did you ever work out what was causing these crashes?

Hey there,

I wouldn't say it's "FIXED" per se, but the frequency of the issues has drastically reduced. Generally all the mini's will drop off the network once a month, need a reset and be good to go for another month. DNS over TLS hasn't resolved it permanently for me as it has for @SenorClean unfortunately. I have shifted everything to Quad9 with DNSSEC as my primary outbound through the pihole at this point. The amount of "phoning home" has been ridiculous, now i have a good handle on how frequently the minis speak to Google's servers, and it's quite ridiculous. Unfortunately, they all dropped off again yesterday. /rollseyes

Regardless, once a month is better than several times a day, the only thing I can contribute that for at this point is firmware changes.

 

[ColinTaylor]

Thanks for the update @shadowfiber. I recently got an RT-AX86U also and am seeing the same crashes. In my case one of the trigger conditions is if radar is detected and the router changes the 5GHz WiFi channel. That always results in an endless loop of crashing.