Nest Protect cannot connect with DoT enabled - RT-AX88U 384.18

[wraithdu]

After updating to 384.18, all 8 of my Nest Protect devices went offline, and I was unable to add any new Protect devices onto my account. I have one 2nd Gen and the rest 1st Gen. I went through all troubleshooting, and finally after totally disabling DoT, they were able to connect and I could add new devices again.

What I tried:

- DoT with Quad 9 (ipv4 and ipv6 servers)
- DoT with Cloudflare (ipv4 and ipv6 servers)
- Disable "Prevent client auto DoH"
- Profile as both Strict and Opportunistic

None of the settings made a difference, only disabling DoT restored access.

I have several other IoT devices which have been and continue to work fine, including a Nest camera. Only the Protect devices are affected.

I've had these devices for over a year now with no issues, so this is definitely related to the .18 fw version. I'd be happy to provide any info or troubleshooting help.

I almost went the next step to install tcpdump on the router to see what's happening with the DNS queries, but I'm exhausted dealing with this and Nest's twitter support over the last 24 hours. At their request, I confirmed I could add devices to my account via a mobile phone hotspot, so that was the point at which they basically told me to go F*** myself.

Let me know how I can help.

 

[dave14305]

What DNS servers are you using now without DoT?

 

[fields987]

I’ve got 3 nest protect gen 2’s that are working on my ax88u with 384.19 alpha1 installed. I’m using quad 9 for router dns and nextdns for DoT with dns filtering disabled on the lan side.

 

[wraithdu]

I'm using Cloudflare without DoT, both ipv4 and ipv6. Should mention I'm on Xfinity cable.

 

[wraithdu]

@fields987 , I did not have rebind or DNSSEC options enabled, and I set the Quad9/Cloudflare ipv6 servers in the list as well.

 

[CriticJay]

What were your DNSFilter settings?

 

[wraithdu]

I do not use DNSFilter.

 

[dave14305]

And this used to work with Cloudflare DoT 1.1.1.1 before 384.18? Was 384.17 your previous firmware?

 

[Gar]

I have not been able to use DoT since moving to a new fiber optic ISP last year. Nothing I try will allow it. Never a problem with my old cable.

 

[wraithdu]

@dave14305 yes. It has worked with DoT and Cloudflare as well as Quad9. I've kept up with every stable fw release, so I upgraded from .17. I had considered downgrading to test, but .18 has those breakages, so I can't without a full reset.

 

[SheikhSheikha]

You mention that this happens after an update of the firmware. I have encountered (other) problems with Nest products in the past too that all went away by re-configuring the Nest devices from scratch. Somehow they store some unique "goodies" when configured (like Apple devices) that make this go away (but that cause a problem when you upgrade your firmware).

 

[wraithdu]

I tried that actually. Removed from my account, full factory reset. It fails when trying to add back. Best guess is the device can't communicate at the last stage when associating with the account (P009 0.80 error).

I guessed at the root problem really. Tried it with my laptop as a hotspot and ran Wireshark. I saw a DNS resolution failure to a nest domain from the device, and it resulted in the same error.

Started playing with the router DNS config, and it worked after I disabled DoT. I was able to add back the device and all my others came back online.

 

[john9527]

@wraithdu I had to do a patch to fix DoT name resolution for some sites on my fork. I don't know if you are seeing the same problem, but I did an AX88U 384.18 build that adds that one fix. If you want to give it a try, drop me a PM.

Caveats: I don't have any AX routers so can't test it myself.

 

[dave14305]

@wraithdu I had to do a patch to fix DoT name resolution for some sites on my fork. I don't know if you are seeing the same problem, but I did an AX88U 384.18 build that adds that one fix. If you want to give it a try, drop me a PM.

Caveats: I don't have any AX routers so can't test it myself.

Is this yet-to-be-published on the fork?

 

[john9527]

Is this yet-to-be-published on the fork?

Already included since V42

 

[CriticJay]

@wraithdu I had to do a patch to fix DoT name resolution for some sites on my fork. I don't know if you are seeing the same problem, but I did an AX88U 384.18 build that adds that one fix. If you want to give it a try, drop me a PM.

Caveats: I don't have any AX routers so can't test it myself.

Can you elaborate on the mechanics behind the patch you did?

 

[dave14305]

Can you elaborate on the mechanics behind the patch you did?

I'm speculating that it's this max UDP payload patch:

https://github.com/john9527/asuswrt...d941001#diff-d7357bd56da084ca69b8f9c27aa51157

 

[john9527]

I'm speculating that it's this max UDP payload patch:

Good guess
One of those patches that was slipped in a few days before they made their release. It broke name resolution for 'large' DNS responses (multiple IPs returned). In my case, it broke name resolution for my VPN provider servers. dnsmasq would never receive a response to the request forwarded to DoT.

 

[CriticJay]

Cool, interesting stuff. If this patch helps with the initial issue maybe there's value in adding it for all router models, not just the AX88U.

 

[wraithdu]

Got some 4th stuff going on today, but I'll reach out over PM to talk about testing your build, thanks! The problem description looks promising. Was this a recent breakage? My network was fine on .17.