"Maximum number of concurrent DNS queries" with strange lb._dns-sd._udp. ... .in-addr.arpa reverse DNS queries (has Pi running Adguard Home + Unbound)

[loveleeyoungae]

Hi,

My internet line has been stuttered and intermittent for some weeks. Last night, I took some time to check.
1. First, my setup:
192.168.7.1: an AC68U with Merlin firmware
192.168.7.110: a Pi running Adguard Home + Unbound only acts as DNS server
Both WAN and LAN DNS servers settings on the router are set to point only to the Pi's IP address
Adguard Home settings:

Spoiler

2.
a. In the router, the syslog was flooded with the error: "dnsmasq[254]: Maximum number of concurrent DNS queries reached (max: 150)"
b. googled around for the error, saw suggestions to run the tcpdump command, here is a part of the result:

16:21:34.179847 vlan1 In  IP 192.168.7.110.53 > 192.168.7.1.16005: 62299 ServFail 0/0/0 (60)
16:21:34.179847 br0   In  IP 192.168.7.110.53 > 192.168.7.1.16005: 62299 ServFail 0/0/0 (60)
16:21:34.181195 br0   Out IP 192.168.7.1.16005 > 192.168.7.110.53: 51220+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.181237 vlan1 Out IP 192.168.7.1.16005 > 192.168.7.110.53: 51220+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.181513 vlan1 In  IP 192.168.7.110.53 > 192.168.7.1.40239: 44526 ServFail 0/0/0 (60)
16:21:34.181513 br0   In  IP 192.168.7.110.53 > 192.168.7.1.40239: 44526 ServFail 0/0/0 (60)
16:21:34.182162 br0   Out IP 192.168.7.1.29256 > 192.168.7.110.53: 1291+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.182199 vlan1 Out IP 192.168.7.1.29256 > 192.168.7.110.53: 1291+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.182716 br0   Out IP 192.168.7.1.16005 > 192.168.7.110.53: 14266+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.182755 vlan1 Out IP 192.168.7.1.16005 > 192.168.7.110.53: 14266+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.182934 br0   Out IP 192.168.7.1.40239 > 192.168.7.110.53: 44526+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.182961 vlan1 Out IP 192.168.7.1.40239 > 192.168.7.110.53: 44526+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.183038 vlan1 In  IP 192.168.7.110.54932 > 192.168.7.1.53: 17048+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.183038 br0   In  IP 192.168.7.110.54932 > 192.168.7.1.53: 17048+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.183733 br0   Out IP 192.168.7.1.34857 > 192.168.7.110.53: 28070+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.183770 vlan1 Out IP 192.168.7.1.34857 > 192.168.7.110.53: 28070+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.184159 br0   Out IP 192.168.7.1.16005 > 192.168.7.110.53: 26436+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.184193 vlan1 Out IP 192.168.7.1.16005 > 192.168.7.110.53: 26436+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.184525 vlan1 In  IP 192.168.7.110.53 > 192.168.7.1.47843: 3898 ServFail 0/0/0 (60)
16:21:34.184525 br0   In  IP 192.168.7.110.53 > 192.168.7.1.47843: 3898 ServFail 0/0/0 (60)
16:21:34.184719 vlan1 In  IP 192.168.7.110.53 > 192.168.7.1.51517: 25900 ServFail 0/0/0 (60)
16:21:34.184719 br0   In  IP 192.168.7.110.53 > 192.168.7.1.51517: 25900 ServFail 0/0/0 (60)
16:21:34.184826 vlan1 In  IP 192.168.7.110.53 > 192.168.7.1.4832: 47891 ServFail 0/0/0 (60)
16:21:34.184826 br0   In  IP 192.168.7.110.53 > 192.168.7.1.4832: 47891 ServFail 0/0/0 (60)
16:21:34.185598 br0   Out IP 192.168.7.1.36266 > 192.168.7.110.53: 10075+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.185637 vlan1 Out IP 192.168.7.1.36266 > 192.168.7.110.53: 10075+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.185783 vlan1 In  IP 192.168.7.110.57634 > 192.168.7.1.53: 14266+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.185783 br0   In  IP 192.168.7.110.57634 > 192.168.7.1.53: 14266+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.185791 vlan1 In  IP 192.168.7.110.34812 > 192.168.7.1.53: 22028+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.185791 br0   In  IP 192.168.7.110.34812 > 192.168.7.1.53: 22028+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.186084 vlan1 In  IP 192.168.7.110.44716 > 192.168.7.1.53: 1291+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.186084 br0   In  IP 192.168.7.110.44716 > 192.168.7.1.53: 1291+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.186420 br0   Out IP 192.168.7.1.16005 > 192.168.7.110.53: 30958+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.186681 br0   Out IP 192.168.7.1.47843 > 192.168.7.110.53: 3898+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.187035 br0   Out IP 192.168.7.1.51517 > 192.168.7.110.53: 25900+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)
16:21:34.187068 vlan1 Out IP 192.168.7.1.51517 > 192.168.7.110.53: 25900+ PTR? lb._dns-sd._udp.157.40.235.10.in-addr.arpa. (60)

c. googled around for the "lb._dns-sd..." stuff, and realized that there might be some differences for my situation:
- in other reports:
+ the reversed IP in the "lb._dns-sd._udp.*.*.*.*.in-addr.arpa" query are usually "0.0.168.192" and issues are usually caused by Apple devices/ Bonjour services, DNS Discovery etc,
+ IIRC, a comment suggested that if the ISP internet connection itself is bad, unbound might have unstable connection to root servers and might cause the issue (unfortunately I didn't know if I read correctly, and couldn't remember which link has that comment among 50-ish links I googled)
- in my system:
+ the reversed IP on my system is "157.40.23.10" and my system doesn't have any "10.x.x.x" subnet
+ I have some iPhones, iPads, and icloud/ iTunes installed on some computers, I tried disconnecting them all from network but the dnsmasq errors were still there


d. Guessed I might need to see more detailed logs:
- AGH: didn't see any strange queries
- unbound: tried to enable log using this guide but unsuccessful (apparmor_parser step). Looking at the Pi's syslog only has several lines related to unbound.

Haven't tried setting dnsmasq max connection to 1024 or disabling "Use private reverse DNS resolvers" setting in AGH, but even though those help, obviously the uneasy feeling is still there when we know that something is flooding the DNS queries. So if anyone has any idea on what the issue might be related to, how to fix it etc., please help me.

Thanks in advance.

 

[loveleeyoungae]

More info that might be helpful for diagnosing the issue:
1.
a) People who reported about these DNS-DS queries usually told that they saw the queries in their Pi-hole or AGH's query log. But my AGH's Query Log never shows such queries.
b) Sometimes during my test, changing configurations etc., those queries become some kind of SOA queries

=> Looks like the queries are actually between the Pi and the router, not some kind of queries forwarding from Pi (AGH)?


2. This piece of information may not be related to my issue, but it's such an interesting read that I'll still post here for reference. (Original comment on Reddit)

I know this is an old thread, but maybe others will stumble on this thread when searching for the same thing. I’ve been fighting this issue for about 2 months now. I’ve seen a lot of people saying it’s mdns service discovery for the lan network but I don’t believe that’s true for a couple reasons. Mainly because any bonjour or other mDNS request shouldn’t ever get resolved by the dns server. They operate on the reserved .local suffix. And use multicast which allows devices to resolve their own hostnames for other devices without any intervention from the dns server or router. Thats the whole point of mDNS. Also, I highly suspect the “lb” at the beginning of these requests stands for “load balancer” which seems to imply the iPhone is searching for some kind of network edge load balancer. I doubt Apple expects many iPhone users to have a load balancer available on their home networks.

Im just going to list the other things I’ve noticed and maybe someone else will see this and possibly piece together some more info on how to stop this nonsense.

A lot of the PTRs are reserved in a reserved address space. (lb._dns-sd._udp.2.0.0.192.in-addr.arpa) 192.0.0.0/24 Is used for ipv4 over ipv6 routing. I believe that’s why these requests show up in the request log because these addresses are actually external addresses, so the dns server tries to resolve them with the upstream resolvers (which fails with nxdomain if you unblock them)

I’m using PFsense on my router which has an option to specify the dns search domain for dhcp devices. Im also using unbound as a secondary dns resolver for local PTR, SOA, and NS requests for ARPA domains. (To mimic mDNS dynamic DNS hostnames for dhcp non-mDNS devices while staying away from the .local suffix) A clue from this is that when I assign home.lan as the search domain on my iPhone, it starts sending the same type of reverse dns requests except with the home.lan suffix. Which leads me to believe the iPhone isn’t searching for a specific service or anything, but instead it is blindly searching for any available dns servers and is trying to reverse resolve a failing domain (that is most likely blocked as a tracker) My best guess is that this if for iCloud private relay. Many of the private relay domains are blocked in common block lists, so the iPhone can’t reach the servers. It probably assumes that either the private relay servers are blocked (they are) or that the server domains changed. So it tries to find any other (non blocking) dns resolver and sends reverse dns requests to get an updated domain name. (Which is futile in this case) I did some minimal testing a couple days ago and it seems like private relay almost exclusively uses ipv6 so that would explain the 192.0.0.0/24 ipv4 over ipv6 reverse dns requests. And also explains why it just blindly blasts out reverse dns over any available route. That also implies that if it DOES succeed at finding a viable route (maybe a dns leak to ISP dns server) and you have IPv6 properly setup with globally unique public facing IPv6 address. (I do) It could theoretically connect directly to the private relay servers over IPv6 and totally bypass your dns server (and potentially even your router firewall if you have default allow outbound IPv6) so you wouldn’t see any requests logged. (Ask me how I know all of this) The result of this is your iPhone will have an IPv6 only connection to iCloud private relay and will send the vast majority of any IPv6 traffic over over it which, subsequently, completely bypasses your dns server, filtering, and firewalls. You can disable private relay in the WiFi setting, but then you need to do this for every iPhone that connects to your network, and it gives an insecure network warning. AND still continues to send the same requests for whatever reason.

The only thing this doesn’t explain (or really I just have no clue why it would even do this is the first place) is the private subnet reverse requests (lb._dns-sd._udp.0.0.168.192.in-addr.arpa) I can’t think of any way a reverse DNS request for 192.168.0.0 could ever accomplish anything. So that’s got me a little stumped.

The best solution I’ve come up with is explicitly blocking iCloud private relay (mask-h2.icloud.com and mask.icloud.com) in the dns filter coupled with being absolutely sure that no device on my network is configured to use a dns server other than my own. Which is really hard to do (or borderline impossible) with some ISP (AT&T fiber) provided hardware.

Just to show how ridiculously pushy private relay is, I have AT&T fiber and AT&T will not allow you to change the default dns server in their router, and will not allow you to replace the router with a 3rd party router. (Knowingly) I even tried totally disabling dhcp and dhcp6 in their router and ran my own server, but the router still sent out RAs for IPv6 which meant clients could use SLAAC and NDP to self assign an IPv6 address. So they would end up getting an IPv4 and IPv6 address and dns addresses from my dhcp server but would also self assign a second IPv6 address and IPv6 dns address back to the AT&T router no matter what I did.

Thankfully some people WAY smarter than me have reversed engineered AT&Ts 802.1x authentication and figured out how to reprogram a 3rd party ONT to mimic the AT&T router and authenticate with their OLTs. (Look for the 8331 discord server if interested)

 

[loveleeyoungae]

Days after discovering the issue, I guess I've found the solution:

1. It turns out that AGH's query log is not a live log. It is not written immediately, but written to ram/memory first, and would be flushed to the log file later
-> When the issue happened and I checked the AGH web log right away, it didn't show anything related to those "lb._dns-sd._udp.*.*.*.*.in-addr.arpa" queries, which in turn made me have false assumption in the previous post

2. So, I managed to enable logging and more verbose/ more detailed logging for AGH verbose live log & unbound log (on Pi) and dnsmasq log (on Router)
It turned out that I actually had DNS loop caused by an iPhone, which is actually related to the infamous Apple mDNS discovery service issue!
(The iPhone's mDNS service is somehow triggered and sends those queries to Pi-AGH, which forwards them to the Router. Since the Router doesn’t know the answers, it continues to forward the queries to its upstream DNS server, which I have set as Pi-AGH )

3. In all topics and threads I read, people who know and discuss about "lb._dns-te ._udp.*.*.*.*.in-addr.arpa" queries don't mention about Apple Internet Private Relay, and vice versa. But thanks to the one comment I quoted in my previous post, both issues are connected.

4. A quick google search shows that the domains for Apple Internet Private Relay are blocked by default in Pi-hole, ControlD DNS service, and has its own block switch on Blocked Services page of AGH.

5. I've tried some manual steps, and AGH has been responding faster than ever, so things look like solved. I'll wait for some days before marking this topic as solved. And I'll list the steps I've done:
a. In iCloud, disable Internet Private Relay
b. In Find My, disable Find My Network. Or if not necessary, also disable Find My iPhone (iPad)
c. In Settings> Mail, disable Privacy Protection
d. In Wi-Fi, disable Limit IP Address Tracking, Private Wi-Fi Address, and may need to set Configure DNS to Manual
e. Create NXDOMAIN filters for these domains. Main "culprit" domains are the first two. You can comment out the others if there's some issue with your Apple devices (Examples below are for AGH)

||mask.icloud.com^$important,dnsrewrite=NXDOMAIN;;
||mask-h2.icloud.com^$important,dnsrewrite=NXDOMAIN;;
||mask-api.icloud.com^$important,dnsrewrite=NXDOMAIN;;
||metrics.icloud.com^$important,dnsrewrite=NXDOMAIN;;
||mask-t.apple-dns.net^$important,dnsrewrite=NXDOMAIN;;
||mask.apple-dns.net^$important,dnsrewrite=NXDOMAIN;;
||mask-api.fe.apple-dns.net^$important,dnsrewrite=NXDOMAIN;;