DNS-over-TLS very slow (up to 10 seconds/query) via stubby, fast if queried directly on RT-AC86U

[Murgi]

Hey,

I recently ran into some trouble with domains not resolving when using DoT on my RT-AC86U. I figured out that this is due to a timeout because the the resolution takes to long. I verified this by trying to query stubby directly on the router:

# time nslookup snbforums.com 127.0.1.1
Server:    127.0.1.1
Address 1: 127.0.1.1

Name:      snbforums.com
Address 1: 2606:4700:20::ac43:4551
Address 2: 2606:4700:20::681a:942
Address 3: 2606:4700:20::681a:842
Address 4: 104.26.8.66
Address 5: 104.26.9.66
Address 6: 172.67.69.81
real    0m 3.94s
user    0m 0.00s
sys     0m 0.00s

As can be seen, the DoT query takes almost 4 seconds (I have seen values up to 10 seconds). This is with Cloudflare DNS, but I also tried quad9 and google DNS, none of which work any better.

This slowness does not seem to be due to my connection being slow when using DoT. I tested this using a script from https://github.com/dcid/dns-over-tls-php-client on my router which gives me resolve times under 0.1 seconds with Cloudflare. Disabling DNSSEC does not make a difference as well. The stubby config is left at default:

# cat /etc/stubby/stubby.yml
resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 128
appdata_dir: "/var/lib/misc"
resolvconf: "/tmp/resolv.conf"
edns_client_subnet_private: 1
round_robin_upstreams: 1
idle_timeout: 9000
tls_connection_retries: 2
tls_backoff_time: 900
timeout: 3000
listen_addresses:
  - 127.0.1.1@53
upstream_recursive_servers:
  - address_data: 1.1.1.1
    tls_auth_name: "cloudflare-dns.com"

I have already tried a factory reset of all setting, but no changes. MerlinWRT version is 384.19.

I would appreciate any debugging/configuration advice.

Best regards,
Murgi

 

[eibgrad]

FWIW, I've had the same experience w/ Stubby on FreshTomato. Eventually had to abandon it. Currently using DNSCrypt, but it too can be slow at times. In fact, that's a problem w/ many of these new methodologies when it comes to DNS. Just too painfully slow at times. Recently tried changing the cache size in DNSMasq to see if it helps (at least once I have it, I can minimize the impact on the resolver), but too early to tell.

Frankly, I think this whole thing w/ privacy and security involving DNS is a mess. Waaaay too complicated, and too many issues. It's enough to make you go back to the old way. How do regular/normal folk ever figure this stuff out??

 

[bbunge]

Looks like you only have one upstream server configured for DoT. Not a great idea. Two is better and as you are using CF I suggest you add the 1.0.0.1 DoT setting. Stubby will alternate between the upstream resolvers. If you are using IPV6 you can alternate between the IPV4 and IPV6 CF resolvers although the CF IPV4 DNS resolvers will do IPV6 addresses.
I have had some concerns with the Merlin Stubby settings since it was included in the firmware. Likely just my difference of opinion and the way we set stubby up in Entware before the adoption. You might try to disable DNSSEC in the Merlin GUI and enable it in Stubby with a config add file.
As for me I am using Stubby for DoT and DNSSEC on a Pi-Hole with great success.

 

[Treadler]

I use just one server, Cloudflare’s secondary IPv6 one.
Works just fine.

Previously I had all sorts of bother with Cloudflare via DoT, disabled pixelserv within Diversion, & all fixed! YMMV.

 

[Murgi]

I already tried using multiple resolvers (the config above is my minimal test config), using just the IPv6 CF resolver, as well as disabling DNSSEC completely. While there might be some differences in the delay between these config (it's not always easy to measure this), it remains way to high to leave this enabled.

I currently use nothing but stock Merlin (no Diversion etc.). For good measure I set stubby with DoT up on my Raspberry Pi 4 and it works like a charm. However, I would love to be able to run it directly on the router without adding this much delay.

 

[ColinTaylor]

Try turning off IPv6 completely on your router.

 

[Murgi]

Try turning off IPv6 completely on your router.

Did not change anything.

I tried using 8.8.8.8 and 8.8.4.4 with DoT and they seem to be much faster ~200-300ms/query. I would prefer using either quad9 or Cloudflare, but I am starting to suspect an ISP-side routing problem ... DoT queries on the Raspberry Pi against Quad9 and 1.1.1.1 also seem to be slow again.

 

[bbunge]

Did not change anything.

I tried using 8.8.8.8 and 8.8.4.4 with DoT and they seem to be much faster ~200-300ms/query. I would prefer using either quad9 or Cloudflare, but I am starting to suspect an ISP-side routing problem ... DoT queries on the Raspberry Pi against Quad9 and 1.1.1.1 also seem to be slow again.

That seems to be a problem with Anycast IP addresses. CF and Quad9 both have resolvers within 100 miles of my place but my ISP routes Quad9 DNS queries to a resolver 1,000 miles away. Other ISP's do the opposite! I've considered complaining to my ISP. But my long term experience is that "they know what is best for us."

 

[criminala]

I am also facing some issues with DNS over TLS . Every couple of weeks , webpages don't load or only sporadically load . images not appearing , etc.

I am using quad9 , with DoT and DNSsec .

Today I again had to disable the DNS privacy protocol setting in Merlinwrt 384.19 . From the moment i disable it everything works perfectly again .
I enable that one option and the issues start . I have no idea why I am getting this every couple of weeks .
(rebooting the router or trying other dns services are not solving the issue)

I am attaching a screenshot of the option I have to disable in order for the dns resolving to get up to speed again . (marked in red)

 

[Centrifuge]

I am also facing some issues with DNS over TLS . Every couple of weeks , webpages don't load or only sporadically load . images not appearing , etc.

I am using quad9 , with DoT and DNSsec .

Today I again had to disable the DNS privacy protocol setting in Merlinwrt 384.19 . From the moment i disable it everything works perfectly again .
I enable that one option and the issues start . I have no idea why I am getting this every couple of weeks .
(rebooting the router or trying other dns services are not solving the issue)

I am attaching a screenshot of the option I have to disable in order for the dns resolving to get up to speed again . (marked in red)

If you search around there have been numerous discussions about the reliability of different services, and depends where you are at also. I experienced the same with Quad 9 last year, and after a daliance with Unbound, I went back to DoT and I switched to Cleanbrowsing (security) 1 and 2. It's worked without any intermittent problems.

 

[criminala]

I have removed all traces of the quad9 configuration , and replaced it with a single server from Snopyta . No backup dns server configured as I did before .
This seems to have improved things a lot .

From the moment I insert quad9 (even as a backup DNS) , troubles start again (needing multiple refreshes to get a webpage to load , not loading of images , ...)

So even though my DNS tests show quad9 as being responsive and snappy , in reality there is an issue with quad9 (even when used as a backup/second DNS) .
Snopyta response times are much slower yet their DNS server provides a much better experience to me .

DNS Test ran from a linux machine in the network :

test1 test2 test3 test4 test5 test6 test7 test8 test9 test10 Average
neustar 14 ms 14 ms 14 ms 14 ms 14 ms 14 ms 14 ms 15 ms 14 ms 14 ms 14.10
norton 15 ms 15 ms 15 ms 15 ms 15 ms 15 ms 15 ms 15 ms 15 ms 15 ms 15.00
comodo 16 ms 16 ms 17 ms 16 ms 16 ms 16 ms 15 ms 16 ms 16 ms 18 ms 16.20
adguard 17 ms 17 ms 16 ms 17 ms 19 ms 17 ms 19 ms 18 ms 20 ms 18 ms 17.80
cleanbrowsing 19 ms 17 ms 15 ms 15 ms 15 ms 16 ms 15 ms 15 ms 15 ms 45 ms 18.70
opendns 26 ms 17 ms 17 ms 25 ms 18 ms 26 ms 17 ms 24 ms 17 ms 20 ms 20.70
cloudflare 31 ms 18 ms 21 ms 12 ms 17 ms 21 ms 28 ms 32 ms 17 ms 23 ms 22.00
level3 39 ms 27 ms 21 ms 27 ms 25 ms 27 ms 30 ms 21 ms 22 ms 27 ms 26.60
google 32 ms 19 ms 38 ms 30 ms 36 ms 24 ms 17 ms 52 ms 31 ms 21 ms 30.00
quad9 23 ms 80 ms 26 ms 26 ms 23 ms 104 ms 21 ms 27 ms 29 ms 26 ms 38.50
snopyta(active) 46 ms 41 ms 42 ms 44 ms 41 ms 45 ms 41 ms 41 ms 38 ms 41 ms 42.00
yandex 41 ms 42 ms 43 ms 42 ms 42 ms 42 ms 42 ms 42 ms 42 ms 68 ms 44.60
127.0.0.53 63 ms 874 ms 78 ms 47 ms 921 ms 404 ms 48 ms 114 ms 1 ms 84 ms 263.40
freenom 1000 ms 580 ms 1189 ms 1000 ms 1000 ms 1648 ms 1000 ms 1000 ms 1000 ms 1000 ms 1041.70