Multiple DNS servers - Requests don't split between them

[zakazak]

I am running several different DNS servers and my goal was to have all requests being split on those different DNS servers.
For this example I used: 1x NextDNS + 1x AdguardDNS

Both have a simple webinterface that logs the requests.
When I now check the requests from the last 10 minutes, both DNS servers show exactly the same output (maybe with 1 request missing in one of them).
So that means that when opening e.g. "testlink.com" the DNS request will be sent to both DNS servers instead of just one.
I was hoping that when opening e.g. "testlink.com + testlink2.com + testlink3.com" the output would be:

• testlink.com -> DNS Server 1
• testlink2.com -> DNS Server 2
• testlink3.com -> DNS Server 1

Is this a bug, or intended, or a configuration fault on my side?
Btw: RT-AX86U with 388.1

Thanks in advance!

 

[drinkingbird]

As far as I know dnsmasq does not support round robin, you need special hardware/software for that.

 

[zakazak]

As far as I know dnsmasq does not support round robin, you need special hardware/software for that.

According to this post by @RMerlin is is supported and enabled by default. This was 2019 though:

[Preview] Asuswrt-Merlin 384.11 with DNS over TLS

Any feedback on the webui implementation? Does it look intuitive to use in its current form?Well, it seems OK. Would like option for round robin. Not sure if spki is needed as it seems to confuse folks and is not required by major DNS providers. Do like ability to add resolvers As for the...

www.snbforums.com

 

[ColinTaylor]

Are you talking about regular DNS or DoT?

 

[zakazak]

Are you talking about regular DNS or DoT?

I am using Dns over TLS with adguard and nextdns

 

[bbunge]

Did you consider that dnsmasq caches requests? Stubby does do roundrobin requests. Log into a terminal and run stubby -l to see stubby work.

 

[zakazak]

Did you consider that dnsmasq caches requests? Stubby does do roundrobin requests. Log into a terminal and run stubby -l to see stubby work.

That doesn't make sense.

I see the same request on both dns servers at the same second.
What has this to do with cache?

Over a period of 30 days I have nearly the same DNS request count on both DNS Server. E.g. 265.123 and 265.118

 

[ColinTaylor]

Post the output of your /etc/stubby/stubby.yml and /etc/dnsmasq.conf so that we can see what's going on.

 

[zakazak]

Post the output of your /etc/stubby/stubby.yml and /etc/dnsmasq.conf so that we can see what's going on.

stubby.yml:

resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
- GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 128
appdata_dir: "/var/lib/misc"
resolvconf: "/tmp/resolv.conf"
edns_client_subnet_private: 1
round_robin_upstreams: 1
idle_timeout: 9000
tls_connection_retries: 2
tls_backoff_time: 900
timeout: 3000
listen_addresses:
- 127.0.1.1@53
upstream_recursive_servers:
- address_data: xx.xx.xx.xx
tls_auth_name: "myalias.dns.nextdns.io"
- address_data: xx.xx.xx.xx
tls_auth_name: "myalias.d.adguard-dns.com"

dnsmasq.conf:

pid-file=/var/run/dnsmasq.pid
user=nobody
bind-dynamic
interface=br0
interface=pptp*
no-dhcp-interface=pptp*
no-resolv
servers-file=/tmp/resolv.dnsmasq
no-poll
no-negcache
cache-size=1500
min-port=4096
domain=RouterDomain
expand-hosts
bogus-priv
domain-needed
local=/RouterDomain/
dhcp-range=lan,192.168.50.2,192.168.50.254,255.255.255.0,86400s
dhcp-option=lan,3,192.168.50.1
dhcp-option=lan,15,RouterDomain
dhcp-option=lan,252,"\n"
dhcp-authoritative
interface=tun21
interface=br1
dhcp-range=br1,192.168.101.2,192.168.101.254,255.255.255.0,86400s
dhcp-option=br1,3,192.168.101.1
interface=br2
dhcp-range=br2,192.168.102.2,192.168.102.254,255.255.255.0,86400s
dhcp-option=br2,3,192.168.102.1
dhcp-host=1C:1B:0D:9E:FB:B5,set:1C:1B:0D:9E:FB:B5,192.168.50.27
dhcp-host=58:24:29A:A0D,set:58:24:29A:A0D,192.168.50.15
dhcp-host=0C:96:E6:C0:9D:5C,set:0C:96:E6:C0:9D:5C,192.168.50.153
trust-anchor=.x
dnssec
stop-dns-rebind
rebind-domain-ok=dns.msftncsi.com
address=/use-application-dns.net/
address=/_dns.resolver.arpa/
dhcp-name-match=set:wpad-ignore,wpad
dhcp-ignore-names=tag:wpad-ignore
dhcp-script=/sbin/dhcpc_lease
script-arp
edns-packet-max=1280

 

[zakazak]

stuby -l example:

[12:49:03.694098] STUBBY: Read config from file /etc/stubby/stubby.yml
[12:49:03.694423] STUBBY: Stubby version: Stubby 0.4.2
[12:49:03.694839] STUBBY: DNSSEC Validation is OFF
[12:49:03.694886] STUBBY: Transport list is:
[12:49:03.694929] STUBBY: - TLS
[12:49:03.694972] STUBBY: Privacy Usage Profile is Strict (Authentication required)
[12:49:03.695015] STUBBY: (NOTE a Strict Profile only applies when TLS is the ONLY transport!!)
[12:49:03.695059] STUBBY: Starting DAEMON....
[12:49:10.002375] STUBBY: 45.90.28.41 : Upstream : Could not setup TLS capable TFO connect

[12:49:10.002781] STUBBY: 45.90.28.41 : Conn opened: TLS - Strict Profile

[12:49:10.003117] STUBBY: 94.140.14.49 : Conn opened: TLS - Strict Profile

[12:49:10.067486] STUBBY: 45.90.28.41 : Verify passed : TLS

[12:49:10.073071] STUBBY: 94.140.14.49 : Verify passed : TLS

[12:49:28.869861] STUBBY: 45.90.28.41 : Conn closed: TLS - Resps= 4, Timeouts = 0, Curr_auth =Success, Keepalive(ms)= 9000

[12:49:28.869913] STUBBY: 45.90.28.41 : Upstream : TLS - Resps= 4, Timeouts = 0, Best_auth =Success

[12:49:28.869935] STUBBY: 45.90.28.41 : Upstream : TLS - Conns= 1, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0

[12:49:28.889997] STUBBY: 94.140.14.49 : Conn closed: TLS - Resps= 4, Timeouts = 0, Curr_auth =Success, Keepalive(ms)= 9000

[12:49:28.890026] STUBBY: 94.140.14.49 : Upstream : TLS - Resps= 4, Timeouts = 0, Best_auth =Success

[12:49:28.890047] STUBBY: 94.140.14.49 : Upstream : TLS - Conns= 1, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0

[12:49:34.308959] STUBBY: 45.90.28.41 : Conn opened: TLS - Strict Profile

[12:49:34.309216] STUBBY: 94.140.14.49 : Conn opened: TLS - Strict Profile

[12:49:34.363735] STUBBY: 45.90.28.41 : Verify passed : TLS

[12:49:34.368249] STUBBY: 94.140.14.49 : Verify passed : TLS

[12:49:43.437154] STUBBY: 45.90.28.41 : Conn closed: TLS - Resps= 1, Timeouts = 0, Curr_auth =Success, Keepalive(ms)= 9000

[12:49:43.437207] STUBBY: 45.90.28.41 : Upstream : TLS - Resps= 5, Timeouts = 0, Best_auth =Success

[12:49:43.437229] STUBBY: 45.90.28.41 : Upstream : TLS - Conns= 2, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0

[12:49:43.440960] STUBBY: 94.140.14.49 : Conn closed: TLS - Resps= 1, Timeouts = 0, Curr_auth =Success, Keepalive(ms)= 9000

[12:49:43.440989] STUBBY: 94.140.14.49 : Upstream : TLS - Resps= 5, Timeouts = 0, Best_auth =Success

[12:49:43.441011] STUBBY: 94.140.14.49 : Upstream : TLS - Conns= 2, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0

[12:50:02.307702] STUBBY: 45.90.28.41 : Conn opened: TLS - Strict Profile

[12:50:02.307986] STUBBY: 94.140.14.49 : Conn opened: TLS - Strict Profile

[12:50:02.361237] STUBBY: 45.90.28.41 : Verify passed : TLS

[12:50:02.365854] STUBBY: 94.140.14.49 : Verify passed : TLS

[12:50:11.440582] STUBBY: 45.90.28.41 : Conn closed: TLS - Resps= 1, Timeouts = 0, Curr_auth =Success, Keepalive(ms)= 9000

[12:50:11.440634] STUBBY: 45.90.28.41 : Upstream : TLS - Resps= 6, Timeouts = 0, Best_auth =Success

[12:50:11.440657] STUBBY: 45.90.28.41 : Upstream : TLS - Conns= 3, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0

[12:50:11.445105] STUBBY: 94.140.14.49 : Conn closed: TLS - Resps= 1, Timeouts = 0, Curr_auth =Success, Keepalive(ms)= 9000

[12:50:11.445134] STUBBY: 94.140.14.49 : Upstream : TLS - Resps= 6, Timeouts = 0, Best_auth =Success

[12:50:11.445157] STUBBY: 94.140.14.49 : Upstream : TLS - Conns= 3, Conn_fails= 0, Conn_shuts= 0, Backoffs = 0

/etc/resolv.conf:

nameserver 1.1.1.1
nameserver 9.9.9.9
nameserver 127.0.1.1

 

[bbunge]

stuby -l example:



/etc/resolv.conf:

Sure looks like things are working as intended. If it worrys you, quit looking at the logs.

 

[zakazak]

Sure looks like things are working as intended. If it worrys you, quit looking at the logs.

The logs show that it does not work as intented.

Each request goes to both DNS servers instead of just one?!

 

[ColinTaylor]

Your configs look correct. I suggest you log it as an issue upstream:

GitHub - getdnsapi/stubby: Stubby is the name given to a mode of using getdns which enables it to act as a local DNS Privacy stub resolver (using DNS-over-TLS).

Stubby is the name given to a mode of using getdns which enables it to act as a local DNS Privacy stub resolver (using DNS-over-TLS). - getdnsapi/stubby

github.com

P.S. Your /etc/resolv.conf looks wrong (are you using a VPN client?) but that's another issue.

 

[bbunge]

Your configs look correct. I suggest you log it as an issue upstream:

GitHub - getdnsapi/stubby: Stubby is the name given to a mode of using getdns which enables it to act as a local DNS Privacy stub resolver (using DNS-over-TLS).

Stubby is the name given to a mode of using getdns which enables it to act as a local DNS Privacy stub resolver (using DNS-over-TLS). - getdnsapi/stubby

github.com

P.S. Your /etc/resolv.conf looks wrong (are you using a VPN client?) but that's another issue.

The resolve config is OK. The loop back ip is used by Stubby.

 

[ColinTaylor]

@zakazak What do you have in /tmp/resolv.dnsmasq and /tmp/resolv.conf

 

[zakazak]

@zakazak What do you have in /tmp/resolv.dnsmasq and /tmp/resolv.conf

/tmp/resolv.conf:

nameserver 1.1.1.1
nameserver 9.9.9.9
nameserver 127.0.1.1

/tmp/resolv.dnsmasq:

server=127.0.1.1

 

[ColinTaylor]

I can't recreate your problem but then I'm not using the 388.1 firmware. My version of stubby is 0.4.0.

I used namebench to send 200 uncached queries to the router. 100 went to my first NextDNS server and 100 went to the second NextDNS server.

 

[zakazak]

I can't recreate your problem but then I'm not using the 388.1 firmware. My version of stubby is 0.4.0.

I used namebench to send 200 uncached queries to the router. 100 went to my first NextDNS server and 100 went to the second NextDNS server.

That is exactly the outcome I want :S

 

[drinkingbird]

I am using Dns over TLS with adguard and nextdns

Sorry didn't realize you were using DOT.

Out of curiosity I tested mine (2 servers, not using DOT, just regular dnsmasq) and the vast majority of requests go to the first server as expected and only a few that hit timeout or fail go to the second.

I believe there is a way to make dnsmasq query all servers in parallel and return whichever responds first, but I don't want to create additional load especially when I already know cleanbrowsing's secondary server is significantly slower than the primary for me.

Out of curiosity why would you want to round robin two totally different DNS filtering services? Or was that just for example?

EDIT - actually seems somewhat inconsistent. If I do a single nslookup from my laptop (or browse to a site) it looks up to both of my WAN servers. Same source port on my WAN IP, going to both DNS. But I know based on the response time on my PC, the router is definitely returning the one from the primary server. So it appears this implementation of dnsmasq is using the "all" parameter by default.

But oddly if I use namebench to test 250 IPs, the vast majority go to primary only and only some go to secondary which I was assuming are ones that exceeded timeout or got NXDOMAIN etc. But maybe there is some throttling going on where it no longer does the dual lookups if the queue starts building up or something. Not really sure.

EDIT 2 - Just did a test, and it confirms that the behavior seems to be to send to both servers and reply with quickest:
Test 1 - Router set to single WAN DNS server only, known fast one - namebench average 38.79msec
Test 2 - Same but with known slow DNS server - namebench average 258.33msec
Test 3 - Both servers defined, faster one as #1 - namebench average 31.26msec
Test 4 - Same but with faster one as #2 - namebench average 23.94msec
Test 5 - Ran Test 2 again to make sure above wasn't just excessive caching - namebench average 250.84

As to what I was seeing in in conntrack where it showed most requests going to the primary server, maybe it stops querying the slower server after a while or maybe the log just wraps too quickly for it to be useful.

 

[bbunge]

That is exactly the outcome I want :S

You need to choose one upstream DNS provider. Using two different will give you erratic results as it is a good chance the two companies will be in different data centers. The Stubby roundrobin is designed to share the DoT load. I feel that disabling roundrobin provides as good as results. When I run Merlin I also turn on DNSSEC in Stubby. For now the Asus firmware works well for me.