Multiple DNS servers - Requests don't split between them

[zakazak]

You need to choose one upstream DNS provider. Using two different will give you erratic results as it is a good chance the two companies will be in different data centers. The Stubby roundrobin is designed to share the DoT load. I feel that disabling roundrobin provides as good as results. When I run Merlin I also turn on DNSSEC in Stubby. For now the Asus firmware works well for me.

Well I tried the same with 2x NextDNS servers...even with 4x NextDNS servers. All show the exact same queries instead of splitting them on the different servers.

 

[drinkingbird]

Well I tried the same with 2x NextDNS servers...even with 4x NextDNS servers. All show the exact same queries instead of splitting them on the different servers.

My guess is it is acting the same way I just observed dnsmasq acting. Querying all servers and returning the first response it gets back, in order to ensure the fastest resolution every time.

 

[zakazak]

My guess is it is acting the same way I just observed dnsmasq acting. Querying all servers and returning the first response it gets back, in order to ensure the fastest resolution every time.

So I would need to disable the "speed check" in dnsmasq?

It doesn't make sense anyway, since the DNS servers I add are usually the fastest ones I can get.

Edit: or I need to get dnsmasq to correctly rotate through the list of dns servers.

 

[drinkingbird]

So I would need to disable the "speed check" in dnsmasq?

It doesn't make sense anyway, since the DNS servers I add are usually the fastest ones I can get.

Edit: or I need to get dnsmasq to correctly rotate through the list of dns servers.

DNSMasq won't round robin. Looks like there is an --all-servers that is enabled by default here, which means it queries all at once and sends back the first response it gets. The other behavior, disabling that, is query the primary only, and if primary doesn't respond, go to secondary, or I believe it will actually determine which one is fastest and stick to that as primary, not sure how often it re-checks.

But if you're using DoT I think you need to look at settings for stubby, it sounds like it is behaving the same way. Makes sense, since this results in the fastest response every time.

 

[dave14305]

So I would need to disable the "speed check" in dnsmasq?

It doesn't make sense anyway, since the DNS servers I add are usually the fastest ones I can get.

Edit: or I need to get dnsmasq to correctly rotate through the list of dns servers.

When DNS Privacy (stubby) is enabled, dnsmasq only has one upstream server (stubby listening at 127.0.1.1), so there is no rotation possible. It will always send to stubby and stubby will determine which upstream to use from its’ own configuration.

Just curious how you are seeing the evidence of the problem. Are the query types the exact same at both upstream services (A, AAAA, Type65, dnssec, etc.)?

 

[bbunge]

So I would need to disable the "speed check" in dnsmasq?

It doesn't make sense anyway, since the DNS servers I add are usually the fastest ones I can get.

Edit: or I need to get dnsmasq to correctly rotate through the list of dns servers.

Like it or not your router and the included software is working as it should work. Sure, you can mess with the dnsmasq and Stubby settings and likely make things worse. Some of us have worked with these settings for years to make the features work better. Your intended quest will wear you down. Leave it and get on with life!

 

[zakazak]

I can't recreate your problem but then I'm not using the 388.1 firmware. My version of stubby is 0.4.0.

I used namebench to send 200 uncached queries to the router. 100 went to my first NextDNS server and 100 went to the second NextDNS server.

Since your setup works exactly how I would like it to work:
Which firmware and stubby version are you using?

 

[zakazak]

When DNS Privacy (stubby) is enabled, dnsmasq only has one upstream server (stubby listening at 127.0.1.1), so there is no rotation possible. It will always send to stubby and stubby will determine which upstream to use from its’ own configuration.

Just curious how you are seeing the evidence of the problem. Are the query types the exact same at both upstream services (A, AAAA, Type65, dnssec, etc.)?

Ah okay thanks for clarifyinng. So its stubby that I need to look at.

For evidence I just check the logs on the webinterfaces of e.g. NextDNS (by using two different NextDNS accounts and NextDNS servers).
Then filter for e.g. "the last 24 hours" and I see basically exactly the same log which (according to the Adguard log) are mostly "A" and "DS" types.
When filtering for e.g. the last "31 days" I get pretty much the same armount of queries/blocks statistics on both providers.

Funnily for writing this reply I compared again with having 1x NextDNS and 1x AdguardDNS configured in my router.
Now the logs look different. Another factor is, that an app might try several times in a row to contact it's server when its blocked.
E.g. if it does 4 tries, it will show 2 entries on each DNS resolver, all at pretty much the same time.
Maybe all of this also doesn't work with 2x NextDNS accounts because NextDNS links them as "one account"?

Any suggestions on how to run a more fool proof test?

 

[ColinTaylor]

Any suggestions on how to run a more fool proof test?

Start Stubby with logging enabled and then quickly run namebench using 400 uncached queries. Wait until the connections are closed. Do you see ~200 queries to each server?

[14:54:17.510315] STUBBY: Starting DAEMON....
[14:54:24.555049] STUBBY: 45.90.28.0                               : Upstream   : Could not setup TLS capable TFO connect

[14:54:24.555789] STUBBY: 45.90.28.0                               : Conn opened: TLS - Strict Profile

[14:54:24.605081] STUBBY: 45.90.28.0                               : Verify passed : TLS

[14:54:24.904153] STUBBY: 45.90.30.0                               : Conn opened: TLS - Strict Profile

[14:54:24.951891] STUBBY: 45.90.30.0                               : Verify passed : TLS

[14:54:45.466221] STUBBY: 45.90.30.0                               : Conn closed: TLS - Resps=   202, Timeouts  =     0, Curr_auth =Success, Keepalive(ms)=  9000

[14:54:45.466268] STUBBY: 45.90.30.0                               : Upstream   : TLS - Resps=   202, Timeouts  =     0, Best_auth =Success

[14:54:45.466290] STUBBY: 45.90.30.0                               : Upstream   : TLS - Conns=     1, Conn_fails=     0, Conn_shuts=      0, Backoffs     =     0

[14:54:53.209413] STUBBY: 45.90.28.0                               : Conn closed: TLS - Resps=   203, Timeouts  =     0, Curr_auth =Success, Keepalive(ms)=  9000

[14:54:53.209464] STUBBY: 45.90.28.0                               : Upstream   : TLS - Resps=   203, Timeouts  =     0, Best_auth =Success

[14:54:53.209486] STUBBY: 45.90.28.0                               : Upstream   : TLS - Conns=     1, Conn_fails=     0, Conn_shuts=      0, Backoffs     =     0

 

[zakazak]

Start Stubby with logging enabled and then quickly run namebench using 400 uncached queries. Wait until the connections are closed. Do you see ~200 queries to each server?

View attachment 47713

[14:54:17.510315] STUBBY: Starting DAEMON....
[14:54:24.555049] STUBBY: 45.90.28.0                               : Upstream   : Could not setup TLS capable TFO connect

[14:54:24.555789] STUBBY: 45.90.28.0                               : Conn opened: TLS - Strict Profile

[14:54:24.605081] STUBBY: 45.90.28.0                               : Verify passed : TLS

[14:54:24.904153] STUBBY: 45.90.30.0                               : Conn opened: TLS - Strict Profile

[14:54:24.951891] STUBBY: 45.90.30.0                               : Verify passed : TLS

[14:54:45.466221] STUBBY: 45.90.30.0                               : Conn closed: TLS - Resps=   202, Timeouts  =     0, Curr_auth =Success, Keepalive(ms)=  9000

[14:54:45.466268] STUBBY: 45.90.30.0                               : Upstream   : TLS - Resps=   202, Timeouts  =     0, Best_auth =Success

[14:54:45.466290] STUBBY: 45.90.30.0                               : Upstream   : TLS - Conns=     1, Conn_fails=     0, Conn_shuts=      0, Backoffs     =     0

[14:54:53.209413] STUBBY: 45.90.28.0                               : Conn closed: TLS - Resps=   203, Timeouts  =     0, Curr_auth =Success, Keepalive(ms)=  9000

[14:54:53.209464] STUBBY: 45.90.28.0                               : Upstream   : TLS - Resps=   203, Timeouts  =     0, Best_auth =Success

[14:54:53.209486] STUBBY: 45.90.28.0                               : Upstream   : TLS - Conns=     1, Conn_fails=     0, Conn_shuts=      0, Backoffs     =     0

Hmm namebench gives me the error "Outgoing requests were intercepted!"

Your router or internet service provider appears to be intercepting and redirecting all outgoing DNS requests.
This means you cannot benchmark or utilitize alternate DNS servers.
Please adjust your router configuration or file a support request with your ISP.

I used my routers IP as nameserver in namebench.
DNS rebind protection is disabled.

Edit: It was the DNS Director that made namebench fail. I disabled it, did 400 requests.
200 queries show up on NextDNS. The other 200 are probably done via adguard but I can't say for sure since the webinterface of adguard seems to be down atm lol

Edit2: So I now added 2 different NextDNS servers / accounts.
The 400 queries split up 200/200 on each account.
So now it suddenly seems to work.

 

[ColinTaylor]

Disable DNS Director.

 

[zakazak]

Disable DNS Director.

In general or just for benchmarkin with namebench?

 

[ColinTaylor]

In general or just for benchmarkin with namebench?

Just while you do the testing.

 

[zakazak]

Just while you do the testing.

Yep I did that (well, I changed it to "no redirection on global level) and it allowed me to run namebench.
It seems like the 400 requests got pretty evenly split on the added DNS servers.

For now I added 4x NextDNS and 1x AdguardDNS. They all use the same filterlist.

Since one free account allows 300.000 queries per month, this now would allow me 1.500.000 queries per month since those queries get split on the 5 accounts.
Lets see if that works...

 

[JaneDilbert]

Hi @zakazak. I'm trying to do exactly the same thing you did, and your post was the only one I found around the web.

But I'm having a similar issue: Any request forwarded to stubby, gets simultaneously spread (multiplied) across all NextDNS servers; definitely not round robin.

Did your setup work and now everything gets evenly distributed? And if so, would you please share your config? Not sure what am I missing in here.

Cheers

 

[bbunge]

Hi @zakazak. I'm trying to do exactly the same thing you did, and your post was the only one I found around the web.

But I'm having a similar issue: Any request forwarded to stubby, gets simultaneously spread across all NextDNS servers; definitely not round robin.

Did your setup work and now everything gets evenly distributed? And if so, would you please share your config? Not sure what am I missing in here.

Cheers

Yes, that is what roundrobin does. Share the load. Nothing has changed with Stubby over the years.

 

[JaneDilbert]

Yes, that is what roundrobin does. Share the load. Nothing has changed with Stubby over the years.

Thanks @bbunge for your reply.

What I observed was, somehow, stubby not acting as I would've expected on a round robin (one query on the first one, next query on the second one, and so on) but hitting all the servers simultaneously for the same query.

zakazak described a different behaviour with everything properly distributed across the servers: Any dig/nslookup/page/etc I trigger, I can see that hitting all the DNS servers at the same time.

But now I tried again, and like that, is gone: Everything is being done round robin. I must have missed something, but I don't have a clue what could've been.

Anyhow, thanks.