Diversion Diversion 4.3.3 - the Router Ad-Blocker, released April 02 2023

[JimbobJay]

Yea so you should add those domains through diversion then.

I've tried adding a few, such as the more obvious paypal.com, paypal.au, .co.uk etc.
Still not working. I've also now noticed the same behavior on a random football fans forum page too. Adding that page, which thankfully only has one possible variation, to diversion's whitelist has no effect.

Surely this must be some kind of bug in diversion, no? If disabling diversion suddenly enables access to these sites, and that is the only change I make to get it to work, and re-enabling diversion breaks access again, then shouldn't diversion be reporting in its log what it must clearly be blocking?

 

[SomeWhereOverTheRainBow]

I've tried adding a few, such as the more obvious paypal.com, paypal.au, .co.uk etc.
Still not working. I've also now noticed the same behavior on a random football fans forum page too. Adding that page, which thankfully only has one possible variation, to diversion's whitelist has no effect.

Surely this must be some kind of bug in diversion, no? If disabling diversion suddenly enables access to these sites, and that is the only change I make to get it to work, and re-enabling diversion breaks access again, then shouldn't diversion be reporting in its log what it must clearly be blocking?

Try using the smallest block list only then if you don't feel like whitelisting all that stuff. There are two ways you can whitelist, you can use diversions dnsmasq log diagnostics to watch for your specific devices domains to be blocked yourself, and then manually whitelist till you get it right. Or you can whitelist the domains I gave you. Or you can try a different blocklist. It is not a "bug" in diversion. More so a problem with the blocklist selected because it is blocking a false positive. If you have pixelserv-tls enabled, you could even try disabling that. The only thing diversion is doing is what ever you request it to. (Or what ever blocklist you chose to use when installing it).

 

[JimbobJay]

Try using the smallest block list only then if you don't feel like whitelisting all that stuff. There are two ways you can whitelist, you can use diversions dnsmasq log diagnostics to watch for your specific devices domains to be blocked yourself, and then manually whitelist till you get it right. Or you can whitelist the domains I gave you. Or you can try a different blocklist. It is not a "bug" in diversion. More so a problem with the blocklist selected because it is blocking a false positive. If you have pixelserv-tls enabled, you could even try disabling that. The only thing diversion is doing is what ever you request it to. (Or what ever blocklist you chose to use when installing it).

But that’s the point I’m trying to make. I have been following diversion’s log to see which domain is being blocked to see what I can try whitelisting, but nothing is appearing when I am going to these blocked sites, but clearly something is being blocked by diversion that isn’t being shown.

Except in the case of the football forum, when a generic google ad serve pops up in the log, which I have tried whitelisting just to see but as expected no change.

I have also tried changing the block list, trying every list down including minimal. Clearing history and cache in browsers after each change.


The only thing that consistently works at making these sites work again is disabling diversion. But seeing as following the log when it is enabled isn’t bringing up any blocks I feel like a bug isn’t beyond the realms of possibility, surely?

 

[JimbobJay]

But that’s the point I’m trying to make. I have been following diversion’s log to see which domain is being blocked to see what I can try whitelisting, but nothing is appearing when I am going to these blocked sites, but clearly something is being blocked by diversion that isn’t being shown.

Except in the case of the football forum, when a generic google ad serve pops up in the log, which I have tried whitelisting just to see but as expected no change.

I have also tried changing the block list, trying every list down including minimal. Clearing history and cache in browsers after each change.


The only thing that consistently works at making these sites work again is disabling diversion. But seeing as following the log when it is enabled isn’t bringing up any blocks I feel like a bug isn’t beyond the realms of possibility, surely?

Ok I've narrowed it down and figured it out. Apparently it's Blocking Type 65 being enabled in Diversion that was causing this issue. Disabling Type 65 blocking instantly makes these sites work again. I think I was experiencing a similar issue to what's mentioned in this macrumours thread

Seems to be a combination of diversion and how apple devices are handling type 65 queries, but yes, enabling type 65 in diversion is breaking websites and apps on apple devices that aren't even in the diversion block list. However diversion is responding to type 65 queries, Apple devices aren't handling well, because they just hang waiting for a response.

Not sure if there's anything that can be done on diversion's end or if it's purely down to apple fixing their broken handling of this (which I won't hold my breath on), but with type 65 block disabled I'm now seeing ads come through, but no broken websites. So not a great solution for adblocking software unfortunately

 

[SomeWhereOverTheRainBow]

Ok I've narrowed it down and figured it out. Apparently it's Blocking Type 65 being enabled in Diversion that was causing this issue. Disabling Type 65 blocking instantly makes these sites work again. I think I was experiencing a similar issue to what's mentioned in this macrumours thread

Seems to be a combination of diversion and how apple devices are handling type 65 queries, but yes, enabling type 65 in diversion is breaking websites and apps on apple devices that aren't even in the diversion block list. However diversion is responding to type 65 queries, Apple devices aren't handling well, because they just hang waiting for a response.

Not sure if there's anything that can be done on diversion's end or if it's purely down to apple fixing their broken handling of this (which I won't hold my breath on), but with type 65 block disabled I'm now seeing ads come through, but no broken websites. So not a great solution for adblocking software unfortunately

Unfortunately there is no cut and dry solution for type65 blocking with dnsmasq. (Not particularly ideal for Diversion because it is relying on the iptables to drop the requests.) For type 65 blocking in dnsmasq, you either have to block the domain 100percent, or you have to make special rules for dnsmasq per hostname that uses type65. There is no wildcard support yet for all type65. Maybe someone will eventually put a request in with Simon of dnsmasq development for such a feature. Until that happens, the only effective sources for blocking type65 requests are adguardhome and pihole. With my pihole, I have one regex rule that returns empty responses for type65 requests.

 

[charlesb224]

Unfortunately there is no cut and dry solution for type65 blocking with dnsmasq. (Not particularly ideal for Diversion because it is relying on the iptables to drop the requests.) For type 65 blocking in dnsmasq, you either have to block the domain 100percent, or you have to make special rules for dnsmasq per hostname that uses type65. There is no wildcard support yet for all type65. Maybe someone will eventually put a request in with Simon of dnsmasq development for such a feature. Until that happens, the only effective sources for blocking type65 requests are adguardhome and pihole. With my pihole, I have one regex rule that returns empty responses for type65 requests.

Do you have that type 65 regex that you would be willing to share for other pihole users?

 

[SomeWhereOverTheRainBow]

Do you have that type 65 regex that you would be willing to share for other pihole users?

.*;querytype=HTTPS

 

[charlesb224]

.*;querytype=HTTPS

View attachment 48040

Thank-you. A very simple filter!

 

[SomeWhereOverTheRainBow]

Thank-you. A very simple filter!

if you decide you want to try something a little more precise you could do,

(\.|^)(([[:alnum:]_-]{1,63}\.)+([-[:alnum:]]{1,})?([[:alnum:]]{0,}[[:alpha:]]{1}){2,7})$;querytype=HTTPS

However it will come at a cost of additional processing time per https query. The original method is the preferred solution offered in the pihole guides because its simplicity allows for the fastest response time.

History of type65 request in pihole:

Initially the request was categorized as a q-type in the "other" category.

Eventually the Pihole Developers changed the q-type to match "https" category.

I also block the "ANY" q-type:

.*;querytype=ANY

 

[thelonelycoder]

There is no wildcard support yet for all type65. Maybe someone will eventually put a request in with Simon of dnsmasq development for such a feature.

I did so, on the fourth of November 2020. Never got a reply.

 

[dave14305]

The new oisd format for dnsmasq 2.86 or later (https://dnsmasq2.oisd.nl/) seems to work OK to prevent HTTPS (Type65) queries.

Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 query[HTTPS] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 config api.mixpanel.com is NXDOMAIN
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 query[AAAA] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 config api.mixpanel.com is NXDOMAIN
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 query[A] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 config api.mixpanel.com is NXDOMAIN

local=/api.mixpanel.com/

 

[SomeWhereOverTheRainBow]

The new oisd format for dnsmasq 2.86 or later (https://dnsmasq2.oisd.nl/) seems to work OK to prevent HTTPS (Type65) queries.

Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 query[HTTPS] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 config api.mixpanel.com is NXDOMAIN
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 query[AAAA] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 config api.mixpanel.com is NXDOMAIN
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 query[A] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 config api.mixpanel.com is NXDOMAIN

local=/api.mixpanel.com/

I don't think that is the issue, I think the issue is it doesnt do this via a wild card option. Essentially, this means every address has to be added as such to block type-65. DNSMASQ will block any type of query given the domain is added to a blocklist or added in the fashion you have described. The challenged is, these users only specifically want to block type-65 requests.

e.g. of what they expect to happen...

Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 query[HTTPS] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 config api.mixpanel.com is NXDOMAIN
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 query[AAAA] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 config api.mixpanel.com is Someaddress
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 query[A] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 config api.mixpanel.com is Someaddress

This Reddit post might help you understand a bit more as well.

https://www.reddit.com/r/pihole/comments/ixwgrb

 

[Treadler]

The new oisd format for dnsmasq 2.86 or later (https://dnsmasq2.oisd.nl/) seems to work OK to prevent HTTPS (Type65) queries.

Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 query[HTTPS] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 config api.mixpanel.com is NXDOMAIN
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 query[AAAA] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 config api.mixpanel.com is NXDOMAIN
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 query[A] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 config api.mixpanel.com is NXDOMAIN

local=/api.mixpanel.com/

Interesting!
Is that list compatible with Diversion?

 

[Treadler]

I don't think that is the issue, I think the issue is it doesnt do this via a wild card option. Essentially, this means every address has to be added as such to block type-65. DNSMASQ will block any type of query given the domain is added to a blocklist or added in the fashion you have described. The challenged is, these users only specifically want to block type-65 requests.

e.g. of what they expect to happen...

Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 query[HTTPS] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1019 2601::8ee3/49749 config api.mixpanel.com is NXDOMAIN
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 query[AAAA] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1020 2601::8ee3/59012 config api.mixpanel.com is Someaddress
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 query[A] api.mixpanel.com from 2601::8ee3
Feb 18 20:13:06 dnsmasq[2085176]: 1021 2601::8ee3/58805 config api.mixpanel.com is Someaddress

This Reddit post might help you understand a bit more as well.

https://www.reddit.com/r/pihole/comments/ixwgrb

I’m happy if the only type 65 queries blocked, are the ones on my block list/s.
The rest, don’t care really.

 

[dave14305]

Interesting!
Is that list compatible with Diversion?

No, not today. But in theory you can download the list somewhere and add it to dnsmasq.conf.add as conf-file=/path/to/oisd_big_dnsmasq2.txt. But that’s off-topic for the Diversion thread.

I’m happy if the only type 65 queries blocked, are the ones on my block list/s.
The rest, don’t care really.

Exactly. I don’t see the use for blocking Type65 queries for domains you don’t want to block otherwise.

 

[SomeWhereOverTheRainBow]

yep in typical fashion, i would block domains the same way. But there seems to be some strange behavior with the new IOS devices and making local requests. That is why I regex block all type65. I have no need for my request on the IOS device to first travel https for a request for something that is answered locally.

 

[dave14305]

But there seems to be some strange behavior with the new IOS devices and making local requests

I have no need for my request on the IOS device to first travel https for a request for something that is local.

”Local” means what in this case? The LAN domain? That will already have a local=/home.lan/ config entry in dnsmasq.conf, so dnsmasq will not forward anything upstream for the domain. This is the same mechanism the oisd guy is using now.

 

[Treadler]

yep in typical fashion, i would block domains the same way. But there seems to be some strange behavior with the new IOS devices and making local requests. That is why I regex block all type65. I have no need for my request on the IOS device to first travel https for a request for something that is answered locally.

All Apple here; when type 65 blocked I got some odd behaviours.
App store sometimes unreachable, or, super slow.
My replies to emails forwarded to me would look & sound like they had gone, but nothing was ever actually sent.
Unblocking type 65 instantly fixed the above two issues.
Go figure……

 

[SomeWhereOverTheRainBow]

All Apple here; when type 65 blocked I got some odd behaviours.
App store sometimes unreachable, or, super slow.
My replies to emails forwarded to me would look & sound like they had gone, but nothing was ever actually sent.
Unblocking type 65 instantly fixed the above two issues.
Go figure……

well the firewall technique was breaking things from what my tests. the pihole method doesnt seem to. it actually resolves the issue for me.

 

[SomeWhereOverTheRainBow]

”Local” means what in this case? The LAN domain? That will already have a local=/home.lan/ config entry in dnsmasq.conf, so dnsmasq will not forward anything upstream for the domain. This is the same mechanism the oisd guy is using now.

Typically, what we would want is a locally defined response returned (basically where I have added my own cname records inside dnsmasq to control the query.)

here is what happens with a type65 response.

Diversion - Is it possible to block DNS query type

Sorry to bring this topic up again. I'm sure I have read it before but don't seems to have a way to block it. Do we have a way to block it now? Be it by firewall, unbound dns, diversion or any other means?

www.snbforums.com

The CNAME returned is coming from a different source than dnsmasq.