Diversion Diversion - the Router Ad-Blocker

[thelonelycoder]

But when I click on a safelinks URL (e.g.
https://nam01.safelinks.protection....qofj/PZ5n+ZAee5AEYrqq7p/X6VRNGpBs=&reserved=0 which is redirected to the Android app store) I get redirected to the link originally obfuscated by Safelinks instead of getting the usual error screen in the browser.

Clear your browser and OS cache.

I looked at the log, and I see:

Nov 26 16:45:11 dnsmasq[1240]: query[A] nam01.safelinks.protection.outlook.com from 192.168.1.66
Nov 26 16:45:11 dnsmasq[1240]: config nam01.safelinks.protection.outlook.com is 192.168.1.2

instead of the usual

Nov 26 16:45:21 dnsmasq[1240]: query[A] www.googleadservices.com from 192.168.1.66
Nov 26 16:45:21 dnsmasq[1240]: /opt/share/diversion/list/blockinglist www.googleadservices.com is 192.168.1.2

The wildcard-blacklist works differently than the blacklist or blockingfile.
wc_blacklisted entries get added to the dnsmasq.conf file directly. You can look this file up in sf.

What's also weird is that Diversion is installed on my 192.168.1.1 router, but somehow Diversion seems to redirect to 192.168.1.2 which does not exist on my network currently (usually it's another router)

192.168.1.2 is your blocking IP, or the IP pixelserv-tls is listening on.

16:52:14 blocked by blockinglist teredo.ipv6.microsoft.com
16:52:18 blocked by blockinglist mobile.pipe.aria.microsoft.com
16:52:32 blocked by blockinglist ads.nexage.com
16:52:38 blocked by wc-blacklist nam01.safelinks.protection.outlook.com

Diversion is about simplifying things. That includes rewriting output of the Dnsmasq log file when followed in f. Not everyone understands when Dnsmasq says "config nam01.safelinks.protection.outlook.com is 192.168.1.2". So I rewrite this to "blocked by wc-blacklist" or similar, depending on which follow option you select.

 

[Jest]

Might add the firmware being run is of the "not supported" flavour

I thought merlin's forks are supported.

Yeah, i did fresh install like thelonelycoder suggested and it's working now, installed without errors.

Thanks.

 

[thelonelycoder]

I thought merlin's forks are supported.

They are, for legal reasons some of these forks may not be discussed on this forum, that includes yours.
But AFAIK Diversion runs just fine on them. Its just that I don’t test it explicitly. I develop and test Diversion on routers with official Asuswrt-Merlin and @john9527’s fork firmware.

 

[Kingp1n]

Dumb question to the experts here...I've searched thru the threads here and found how to whitelist certain domains for the amazon app for android phones. I've done all instructions found here and whitelisting numerous domains and 85% of the time I still get the error message within the amazon app when a page can't be found...I believe from reading here that it may be due to pixelserver...is this correct? I have it enabled currently but I wanted to get input from the field???? Any info is greatly appreciated.

 

[Jest]

They are, for legal reasons some of these forks may not be discussed on this forum, that includes yours.
But AFAIK Diversion runs just fine on them. Its just that I don’t test it explicitly. I develop and test Diversion on routers with official Asuswrt-Merlin and @john9527’s fork firmware.

I understand, but for N18U there is no other alternative unfortunately.

Anyway, thanks for your help.

 

[dave14305]

Dumb question to the experts here...I've searched thru the threads here and found how to whitelist certain domains for the amazon app for android phones. I've done all instructions found here and whitelisting numerous domains and 85% of the time I still get the error message within the amazon app when a page can't be found...I believe from reading here that it may be due to pixelserver...is this correct? I have it enabled currently but I wanted to get input from the field???? Any info is greatly appreciated.

Pixelserv will only interfere with requests that Diversion redirects to the Pixelserv IP. So I would guess your issue still exists in watching the dnsmasq log in Diversion while you try to use the Amazon app, and whitelisting any new Amazon related hostnames you see.

 

[Bilibox]

So I'm unsure if this is a limitation, bug, or something else. But I have noticed that once you hit 1 million domains in your blocklist the router CPU maxes out which causes Skynet/Diversion and the webui client for asus to disconnect from the internet. This stops skynet and diversion from being able to update. I'm on the latest version of everything my router is AC87U with Merlin 384.7_2 and a 500mb Swap. With 900 thousand it works just fine with about 10% usage spikes so I wouldn't think it has to do with the size...

 

[thelonelycoder]

So I'm unsure if this is a limitation, bug, or something else. But I have noticed that once you hit 1 million domains in your blocklist the router CPU maxes out which causes Skynet/Diversion and the webui client for asus to disconnect from the internet. This stops skynet and diversion from being able to update. I'm on the latest version of everything my router is AC87U with Merlin 384.7_2 and a 500mb Swap. With 900 thousand it works just fine with about 10% usage spikes so I wouldn't think it has to do with the size...

Looks more like your router runs out of memory. Diversion usually has very low CPU usage except when the blocking file is updated.
Your router running out of memory would cause Dnsmasq to stop or crash, whitch means no internet.
Diversion has no limitation of the blocking file size nor should there be a bug.

 

[john9527]

Diversion has no limitation of the blocking file size nor should there be a bug.

There was a user on my fork that was using a really big custom blocking file (IIRC it was approaching 2M entries) and it would crash dnsmasq. I also remember when I was researching it, I found some old posts that seemed to indicate that it was originally thought there would be only about 5000 domains entered in dnsmasq in a 'large' install. So it's entirely possible that dnsmasq is being stretched beyond it's limits.

 

[Budiman]

Hi, thanks for the great script @thelonelycoder. I just successfully get Diversion running on my Asus RT-AC68U and tested working with some ad blocker checking sites). I'm so happy that I can get rid of ads on the sites I frequently visit. But I'm still seeing in-app ads in Android (free games that just pop ads on every new game).
Are those supposed to be blocked too by Diversion? Thanks before.

 

[Kronyx]

I just installed it on my AC3200 and it work very well ! Thanks !!

 

[iManuB]

I am waiting for the final 384.8 version to install everything again. I had read some time ago of a suitable SWAP size for my AC-86U, but I can not find the discussion.

When I install everything, what size do you recommend?

Thanks so much!

 

[Bilibox]

Looks more like your router runs out of memory. Diversion usually has very low CPU usage except when the blocking file is updated.
Your router running out of memory would cause Dnsmasq to stop or crash, whitch means no internet.
Diversion has no limitation of the blocking file size nor should there be a bug.

There was a user on my fork that was using a really big custom blocking file (IIRC it was approaching 2M entries) and it would crash dnsmasq. I also remember when I was researching it, I found some old posts that seemed to indicate that it was originally thought there would be only about 5000 domains entered in dnsmasq in a 'large' install. So it's entirely possible that dnsmasq is being stretched beyond it's limits.

Shouldn't the Swap file take care of that? (I also have enough hosts to have 2mill) and with all of them im only showing 100mb used on the swap. I'm thinking it may be what John wrote, dnsmasq might just be unable to handle 1mill+ since 980k enteries works just fine...

 

[skeal]

I am waiting for the final 384.8 version to install everything again. I had read some time ago of a suitable SWAP size for my AC-86U, but I can not find the discussion.

When I install everything, what size do you recommend?

Thanks so much!

There is no need for any more than the size of your ram. In my case its 512mb so I use that size swap.

 

[iManuB]

There is no need for any more than the size of your ram. In my case its 512mb so I use that size swap.

Thanks a lot. You're always very kind. Also the AC86U router has 512MB, so I will also put 512 of Swap.

 

[thelonelycoder]

There was a user on my fork that was using a really big custom blocking file (IIRC it was approaching 2M entries) and it would crash dnsmasq. I also remember when I was researching it, I found some old posts that seemed to indicate that it was originally thought there would be only about 5000 domains entered in dnsmasq in a 'large' install. So it's entirely possible that dnsmasq is being stretched beyond it's limits.

I know, this is stretching the boundaries. It's not that AB-Solution or now Diversion does something new. Hosts based ad-blocking has been around for a while. Simon Kelley did a really good job with the caching mechanism in Dnsmasq.
Just did a test to see how far I could go. I stopped at a 4 million hosts/125MB file, with no crash. This is on the RT-AC1900P with 384.8_beta2 firmware.
While I don't recommend files lager than 1 million hosts, it never crashed Dnsmasq while testing during the early development stages of Diversion.

Shouldn't the Swap file take care of that? (I also have enough hosts to have 2mill) and with all of them im only showing 100mb used on the swap. I'm thinking it may be what John wrote, dnsmasq might just be unable to handle 1mill+ since 980k enteries works just fine...

It should and does, but this may be Dnsmasq related. What firmware version are you using?

Edit: You are using an AC87U with Merlin 384.7_2 and a 500mb Swap file.
That's a recent Dnsmasq version and a router that should be capable to handle the load.
Maybe you have a slow USB device? Dnsmasq reads in the blocking file and there's a time limit how long this might take.

 

[thelonelycoder]

But I'm still seeing in-app ads in Android (free games that just pop ads on every new game).
Are those supposed to be blocked too by Diversion? Thanks before.

Maybe, that depends on your blocking file type and whether it includes the necessary domain that needs to be blocked. Or they serve ads from the same domain as the content. In that case blocking is not possible.
Use f to follow the logfile and see if you can catch the domain to be blocked while using the app in Android.

 

[skeal]

I been using Diversion with DNSCrypt & Skynet for some time now on my Asus RT-AC86U router & very happy with it. Just recently added my old Asus RT-AC56U as a wired AP to my main router. Do I need to install Diversion with DNSCrypt & Skynet on the old router or it's not needed?

Not needed. Everything uses the path through the router.

 

[Kronyx]

Finally after 1 day of use, i see a lot of adds in Facebook and also in my gmail, i did not have any with Addblock on chrome

 

[robca]

That would usually be the blocking IP assigned to Pixelserv-tls if you installed Diversion Standard.

I have both installed and tested your wc blacklist:

20:19:25 dnsmasq[2387]: query[A] nam01.safelinks.protection.outlook.com from 192.168.1.161
20:19:25 dnsmasq[2387]: blocked by wc-blacklist nam01.safelinks.protection.outlook.com is 192.168.1.2

Got a page that the site could not be found, as expected.

It also matters in terms of output formatting if you follow the dnsmasq log from within Diversion or tail the raw file in /opt/var/log/dnsmasq.log.

Thanks again for taking the time to look into my issue.

I'm still confused, though. I redid all of the above (i.e. cleared the wc-blacklist, entered .safelinks.protection.outlook.com as the only entry, and clicked on the https://nam01.safelinks.protection....qofj/PZ5n+ZAee5AEYrqq7p/X6VRNGpBs=&reserved=0 URL).

The log (from the Diversion "follow dnsmasq.log" option) shows, correctly

16:38:33 dnsmasq[7443]: query[A] nam01.safelinks.protection.outlook.com from 192.168.1.66
16:38:33 dnsmasq[7443]: blocked by wc-blacklist nam01.safelinks.protection.outlook.com is 192.168.1.2

And my browser shows the certificate error, as expected. What's weird, though, is that if I click on "continue to this website", it actually shows the Play store link... which means that my browser still managed to connect to the obfuscated Safelinks URL and got redirected to the Play store by the Microsoft redirection. I was expecting the usual "blank page", the one served by pixelserv, instead

What am I missing?