Diversion Diversion - the Router Ad-Blocker

[loveleeyoungae]

I suggested Diversion to my friend. As he's not techie, he wouldn't want to mess up his current working network (he likes the philosophy "If something is not broken, don't fix it"). However he has a spare AC66U, we think maybe we can make it work like a pi-hole?! His current network structure:

           ISP Wireless Router/PPPOE Modem (also DHCP/DNS/Gateway)
                 z                             ||
                z                              ||
               z                               ||
(Wireless broadcast to a                (WAN port of Aimesh Main Router)
Wireless Repeater)                  
Network 1                             Network 2 - Different subnet

I got on pi-hole forums, read some guides. And here are the steps I think we can do:
- Install Diversion to AC66U as normal (i.e. keep it running in router mode).
- Connect the ISP Router to LAN port of AC66U.
- Put the IP of AC66U into the Primary DNS Server field of the ISP Router' LAN Configuration.
- Then choose between these 2 methods:
Method 1: Disable ISP Router's DHCP function.
Method 2: Set the DHCP of ISP Router range to ONLY 1 IP which is for the AC66U (*)

So,
1. Do you think this kind of setup would work?
2. As I understand, on AC66U (still router mode), if we can disable DHCP but keep DNS function running, it will be exactly like a pi-hole. Is there any way to do this with config scripts? (I searched the forums, but didn't find discussions on this kind of modifying dnsmasq. Usually people just said disabling DHCP in webgui would mean disabling dnsmasq).

Spoiler: (*)

This usually requires you to disable DHCP on your router. But some models don’t even let you do this. The solution then, is to go into the DHCP settings of the router and see if you can limit its address range. By letting it serve only one IP address (i.e. the IP address of your Pi-hole). Then setup DHCP on the Pi-hole for everything other network device.
https://discourse.pi-hole.net/t/wha...manually-set-the-dns-server-on-my-router/8928

 

[gattaca]

During the install of Diversion Standard, the pixelserv-tls certs are auto-generated by Diversion if not found. Since I imported the certificate long ago into every device, I reuse the same certs on every router. After a reinstall I simply replace the newly generated certs with my backup. This way, I don't have to reimport it.

Hi! Thank you for coordinating and integrating Diversion + pixelserv utilities! The amtm script also really opens this tooling to many more people in the Asus/Merlin community. I've read the main threads for amtm, Diversion, pixelserv, Entware, dnscrypt and skynet. Several of these I'm still digesting. I felt I had a good handle on Diversion + Pixelserv until I read the ca.crt discussion and realized I too had not imported the ca.crt generated by Diversion for pixelserv as part of the Diversion install via amtm.

Given the recent questions re: ca.crt, it would be super if amtm reminded users to import the auto-generated ca.crt via http://pixel-ip/ca.crt as part of the menu. This is an excellent reference per this thread -> https://github.com/kvic-z/pixelserv-tls/wiki/Create-and-Import-the-CA-Certificate Most people simply do not know how these certificates really work, nor did I realize I needed to import it on each machine to to get the best performance from the entire setup.

The entire ASUS/Merlin community here has done *@(*@ SUPER job! Most of this stuff is far too technical or has way too many critical steps for all but the most technical people to execute without this tooling. I've been deep in IT for a long time and I'd not be comfortable doing everything amtm does manually! Many of us know enough linux commands to be dangerous but not totally reliable. Thanks for your diligence and patience! Have a great day! Later.

 

[dave14305]

Looks like you and @Quoc Huynh found a bug!
Will be fixed in the coming update.

Will the stats logic also be modified to avoid false positive reporting of blocked domains (e.g outlook.office365.com, www.apple.com)? The legitimate sites aren’t being blocked but the stats reports list them as blocked since the dots in the name are not escaped in the grep.
Thanks!

 

[apathy]

I was wondering if anyone could give me some insight on my issue. So I have around 13 lists in my blocking file totaling around 1.04million domains being blocked. I've noticed some sites are slow to load for the first time, but it really doesn't bother me. My issue is that like once a week when I log into the UI of the router, right side of the page fails to load and becomes unresponsive. If I putty in to my router and disable Diversion and Re-enable it, everything comes back. I have an 8GB usb installed with the largest swap file you can. I run Diversion, PixelServ, DNSCrypt and Skynet. Diversion seems to be the source of the problem. If you require any additional information to help me, please let me know. I'm tired of needing to restart Diversion. I never had to do this with my Pi-Hole ( from raspberry pi ) before I swapped to Diversion. I'd really prefer to stay with Diversion + PixelServ but this is really annoying. Any help is greatly appreciated.

Edit: Just in-case this may help determine the issue, I noticed that Diversion only uses about 20% of my RAM and the router stays around 50% most of the time, so I don't think that is the issue. Also, I haven't done the PixelServ Certificate import on all devices yet, but I feel like that shouldn't be the source of this issue either.

Edit 2: By the way @thelonelycoder you're awesome for making this stuff and I appreciate any support you may be able to provide.

 

[Mutzli]

I was wondering if anyone could give me some insight on my issue. So I have around 13 lists in my blocking file totaling around 1.04million domains being blocked. I've noticed some sites are slow to load for the first time, but it really doesn't bother me. My issue is that like once a week when I log into the UI of the router, right side of the page fails to load and becomes unresponsive. If I putty in to my router and disable Diversion and Re-enable it, everything comes back. I have an 8GB usb installed with the largest swap file you can. I run Diversion, PixelServ, DNSCrypt and Skynet.

Do you also run DNSmasq? If so check what size the cache is set at.

 

[john9527]

So I have around 13 lists in my blocking file totaling around 1.04million domains being blocked.

There's been another report of problems when you exceed 1M domains. I'd try shrinking things a bit and see if the problems persist.

 

[Goobi]

I noticed that in the weekly diversion stats I got an entry for wpad.home. .home is my domain but wpad is not a host I recognize so I ran a nslookup against it and it shows my pixelserv address. Anyone using pixelserv with diversion seen this host name? That IP is not even in my dhcp range so wondering if that name is hardcoded. Also wondering if that needs to be blocked or allowed? It had 1340 requests in the last week. Thanks in advance.

 

[eclp]

I use diversion and skynet and lately with my ram always almost completely full. Is this condition normal? I created a swap file during installation.

 

[Protik]

I use diversion and skynet and lately with my ram always almost completely full. Is this condition normal? I created a swap file during installation.

View attachment 15483

Yes it is. The blocklist is loaded in the RAM. As you have created a swap, you should be fine. If the OS needs RAM, it will dump some stuff to the swap and free up some space.

Sent from my Moto G (5) Plus using Tapatalk

 

[preacher65]

After a reinstall I simply replace the newly generated certs with my backup. This way, I don't have to reimport it.

This is a useful tip, thank you - I'm looking in /opt/var/cache/pixelserv, is that the correct path? I would assume I need the ca key as well as the cert but would you mind confirming? Thanks.

 

[user1085]

Is it possible to add a feature to follow logs from a specific source/device IP? For eg. I want to tail requests from my Roku only to discover more domains to block ads in apps like HGTV

 

[thelonelycoder]

If you do already or want to follow me on Twitter for Diversion news, there's a change
As I slowly merge everything over from the AB-Solution to the Diversion website, so do my announcements for Diversion news on Twitter.
The old handle @ab_solution will phase out and @DiversionBlock takes over for those of you wanting to stay up to date with what's happening on the Diversion front.

 

[HawkInOz]

I noticed that in the weekly diversion stats I got an entry for wpad.home. .home is my domain but wpad is not a host I recognize so I ran a nslookup against it and it shows my pixelserv address. Anyone using pixelserv with diversion seen this host name? That IP is not even in my dhcp range so wondering if that name is hardcoded. Also wondering if that needs to be blocked or allowed? It had 1340 requests in the last week. Thanks in advance.

I was wondering the same thing, but I'm not using pixelserv with Diversion at this time. For me, nslookup shows it pointing to my router's IP (gateway/DNS/WINS) address. I had 12068 requests last week, making #5 on my Top 25 list...

 

[thelonelycoder]

Will the stats logic also be modified to avoid false positive reporting of blocked domains (e.g outlook.office365.com, www.apple.com)? The legitimate sites aren’t being blocked but the stats reports list them as blocked since the dots in the name are not escaped in the grep.
Thanks!

Thanks for reminding me . Will be fixed in the coming Diversion 4.0.6 release.

 

[thelonelycoder]

I noticed that in the weekly diversion stats I got an entry for wpad.home. .home is my domain but wpad is not a host I recognize so I ran a nslookup against it and it shows my pixelserv address. Anyone using pixelserv with diversion seen this host name? That IP is not even in my dhcp range so wondering if that name is hardcoded. Also wondering if that needs to be blocked or allowed? It had 1340 requests in the last week. Thanks in advance.

wpad is part of the local DHCP and DNS discovery methods. Do not block.
https://en.wikipedia.org/wiki/Web_Proxy_Auto-Discovery_Protocol

 

[thelonelycoder]

I was wondering if anyone could give me some insight on my issue. So I have around 13 lists in my blocking file totaling around 1.04million domains being blocked. I've noticed some sites are slow to load for the first time, but it really doesn't bother me. My issue is that like once a week when I log into the UI of the router, right side of the page fails to load and becomes unresponsive. If I putty in to my router and disable Diversion and Re-enable it, everything comes back. I have an 8GB usb installed with the largest swap file you can. I run Diversion, PixelServ, DNSCrypt and Skynet. Diversion seems to be the source of the problem. If you require any additional information to help me, please let me know. I'm tired of needing to restart Diversion. I never had to do this with my Pi-Hole ( from raspberry pi ) before I swapped to Diversion. I'd really prefer to stay with Diversion + PixelServ but this is really annoying. Any help is greatly appreciated.

Edit: Just in-case this may help determine the issue, I noticed that Diversion only uses about 20% of my RAM and the router stays around 50% most of the time, so I don't think that is the issue. Also, I haven't done the PixelServ Certificate import on all devices yet, but I feel like that shouldn't be the source of this issue either.

Edit 2: By the way @thelonelycoder you're awesome for making this stuff and I appreciate any support you may be able to provide.

Check https://github.com/StevenBlack/hosts/blob/master/readme.md, pgl.yoyo and someonewhocares.org are already included in that list. No need to add it twice, removing the duplicates costs precious CPU cycles and uses even more of the scarce memory while processing.

 

[thelonelycoder]

This is a useful tip, thank you - I'm looking in /opt/var/cache/pixelserv, is that the correct path? I would assume I need the ca key as well as the cert but would you mind confirming? Thanks.

Both, the ca.crt and ca.key need to be replaced in /opt/var/cache/pixelserv, then run ep '3. Purge generated pixelserv-tls certificates' to remove any existing new certs generated by pixelserv-tls with the newly generated ca pair. In the process pixelserv-tls is auto-restarted to use the copied certs.

@all Diversion users:
A wild thought just crossed my mind. What if Diversion does NOT create a new set of ca certs during a fresh install but downloads already (self-) generated certs from the Diversion website. This way all Diversion installations would use the same pixelserv-tls certificate, making subsequent importing of the cert into devices a once-only affair and one does not have to worry about it for future re-installs.

The openssl certs auto-generated by Diversion during the install are good for 10 years, the code used is as follows:

openssl genrsa -out ca.key 1024
openssl req -key ca.key -new -x509 -days 3650 -sha256 -extensions v3_ca -out ca.crt -subj "/CN=Pixelserv CA"

The same code would be used for the common Diversion certificates, with the CA being Pixelserv.
Of course this would mean Diversion users would have to trust me even more not doing anything sketchy on your routers and I would have to make sure the certs on my server are protected from being replaced by a third party.

But this could be a useful feature and end up being more user friendly with clear instructions on the Diversion website of how to import the certificate into browsers/devices. This step would be an opt-in during the installation with the selection in wording being similar to this:
1. Use common Diversion pixelserv-tls certificates
2. Generate new router specific pixelserv-tls certificates

Let me know what you all think about this random thought, maybe @kvic or anyone understanding more than I could add his thoughts on the implications of such a Diversion feature. I can't see any just now as generally, TLS/SSL certificates are issued by a certificate authority (https://en.wikipedia.org/wiki/Certificate_authority) and the same cert is used world wide by all devices/browsers.

Edit: Ignore above random thought.

 

[thelonelycoder]

I was wondering if anyone could give me some insight on my issue. So I have around 13 lists in my blocking file totaling around 1.04million domains being blocked. I've noticed some sites are slow to load for the first time, but it really doesn't bother me. My issue is that like once a week when I log into the UI of the router, right side of the page fails to load and becomes unresponsive. If I putty in to my router and disable Diversion and Re-enable it, everything comes back.

As @john9527 writes, use a smaller blocking file to start troubleshoot this issue.

 

[thelonelycoder]

Is it possible to add a feature to follow logs from a specific source/device IP? For eg. I want to tail requests from my Roku only to discover more domains to block ads in apps like HGTV

This is already built in. Use f '4. Filtered by term' and enter the IP address of the device to filter by.

 

[thelonelycoder]

I use diversion and skynet and lately with my ram always almost completely full. Is this condition normal? I created a swap file during installation.

That's perfectly normal. My main router with 512 MB RAM and Skynet/Diversion and VPN Server running runs at about 85%. About 12% of that are used by Dnsmasq.