Diversion Diversion - the Router Ad-Blocker

[Skeptical.me]

FYI:
https://adaway.org/hosts.txt and http://www.malwaredomainlist.com/hostslist/hosts.txt are included in https://github.com/StevenBlack/hosts which is used in the Standard, Medium and Large blocking files.

https://hosts-file.net/ad_servers.txt is used in the Large blocking file.

I figured out the domains must have been already included as the overall Blocked Domains hardly increased. I'm just doing this to explore how to use Diversion tbh

 

[loveleeyoungae]

When I tried out AdGuard Home, I saw these in Terminal (there may be more but these are the ones I caught). This is the first time I've added Host Files to Diversion. I wasn't sure if others would be interested in these:

https://adaway.org/hosts.txt

https://hosts-file.net/ad_servers.txt

http://www.malwaredomainlist.com/hostslist/hosts.txt

The hpHosts (hosts-file.net) authors seem to be irresponsible. I reported a false positive in their files (thelonelycoder confirmed) last month, they promised to have a look at it, but yet they haven't done anything.
I searched around and saw some complains regarding other false positives in their files.
My advice: avoid those that want to make a big all-in-one blocking list without careful check.

 

[Skeptical.me]

The hpHosts (hosts-file.net) authors seem to be irresponsible. I reported a false positive in their files (thelonelycoder confirmed) last month, they promised to have a look at it, but yet they haven't done anything.
I searched around and saw some complains regarding other false positives in their files.
My advice: avoid those that want to make a big all-in-one blocking list without careful check.

Thanks for letting me know I'll delete it.

 

[gattaca]

About time to complete the documentation on the website, huh?
Total ads blocked, blocked this week since last blocking file update, newly counted ads since last count. Total, Week, New.

TY.. I had been wondering for a long time too. Figured as long as the numbers kept increasing all was good.

 

[Bilibox]

Update since my last post! I decided to take some time today and look at the blocking file. I noticed duplicates inside the file so figured I would do some testing. I combined a bunch of host sources together, and removed all the duplicates. I figured it wouldn't hurt to test it out so i loaded it into diversion (I forgot to turn off the additional hosts (plus) on accident) right now I'm at 3.054 Million blocked domains with "minimal" latency (I obviously expect there to be some performance impact). I still face the issue where the router is @ high CPU usage but it doesn't seem to bother anything as of right now. So then comes the question. Since I've found out that it can handle 3 million without issues... What could have been the issues before? I'm not sure if it is just liking having one big Host file (6 with all the host plus ones) that is cleaned up and sorted already?

This is the file I put together for reference https://github.com/Bilibox/Host (Note: This is without any editing to make sure everything works perfectly! I use my own whitelist in diversion for that)

Any insight would be wonderful!

 

[thelonelycoder]

Update since my last post! I decided to take some time today and look at the blocking file. I noticed duplicates inside the file so figured I would do some testing. I combined a bunch of host sources together, and removed all the duplicates. I figured it wouldn't hurt to test it out so i loaded it into diversion (I forgot to turn off the additional hosts (plus) on accident) right now I'm at 3.054 Million blocked domains with "minimal" latency (I obviously expect there to be some performance impact). I still face the issue where the router is @ high CPU usage but it doesn't seem to bother anything as of right now. So then comes the question. Since I've found out that it can handle 3 million without issues... What could have been the issues before? I'm not sure if it is just liking having one big Host file (6 with all the host plus ones) that is cleaned up and sorted already?

This is the file I put together for reference https://github.com/Bilibox/Host (Note: This is without any editing to make sure everything works perfectly! I use my own whitelist in diversion for that)

Any insight would be wonderful!

I doubt that the sort -u leaves doubles in the blocking file.
And, instead of blocking 4 million websites, why not allow the remaining 2 million websites, much simpler .

 

[Bilibox]

I doubt that the sort -u leaves doubles in the blocking file.
And, instead of blocking 4 million websites, why not allow the remaining 2 million websites, much simpler .

Taking care of that right now ! Just noticed it haha Also taking out dead domains.

Edit: Just got the block file to 1.8 w/ ipv6. Then my issue where it stops working showed up again as well. I had pulled up dnsmasq log and it said "read blocked by blocking file" as well as "read blocked by blacklist" Could that just be from it timing out? I would've posted the actual log but had to restart and lost it

EDIT: LOL I found out one of the lists I used had dns.msftncsi.com in it. Once removed router is back to normal with low CPU usage.

 

[GGavan]

Hi, I updated from AB-Solution (which worked perfect for long time until now) to Diversion Standard 4.0.5 which seems to work well too in some conditions...
The problem is at the restart of the router, it is lost a part of the application. "diversion" command is not recognized and the adds are back on browsers and applications.
It is working to run the install command (recognizing the previous files) and all is working until next restart.
I have ASUS AC87U with HGG 280.65.9 as FW
Any idea how to solve this?
I'm sorry if it was asked before... 79 pages is a lot to read just to check if it was asked/solved.
Thx in advance!

 

[thelonelycoder]

Hi, I updated from AB-Solution (which worked perfect for long time until now) to Diversion Standard 4.0.5 which seems to work well too in some conditions...
The problem is at the restart of the router, it is lost a part of the application. "diversion" command is not recognized and the adds are back on browsers and applications.
It is working to run the install command (recognizing the previous files) and all is working until next restart.
I have ASUS AC87U with () 280.65.9 as FW
Any idea how to solve this?
I'm sorry if it was asked before... 79 pages is a lot to read just to check if it was asked/solved.
Thx in advance!

Post the content of /jffs/scripts/post-mount. Use sf in Diversion to show the file.

 

[GGavan]

Hi, this is what is listed.

/jffs/scripts/post-mount has this content:

START FILE, --- lines are not part of file
---------------------------------------------------
#!/bin/sh


# mount media
if [ "$1" = "/tmp/mnt/ASUS" ];then
mount -t cifs //192.168.0.5/Public -o username=GGavan,password=xxx /mnt/ASUS/MEDIA
logger "Media share mounted"
fi
# mount end. /jffs/scripts/post-mount.div # Added by Diversion
---------------------------------------------------
END FILE

 

[thelonelycoder]

Hi, this is what is listed.

/jffs/scripts/post-mount has this content:

START FILE, --- lines are not part of file
---------------------------------------------------
#!/bin/sh


# mount media
if [ "$1" = "/tmp/mnt/ASUS" ];then
mount -t cifs //192.168.0.5/Public -o username=GGavan,password=xxx /mnt/ASUS/MEDIA
logger "Media share mounted"
fi
# mount end. /jffs/scripts/post-mount.div # Added by Diversion
---------------------------------------------------
END FILE

Make the last line two lines, like so:

 # mount end
. /jffs/scripts/post-mount.div # Added by Diversion

 

[GGavan]

It worked! Thank you!
Great tool, I’m using it from a long time.

 

[Ubimo]

Hello
I found these entries in syslog, never seen "avahi-deamon" before. Is this part of diversion, pixelserv-tls?

Dec 26 14:24:00 amtm: Disk check done on /dev/sda1
Dec 26 14:24:00 rc_service: ntp 421:notify_rc restart_diskmon
Dec 26 14:24:00 disk_monitor: Finish
Dec 26 14:24:01 kernel: EXT2-fs (sda1): warning: mounting unchecked fs, running e2fsck is recommended
Dec 26 14:24:01 syslog: USB ext2 fs at /dev/sda1 mounted on /tmp/mnt/USB
Dec 26 14:24:01 usb: USB ext2 fs at /dev/sda1 mounted on /tmp/mnt/USB.
Dec 26 14:24:02 pppd[395]: System time change detected.
Dec 26 14:24:02 disk_monitor: be idle
Dec 26 14:24:03 custom_script: Running /jffs/scripts/post-mount (args: /tmp/mnt/USB ) - max timeout = 120s
Dec 26 14:24:03 Skynet: [%] Startup Initiated... ( skynetloc=/tmp/mnt/USB/skynet )
Dec 26 14:24:03 avahi-daemon[271]: Registering new address record for 192.168.1.2 on br0.IPv4.
Dec 26 14:24:03 Diversion: created br0:pixelserv-tls 192.168.1.2, from /opt/etc/init.d/S80pixelserv-tls
Dec 26 14:24:03 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 3
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 3
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 3
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 3
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 2
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 2
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 2
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 2
Dec 26 14:24:04 Entware (armv7sf-k2.6): Started pixelserv-tls (Diversion) from /jffs/scripts/post-mount
Dec 26 14:24:04 Diversion: started Entware services, from /jffs/scripts/post-mount
Dec 26 14:24:04 rc_service: service 741:notify_rc restart_dnsmasq
Dec 26 14:24:04 dnsmasq[239]: exiting on receipt of SIGTERM
Dec 26 14:24:04 custom_script: Running /jffs/scripts/dnsmasq.postconf (args: /etc/dnsmasq.conf ) - max timeout = 120s
Dec 26 14:24:05 pixelserv-tls[809]: pixelserv-tls 2.1.2 (compiled: Sep  8 2018 20:33:38) options: 192.168.1.2
Dec 26 14:24:08 rc_service: hotplug 508:notify_rc restart_nasapps
Dec 26 14:24:08 rc_service: waitting "restart_dnsmasq" via  ...
Dec 26 14:24:08 kernel: Adding 262140k swap on /tmp/mnt/USB/myswap.swp.  Priority:-1 extents:69 across:267404k
Dec 26 14:24:08 pixelserv-tls[809]: Listening on :192.168.1.2:443
Dec 26 14:24:08 pixelserv-tls[809]: Listening on :192.168.1.2:80
Dec 26 14:24:10 Diversion: restarted Dnsmasq to apply settings, from /jffs/scripts/dnsmasq.postconf
Dec 26 14:24:11 avahi-daemon[271]: Files changed, reloading.
Dec 26 14:24:11 avahi-daemon[271]: Loading service file /tmp/avahi/services/mt-daap.service.
Dec 26 14:24:11 iTunes: daemon is stopped
Dec 26 14:24:11 FTP_Server: daemon is stopped
Dec 26 14:24:11 Samba_Server: smb daemon is stopped
Dec 26 14:24:11 kernel: gro disabled
Dec 26 14:24:11 avahi-daemon[271]: Got SIGTERM, quitting.
Dec 26 14:24:11 avahi-daemon[271]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.1.
Dec 26 14:24:11 avahi-daemon[271]: Leaving mDNS multicast group on interface lo.IPv4 with address 127.0.0.1.
Dec 26 14:24:12 avahi-daemon[271]: avahi-daemon 0.7 exiting.
Dec 26 14:24:12 Timemachine: daemon is stopped
Dec 26 14:24:12 kernel: gro enabled with interval 2
Dec 26 14:24:12 avahi-daemon[924]: Found user 'nobody' (UID 65534) and group 'nobody' (GID 65534).
Dec 26 14:24:12 avahi-daemon[924]: Successfully dropped root privileges.
Dec 26 14:24:12 avahi-daemon[924]: avahi-daemon 0.7 starting up.
Dec 26 14:24:12 avahi-daemon[924]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Dec 26 14:24:12 avahi-daemon[924]: No service file found in /tmp/avahi/services.
Dec 26 14:24:12 avahi-daemon[924]: Loading new alias name R7000.
Dec 26 14:24:12 avahi-daemon[924]: Joining mDNS multicast group on interface br0.IPv4 with address 192.168.1.2.
Dec 26 14:24:12 avahi-daemon[924]: New relevant interface br0.IPv4 for mDNS.
Dec 26 14:24:12 avahi-daemon[924]: Joining mDNS multicast group on interface lo.IPv4 with address 127.0.0.1.
Dec 26 14:24:12 avahi-daemon[924]: New relevant interface lo.IPv4 for mDNS.
Dec 26 14:24:12 avahi-daemon[924]: Network interface enumeration completed.
Dec 26 14:24:12 avahi-daemon[924]: Registering new address record for 192.168.1.2 on br0.IPv4.
Dec 26 14:24:12 avahi-daemon[924]: Registering new address record for 192.168.1.1 on br0.IPv4.
Dec 26 14:24:12 avahi-daemon[924]: Registering new address record for 127.0.0.1 on lo.IPv4.
Dec 26 14:24:13 avahi-daemon[924]: Server startup complete. Host name is R7000-4485.local. Local service cookie is 1506266674.
Dec 26 14:24:13 avahi-daemon[924]: Alias name "R7000" successfully established.
Dec 26 14:24:13 Samba_Server: daemon is started
Dec 26 14:24:14 rc_service: zcip 937:notify_rc start_firewall
Dec 26 14:24:14 zcip_client: configured 169.254.23.173
Dec 26 14:24:15 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
Dec 26 14:24:16 custom_script: Running /jffs/scripts/firewall-start (args: ppp0)
Dec 26 14:24:35 kernel: ip_set: protocol 6

 

[thelonelycoder]

Hello
I found these entries in syslog, never seen "avahi-deamon" before. Is this part of diversion, pixelserv-tls?

Nothing to do with Diversion:
https://www.snbforums.com/threads/what-is-the-avahi-daemon-for.37760/

 

[octopus]

Hello
I found these entries in syslog, never seen "avahi-deamon" before. Is this part of diversion, pixelserv-tls?

Dec 26 14:24:00 amtm: Disk check done on /dev/sda1
Dec 26 14:24:00 rc_service: ntp 421:notify_rc restart_diskmon
Dec 26 14:24:00 disk_monitor: Finish
Dec 26 14:24:01 kernel: EXT2-fs (sda1): warning: mounting unchecked fs, running e2fsck is recommended
Dec 26 14:24:01 syslog: USB ext2 fs at /dev/sda1 mounted on /tmp/mnt/USB
Dec 26 14:24:01 usb: USB ext2 fs at /dev/sda1 mounted on /tmp/mnt/USB.
Dec 26 14:24:02 pppd[395]: System time change detected.
Dec 26 14:24:02 disk_monitor: be idle
Dec 26 14:24:03 custom_script: Running /jffs/scripts/post-mount (args: /tmp/mnt/USB ) - max timeout = 120s
Dec 26 14:24:03 Skynet: [%] Startup Initiated... ( skynetloc=/tmp/mnt/USB/skynet )
Dec 26 14:24:03 avahi-daemon[271]: Registering new address record for 192.168.1.2 on br0.IPv4.
Dec 26 14:24:03 Diversion: created br0:pixelserv-tls 192.168.1.2, from /opt/etc/init.d/S80pixelserv-tls
Dec 26 14:24:03 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 3
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 3
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 3
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 3
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 2
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 2
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 2
Dec 26 14:24:04 lldpd[306]: custom TLV op replace oui f8:32:e4 subtype 2
Dec 26 14:24:04 Entware (armv7sf-k2.6): Started pixelserv-tls (Diversion) from /jffs/scripts/post-mount
Dec 26 14:24:04 Diversion: started Entware services, from /jffs/scripts/post-mount
Dec 26 14:24:04 rc_service: service 741:notify_rc restart_dnsmasq
Dec 26 14:24:04 dnsmasq[239]: exiting on receipt of SIGTERM
Dec 26 14:24:04 custom_script: Running /jffs/scripts/dnsmasq.postconf (args: /etc/dnsmasq.conf ) - max timeout = 120s
Dec 26 14:24:05 pixelserv-tls[809]: pixelserv-tls 2.1.2 (compiled: Sep  8 2018 20:33:38) options: 192.168.1.2
Dec 26 14:24:08 rc_service: hotplug 508:notify_rc restart_nasapps
Dec 26 14:24:08 rc_service: waitting "restart_dnsmasq" via  ...
Dec 26 14:24:08 kernel: Adding 262140k swap on /tmp/mnt/USB/myswap.swp.  Priority:-1 extents:69 across:267404k
Dec 26 14:24:08 pixelserv-tls[809]: Listening on :192.168.1.2:443
Dec 26 14:24:08 pixelserv-tls[809]: Listening on :192.168.1.2:80
Dec 26 14:24:10 Diversion: restarted Dnsmasq to apply settings, from /jffs/scripts/dnsmasq.postconf
Dec 26 14:24:11 avahi-daemon[271]: Files changed, reloading.
Dec 26 14:24:11 avahi-daemon[271]: Loading service file /tmp/avahi/services/mt-daap.service.
Dec 26 14:24:11 iTunes: daemon is stopped
Dec 26 14:24:11 FTP_Server: daemon is stopped
Dec 26 14:24:11 Samba_Server: smb daemon is stopped
Dec 26 14:24:11 kernel: gro disabled
Dec 26 14:24:11 avahi-daemon[271]: Got SIGTERM, quitting.
Dec 26 14:24:11 avahi-daemon[271]: Leaving mDNS multicast group on interface br0.IPv4 with address 192.168.1.1.
Dec 26 14:24:11 avahi-daemon[271]: Leaving mDNS multicast group on interface lo.IPv4 with address 127.0.0.1.
Dec 26 14:24:12 avahi-daemon[271]: avahi-daemon 0.7 exiting.
Dec 26 14:24:12 Timemachine: daemon is stopped
Dec 26 14:24:12 kernel: gro enabled with interval 2
Dec 26 14:24:12 avahi-daemon[924]: Found user 'nobody' (UID 65534) and group 'nobody' (GID 65534).
Dec 26 14:24:12 avahi-daemon[924]: Successfully dropped root privileges.
Dec 26 14:24:12 avahi-daemon[924]: avahi-daemon 0.7 starting up.
Dec 26 14:24:12 avahi-daemon[924]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Dec 26 14:24:12 avahi-daemon[924]: No service file found in /tmp/avahi/services.
Dec 26 14:24:12 avahi-daemon[924]: Loading new alias name R7000.
Dec 26 14:24:12 avahi-daemon[924]: Joining mDNS multicast group on interface br0.IPv4 with address 192.168.1.2.
Dec 26 14:24:12 avahi-daemon[924]: New relevant interface br0.IPv4 for mDNS.
Dec 26 14:24:12 avahi-daemon[924]: Joining mDNS multicast group on interface lo.IPv4 with address 127.0.0.1.
Dec 26 14:24:12 avahi-daemon[924]: New relevant interface lo.IPv4 for mDNS.
Dec 26 14:24:12 avahi-daemon[924]: Network interface enumeration completed.
Dec 26 14:24:12 avahi-daemon[924]: Registering new address record for 192.168.1.2 on br0.IPv4.
Dec 26 14:24:12 avahi-daemon[924]: Registering new address record for 192.168.1.1 on br0.IPv4.
Dec 26 14:24:12 avahi-daemon[924]: Registering new address record for 127.0.0.1 on lo.IPv4.
Dec 26 14:24:13 avahi-daemon[924]: Server startup complete. Host name is R7000-4485.local. Local service cookie is 1506266674.
Dec 26 14:24:13 avahi-daemon[924]: Alias name "R7000" successfully established.
Dec 26 14:24:13 Samba_Server: daemon is started
Dec 26 14:24:14 rc_service: zcip 937:notify_rc start_firewall
Dec 26 14:24:14 zcip_client: configured 169.254.23.173
Dec 26 14:24:15 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
Dec 26 14:24:16 custom_script: Running /jffs/scripts/firewall-start (args: ppp0)
Dec 26 14:24:35 kernel: ip_set: protocol 6

Loading new alias name R7000
Host name is R7000-4485.local
https://www.snbforums.com/threads/a...d-forks-on-non-asus-devices-is-illegal.44636/

 

[Xentrk]

Not very effective. Diversion blocks by domain on the router. If YouTube ads are served from the same domain as the actual video then there's not much a router based ad-blocker can do.
In addition to that, YT probably uses some smart code in their Java scripts to obfuscate the domains or IPs they use to pull the ads.

Understand that the router never sees the source code of a rendered webpage, only the requested domain. In-Browser ad-blockers can act on the rendered webpage.

This topic comes up in other forums as well. The only effective way I've found to block YouTube ads is to use a browser ad blocker plug-in like uBlock Origin or Adblock Plus. These browser plug-ins can manipulate the web page to remove HTML components in the page. I have read they insert themselves in between your connection and YouTube to block the ads from rolling. Blocking YouTube ads at the router level is not possible these days. The only method I know of in addition to the browser plug-in method is to pay $11.99/month for a YouTube Premium subscription.

 

[thelonelycoder]

I dropped dropped my lists to around 600k domains and the issue still returned. I took out the duplicate lists you suggested and re-enabled another one I had so im still around 600k domains but don't have the duplicates. Any other suggestions? Its getting to the point where i'm either going to create a script that restarts Diversion every other day or uninstall. Its that frustrating.

Any other suggestions? Its getting to the point where i'm either going to create a script that restarts Diversion every other day or uninstall.

BTW, Diversion has a restart command, it restarts all Diversion services:

diversion restart

 

[Skeptical.me]

To solve the routing & DNS leak issue for my Roku (all Australian Channels, Live TV, Foxtel etc.) which I can only use with my ISP's connection. I'm just going to cascade another router off of the ISP's Modem/Router, that is in bridge mode. Plus I can just connect to it when I want to use my iPhone/iPad with an Aussie IP address.

 

[bmn1]

Is it possible to add a online hosts list to Diversion?

Like this one:
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/update.txt

Since the lists are updated every now and then, it would be convenient if Diversion, like Pi-Hole, would grab and update it's combined block list from online lists.
It's possible I'm being blind, but I can't find it.

Thanks for your work regardless!

Edit:
If that's not possible, which file can I edit to just go ahead and past those host file contents into? (Through FTP and a text editor) Instead of manually adding though the blacklist feature.

 

[Skeptical.me]

@thelonelycoder the Ad Blocking warning banner on the SNB forums is showing. I don't have any Ad Blocking add-ons that would cause this. I only have Diversion at the moment. I vaguely remember I had this issue previously and I think you may have given a solution, but I cannot remember what to do.

Firefox 64 on macOS 10.14.2