Diversion Diversion - the Router Ad-Blocker

[jrmwvu04]

While were at it discussing large files, I would like to hear suggestions for the Small blocking file.

I think mvps.org and pgl.yoyo.org are good for a small list. You could maybe make a case for someone who cares, but I would be more inclined to put that in a medium kind of list.

Two that I see pop up commonly, mahakala and Steven Black, both have tons of false positives and require so much whitelisting that it isn't worth the effort to me.

 

[HairyA00]

I think mvps.org and pgl.yoyo.org are good for a small list. You could maybe make a case for someone who cares, but I would be more inclined to put that in a medium kind of list.

Two that I see pop up commonly, mahakala and Steven Black, both have tons of false positives and require so much whitelisting that it isn't worth the effort to me.

Honestly, I've never found a single site that needed to be whitelisted using this list. Hence my recommendation to make it the 'small' one: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

 

[thelonelycoder]

I think mvps.org and pgl.yoyo.org are good for a small list. You could maybe make a case for someone who cares, but I would be more inclined to put that in a medium kind of list.

All these and more are in Steven Blacks list, which is used in the Standard, Medium and Large blocking file.

Dan Pollok (someonewhocares.org) keeps his to a bare minimum, with regular updates.

 

[HairyA00]

The quick test results in the following in my suggestion (I used a pi-hole to build the domain list because it's quicker and it accepts files that are not in hosts format):

Small: 42,828 domains
Standard: 116,172 domains
Large: 743,747 domains

 

[jrmwvu04]

All these and more are in Steven Blacks list, which is used in the Standard, Medium and Large blocking file.

Dan Pollok (someonewhocares.org) keeps his to a bare minimum, with regular updates.

I personally do use SWC myself. I tried the SB list some time ago, but something pulled in from there was breaking various wife apps, and I found I did not have the patience to sort out what. The benefit was not outweighing the cost for me. When considering block sources, there are variables that affect the usefulness of them - what systems and apps you use, what country you live in, etc. So, to each their own. I find ad block lists to be in the same category with religion and politics - usually best not to talk about it.

But to the question at hand - a small list like the one I propose handles most of the obnoxious common things on the web, and keeps "why doesn't xyz work" questions from your fellow network users to a minimum.

 

[HairyA00]

I personally do use SWC myself. I tried the SB list some time ago, but something pulled in from there was breaking various wife apps, and I found I did not have the patience to sort out what. The benefit was not outweighing the cost for me. When considering block sources, there are variables that affect the usefulness of them - what systems and apps you use, what country you live in, etc. So, to each their own. I find ad block lists to be in the same category with religion and politics - usually best not to talk about it.

But to the question at hand - a small list like the one I propose handles most of the obnoxious common things on the web, and keeps "why doesn't xyz work" questions from your fellow network users to a minimum.

I hear ya... interesting.
It's not too often I hear about anyone using Pi-hole out of the box (which includes StevenBlack's list) having issues with whitelisting. Most people run into issues when they add millions of domains to their lists. Hence my suggestion that "Standard = Medium = Pi-hole settings". Just updated Diversion to match those now:

 

[thelonelycoder]

I hear ya... interesting.
It's not too often I hear about anyone using Pi-hole out of the box (which includes StevenBlack's list) having issues with whitelisting. Most people run into issues when they add millions of domains to their lists. Hence my suggestion that "Standard = Medium = Pi-hole settings". Just updated Diversion to match those now:

View attachment 19675

The Malware Domain List is already included in SB list: https://github.com/StevenBlack/hosts

 

[HairyA00]

The Malware Domain List is already included in SB list: https://github.com/StevenBlack/hosts

Including The Malware Domain List:
601,076 blocked domains 9 hosts file(s) in use

Removed the Malware Domain List:
581,397 blocked domains 8 hosts file(s) in use


I think that's why the Pi-hole guys added them both. Unless Diversion doesn't remove duplicates?

EDIT: You're actually looking at a different list.
https://www.malwaredomainlist.com/hostslist/hosts.txt

is different than

https://mirror1.malwaredomains.com/files/domains.hosts

 

[ludespeedny]

Did a router reboot and checked the amtm disk check log. here is what it said. The flash drive was formatted to ext3 so not sure what is going on.

 

[thelonelycoder]

Did a router reboot and checked the amtm disk check log. here is what it said. The flash drive was formatted to ext3 so not sure what is going on.

Time to format /jffs/ and your USB device, then start from scratch.

 

[ludespeedny]

Time to format /jffs/ and your USB device, then start from scratch.

Ok. I'll give that a shot.

 

[HairyA00]

EDIT: Removed the lists that were not in HOSTS format to simplify. My apologies for making 100 edits on this post to get it all right!

This is what I would suggest for lists:

Whitelist:
https://github.com/anudeepND/whitelist/blob/master/domains/whitelist.txt


Small:
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts


Standard (same as Pi-hole out of the box):
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
http://sysctl.org/cameleon/hosts
https://hosts-file.net/ad_servers.txt
https://mirror1.malwaredomains.com/files/domains.hosts


Medium (eliminate)


Large (ticked lists from firebog.net):

https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
https://hosts-file.net/ad_servers.txt
https://hosts-file.net/grm.txt
http://sysctl.org/cameleon/hosts
https://mirror1.malwaredomains.com/files/domains.hosts
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt

Quick, more accurate update of domains being blocked:

Small: 42,828
Standard: 116,172
Large: 178,454
Plus (+) Hosts: 494,886


Whitelisting should be very minimal, even using the "large" lists.

EDIT: Giving it a shot now. Let's see if I get any complaints in the household.

 

[thelonelycoder]

Including The Malware Domain List:
601,076 blocked domains 9 hosts file(s) in use

Removed the Malware Domain List:
581,397 blocked domains 8 hosts file(s) in use


I think that's why the Pi-hole guys added them both. Unless Diversion doesn't remove duplicates?

EDIT: You're actually looking at a different list.
https://www.malwaredomainlist.com/hostslist/hosts.txt

is different than

https://mirror1.malwaredomains.com/files/domains.hosts

Why would a mirror be different than the master? Anyway that hosts list is dead, last changes were made end of 2017. That's ages ago for the fast changing times we live in.
But for some reason, the mirror is still being updated https://mirror1.malwaredomains.com/files/. Go figure. Unless I look at it the wrong way.

 

[martinr]

Why would a mirror be different than the master? .

All the lettering is back to front?

 

[thelonelycoder]

All the lettering is back to front?

duh! (detected user humilation)

 

[HairyA00]

I got rid of https://zerodot1.gitlab.io/CoinBlockerLists/hosts

There are tons of legit domains on there, 49 belonging to coinbase, which is a legit website. So my "large" list keeps dwindling down, but seems pretty stable now. I did need to whitelist app.box.com else Box was broken. That's it so far though...

Also, a subset of that hosts file is already included in Steven Black's list.

 

[dave14305]

A weeks worth of log files for the weekly stats report. The log files get reset on blocking list update day, no matter if stats run or not.

Diversion rotates my dnsmasq.log every morning, but only keeps 3 log files which would mean 3 days. Are the weekly stats really weekly? Or just those 3 days?

 

[thelonelycoder]

Diversion rotates my dnsmasq.log every morning, but only keeps 3 log files which would mean 3 days. Are the weekly stats really weekly? Or just those 3 days?

Without a shadow of doubt, one week.
The daily rotate logs does this:

cat /opt/var/log/dnsmasq.log1 >>/opt/var/log/dnsmasq.log2
        mv /opt/var/log/dnsmasq.log /opt/var/log/dnsmasq.log1
        >/opt/var/log/dnsmasq.log

 

[dave14305]

Without a shadow of doubt, one week.
The daily rotate logs does this:

cat /opt/var/log/dnsmasq.log1 >>/opt/var/log/dnsmasq.log2
        mv /opt/var/log/dnsmasq.log /opt/var/log/dnsmasq.log1
        >/opt/var/log/dnsmasq.log

Oh, you APPEND log1 to log2. Pretty damn tricky!

 

[thelonelycoder]

Oh, you APPEND log1 to log2. Pretty damn tricky!

Appending takes time with a large log, especially if writing is slow with some devices.
During that time, Dnsmasq would not be able to write new entries. Renaming the file and creating a new one only takes about a second.