AdBlocking with combined hosts file

[thelonelycoder]

probably a really dumb question, but what fs does the usb stick need to be formatted in?

I have it in ext2, but fat should be ok:

Edit: Also official Asus document

The wireless router works with most USB HDDs/Flash disks up to
2TB and supports read-write access for FAT16, FAT32, EXT2, EXT3,
and NTFS

 

[Eet_46]

I might have missed it somewhere in the thread, but is there and easy ON/OFF "switch" for the advanced method of adblocking? Or do one have to delete the scripts manually to revert? Tried searching for it, but without luck..

 

[thelonelycoder]

I might have missed it somewhere in the thread, but is there and easy ON/OFF "switch" for the advanced method of adblocking? Or do one have to delete the scripts manually to revert? Tried searching for it, but without luck..

There's no easy OnOff switch. It's still simple though if you went through the installation successful...
I have posted the how to On/Off a few posts back.

Edit: Now part of the first post

 

[waffles]

Add this to dnsmasq.conf.add (adjust path to your liking):

# Logging
log-facility=/tmp/mnt/sda1/logs/dnsmasq.log
log-queries

Then open a terminal and log into your router.
Enter

service restart_dnsmasq

And then

tail -f /tmp/mnt/sda1/logs/dnsmasq.log

Pick some of the entries in the hosts.blocked file and paste it into the address field of your brwoser and open the page.

For blocked hosts you should see something like this:

query[A] settings-win.data.microsoft.com from 192.168.2.200
/tmp/mnt/sda1/hosts/blacklist.txt settings-win.data.microsoft.com is 0.0.0.0
or:
query[A] 7457.accessaw.blueseek.com from 192.168.2.200
/tmp/mnt/sda1/hosts/hosts.blocked.txt 7457.accessaw.blueseek.com is 0.0.0.0

Thanks for your help!

I made these amendments as you suggested (I had to create an (empty) dmsmasq.log file manually though.)

When I enter e.g. "www.moviedollars.com" (one of the lines from the hosts.blocked file, I find this in the log:

Sep  6 13:02:51 dnsmasq[21975]: query[AAAA] rgom10-en.url.trendmicro.com from 127.0.0.1
Sep  6 13:02:51 dnsmasq[21975]: cached rgom10-en.url.trendmicro.com is <CNAME>
Sep  6 13:02:51 dnsmasq[21975]: forwarded rgom10-en.url.trendmicro.com to 209.18.47.61
Sep  6 13:02:51 dnsmasq[21975]: forwarded rgom10-en.url.trendmicro.com to 209.18.47.62
Sep  6 13:02:51 dnsmasq[21975]: reply rgom10-en.url.trendmicro.com is <CNAME>
Sep  6 13:02:51 dnsmasq[21975]: reply trendmicro.com.edgesuite.net is <CNAME>
Sep  6 13:02:51 dnsmasq[21975]: query[AAAA] trendmicro.com.edgesuite.net from 127.0.0.1
Sep  6 13:02:51 dnsmasq[21975]: forwarded trendmicro.com.edgesuite.net to 209.18.47.61
Sep  6 13:02:51 dnsmasq[21975]: reply trendmicro.com.edgesuite.net is <CNAME>
Sep  6 13:02:51 dnsmasq[21975]: query[AAAA] a151.g.akamai.net from 127.0.0.1
Sep  6 13:02:51 dnsmasq[21975]: forwarded a151.g.akamai.net to 209.18.47.61
Sep  6 13:02:51 dnsmasq[21975]: query[A] rgom10-en.url.trendmicro.com from 127.0.0.1
Sep  6 13:02:51 dnsmasq[21975]: forwarded rgom10-en.url.trendmicro.com to 209.18.47.61
Sep  6 13:02:51 dnsmasq[21975]: reply rgom10-en.url.trendmicro.com is <CNAME>
Sep  6 13:02:51 dnsmasq[21975]: reply trendmicro.com.edgesuite.net is <CNAME>
Sep  6 13:02:51 dnsmasq[21975]: reply a151.g.akamai.net is 23.212.53.183
Sep  6 13:02:51 dnsmasq[21975]: reply a151.g.akamai.net is 23.212.53.173
Sep  6 13:02:51 dnsmasq[21975]: query[AAAA] rgom10-en.url.trendmicro.com from 127.0.0.1
Sep  6 13:02:51 dnsmasq[21975]: cached rgom10-en.url.trendmicro.com is <CNAME>
Sep  6 13:02:51 dnsmasq[21975]: cached trendmicro.com.edgesuite.net is <CNAME>
Sep  6 13:02:51 dnsmasq[21975]: forwarded rgom10-en.url.trendmicro.com to 209.18.47.61
Sep  6 13:02:52 dnsmasq[21975]: reply rgom10-en.url.trendmicro.com is <CNAME>
Sep  6 13:02:52 dnsmasq[21975]: reply trendmicro.com.edgesuite.net is <CNAME>
Sep  6 13:02:52 dnsmasq[21975]: query[AAAA] trendmicro.com.edgesuite.net from 127.0.0.1
Sep  6 13:02:52 dnsmasq[21975]: forwarded trendmicro.com.edgesuite.net to 209.18.47.61
Sep  6 13:02:52 dnsmasq[21975]: reply trendmicro.com.edgesuite.net is <CNAME>
Sep  6 13:02:52 dnsmasq[21975]: query[AAAA] a151.g.akamai.net from 127.0.0.1
Sep  6 13:02:52 dnsmasq[21975]: forwarded a151.g.akamai.net to 209.18.47.61
Sep  6 13:02:52 dnsmasq[21975]: query[A] rgom10-en.url.trendmicro.com from 127.0.0.1
Sep  6 13:02:52 dnsmasq[21975]: forwarded rgom10-en.url.trendmicro.com to 209.18.47.61
Sep  6 13:02:52 dnsmasq[21975]: reply rgom10-en.url.trendmicro.com is <CNAME>
Sep  6 13:02:52 dnsmasq[21975]: reply trendmicro.com.edgesuite.net is <CNAME>
Sep  6 13:02:52 dnsmasq[21975]: reply a151.g.akamai.net is 23.3.97.96
Sep  6 13:02:52 dnsmasq[21975]: reply a151.g.akamai.net is 23.3.97.24

The add page shows up in the browser window.

So it seems that something is not working correctly.

Any suggestions?

edit: " I have set selected 'OpenDNS' under AiProtection - DNS Filtering - DNS-based filtering. Turning it off or to a different service, changes the log.
=> What is the recommended setting and why?

 

[thelonelycoder]

@waffles This will not work with OpenDNS. However, there is an excellent solution by @ColinTaylor in this thread.

 

[sfx2000]

Not to be one stating the obvious, but wouldn't it be easier just to block the ads in the browser with a plug-in/extension?

Using DNSMasq to block is a valid approach, but this is going to take up processing horsepower better used for other things.

And it's not like these routers have a surplus of CPU horsepower in any event.

 

[waffles]

Not to be one stating the obvious, but wouldn't it be easier just to block the ads in the browser with a plug-in/extension?

Using DNSMasq to block is a valid approach, but this is going to take up processing horsepower better used for other things.

And it's not like these routers have a surplus of CPU horsepower in any event.

I use it for my iOS devices on my network.
FF + Adblock Plus for my PCs

 

[sfx2000]

I use it for my iOS devices on my network.

Good point - iOS and Androids would be something to consider...

iOS 9 comes out next week, and it's got adblocking built into Safari directly...

 

[waffles]

...

iOS 9 comes out next week, and it's got adblocking built into Safari directly...

That's intriguing. Let's see how well this will actually work.

 

[waffles]

@waffles This will not work with OpenDNS. However, there is an excellent solution by @ColinTaylor in this thread.

Thanks much for the pointer. I will give his suggestion
"As an alternative you could set the filter to "Router" and then set the router's WAN DNS to OpenDNS. That way you would get local name resolution and still force the use of OpenDNS" a try. This seems to be the best of both worlds.

 

[sfx2000]

That's intriguing. Let's see how well this will actually work.

In the public beta's - it works very well, enough so that bloggers/advertisers are all in arms about it, and google has actually recommended changes to their ex-Doubleclick customers to work around it.

Thinking back though - this really could be a value add for a 3rd Party DNS provider to do add blocking there - obviously Google DNS ain't gonna do that (8.8.8.8/8.8.4.4)...

 

[thelonelycoder]

Not to be one stating the obvious, but wouldn't it be easier just to block the ads in the browser with a plug-in/extension?

Using DNSMasq to block is a valid approach, but this is going to take up processing horsepower better used for other things.

And it's not like these routers have a surplus of CPU horsepower in any event.

The obvious advantage of doing it on the router is to not have to install and maintain several AdBlocking packages on all your devices.
With this you have one place to manage your blockers and in a very efficient and effective way.

The workload for the router - from my experience and others using this method - is minimal with a hosts file and there is plenty of horsepower left for other duties for it if you need it.

 

[Authority]

------------------------------------------------------------
Changelog
------------------------------------------------------------
September 09, 2015: Added Quick Enable / Disable Adblocking, TOC added

How can you have a change log with a future date and past tense?

 

[Authority]

Thinking back though - this really could be a value add for a 3rd Party DNS provider to do add blocking there - obviously Google DNS ain't gonna do that (8.8.8.8/8.8.4.4)...

I believe OpenDNS *did* briefly do this, and in a way still can... you can manually add sites to a black list. They say it hurts performance though:

"Significant reduction in speed is partially due to the process used by many browsers to render a Web page. Each element of content on a Web page is loaded sequentially and if several elements are blocked, the browser waits for a timeout to expire before moving to the next element. This will cause very significant delays when there are multiple elements being blocked.

This performance degradation on the user experience is the main reason OpenDNS does not offer an Advertising category in Web Content filtering."

https://support.opendns.com/entries/26022379-Can-I-Block-Advertisers-and-Ad-Servers-

 

[thelonelycoder]

"Significant reduction in speed is partially due to the process used by many browsers to render a Web page. Each element of content on a Web page is loaded sequentially and if several elements are blocked, the browser waits for a timeout to expire before moving to the next element. This will cause very significant delays when there are multiple elements being blocked.

That is boilerplate text to me. I see no noticable loading delays when opening a Website with a lot of ads. The router handles the timeouts which is instant due to dnsmasqs design. And I bet rendering is faster with a hosts file than a heavy Browser Addon like Adblock Plus. That is the experience I have.

 

[mstombs]

Google Chrome has good Network load diagnostics, some sites can appear to hang or take forever to complete loading due to multiple timeouts with some types of adblock. This was a significant motivation for using a pixelserv server which replies quickly with an appropriate response then closes the connection. I have noticed many more ads being served using https connections, there's nothing you can to inject blank images for these, maybe better for speed to iptables 'reject' rather than 'drop'.

 

[thelonelycoder]

Thanks for the scripts! I modified the "advanced" one as follow (AC68):

1) Put the content in wan-start, instead of service-start (which didn't work for me for some reason)
2) Never installed entware, therefore removed the "/opt/etc/init.d/rc.unslung start" line and modified all the /tmp/mnt/sda1/* references.
3) update-hosts.sh: Modified the middle portion like this:

# get hosts files and combine
wget -qO- \
"http://winhelp2002.mvps.org/hosts.txt" \
"http://someonewhocares.org/hosts/zero/hosts" \
"http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" \
"http://www.malwaredomainlist.com/hostslist/hosts.txt" \
"http://hosts-file.net/ad_servers.txt" \
> $dir/temphost

# replace all 127.0.0.1 with 0.0.0.0 if any, grab only the first 2 parts (IP, hostname) of each line, and sort the combined entries
cat $dir/temphost | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u > $dir/temphost2

# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temphost2 > $dir/hosts.blocked

#remove temp files
rm $dir/temphost
rm $dir/temphost2

The reasons are
a) the entries in the 2nd wget aren't sorted against the entries from the 1st wget
b) some entries come with comments
c) different sites use different delimiters: yoyo.org etc use single space, malewaredomainlist.com uses double spaces, and hosts-file.net uses tab. So I use awk to grab only $1 (0.0.0.0) and $2 (hostname) and feed sort -u with only the "cleansed" entries.

Thanks for your input, I just tested your modified version and noticed a significant increase of time to process for the replace part of it.
In addintion to the 5 listed host files I include http://support.it-mate.co.uk/downloads/hosts.txt, a rather large file.
With the current script on my RT-AC66U I get these times:

real    0m55.992s
user    0m22.610s
sys    0m6.250s

While yours I had these times:

real    3m45.761s
user    1m31.790s
sys    1m42.590s

Not a big problem exept when rebooting, it takes longer for the router to come up. What I like is the hosts.blocked file decreases in size from 13524 KB down to 11751 KB with your refinements. That means less memory usage on the router.
For the time being I'll leave your script running.

I also had to go forth and back with the service-start and wan-start placement of the calling line for the script. I activated the logger to see where it failed but never really found the source of the problem. It eventally worked everytime with the service-start so I left it in there.

 

[zekesdad]

im having two issues, and Im sure they're just me, but Im pretty new to linux so please be patient.

1) using the advanced method, when I get to:
chmod a rx /jffs/scripts/*
I get:
chmod: rx: No such file or directory
chmod: invalid mode 'a'

inside /jffs/scripts I have the two files, services-start and update-hosts.sh

2) /tmp/mnt/sda1/hosts/whitelist.txt

I dont have anything in that directory past /tmp/mnt/ except for an ADBLOCK folder
The usb stick is inserted and recognized on the Network Map page of the router's settings, but I don't see it mounted here. on Merlin's newest firmware, 55. Thanks for any help.

 

[Chrysalis]

doing it in the router has all sorts of advantages.
first I rather use my router cpu than my desktop cpu for filtering.
second it still filters when using a vpn which is nice
third it filters for all devices on the lan

Doesnt mean I dont do browser filtering as well, but I think filtering on a router has lots of merits.

 

[thelonelycoder]

im having two issues, and Im sure they're just me, but Im pretty new to linux so please be patient.

1) using the advanced method, when I get to:
chmod a rx /jffs/scripts/*
I get:
chmod: rx: No such file or directory
chmod: invalid mode 'a'

inside /jffs/scripts I have the two files, services-start and update-hosts.sh

That is missing a + inbetween. For some reason that was omitted in an edit I've done recently. Maybe a XenForo bug or something.
The line now again correctly reads chmod a+rx /jffs/scripts/* in the first post.

2) /tmp/mnt/sda1/hosts/whitelist.txt

I dont have anything in that directory past /tmp/mnt/ except for an ADBLOCK folder
The usb stick is inserted and recognized on the Network Map page of the router's settings, but I don't see it mounted here. on Merlin's newest firmware, 55. Thanks for any help.

You have to create the directory /hosts in the root directory of your stick, you will then have a path like this: /tmp/mnt/sda1/hosts/.
Then place the whitelist and blacklist files in it.