AdBlocking with combined hosts file

[elorimer]

At least in Chrome, if you right click and select "Open link in new tab" you can follow a link. I don't know why clicking doesn't work.

N16 here using this method of adblocking, with added blacklist.txt

 

[davejsb]

I use this method but I use a static file created the first time implemented and it works great in the most part.

However my other half likes going on this site:

http://www.digitalspy.co.uk/

which wont work at all as it is an ad fest, I have removed all of the digital spy references but none of the links work on the site unless ad blocking is disabled.

Any suggestions on how to get it to work?

Also, is there a 'cut down' version of the domain file anywhere which isn't as comprehensive and maybbe only covers core ad domains?

I have this exact problem too. N66U.

 

[ColinTaylor]

However my other half likes going on this site:

http://www.digitalspy.co.uk/

which wont work at all as it is an ad fest, I have removed all of the digital spy references but none of the links work on the site unless ad blocking is disabled.

Strange, it's working perfectly for me. Can you copy one of the links that doesn't work and post it here so we can test it?

 

[davejsb]

Strange, it's working perfectly for me. Can you copy one of the links that doesn't work and post it here so we can test it?

The links when taken alone do work.

But clicking on any link from the homepage - http://www.digitalspy.co.uk/ results in no action.

The only way to get the links to work is either copying them into the address bar or right clicking in Chrome and choosing "open link in new tab".

EDIT : interestingly I just tried using Firefox instead of my default browser Chrome, and the links work perfectly. As do they in IE. But Safari on my iPad and iPhone when using my home wi-fi also results in the links not working. When used via 3G though the links work on both.

 

[ColinTaylor]

The links when taken alone do work.

But clicking on any link from the homepage - http://www.digitalspy.co.uk/ results in no action.

The only way to get the links to work is either copying them into the address bar or right clicking in Chrome and choosing "open link in new tab".

EDIT : interestingly I just tried using Firefox instead of my default browser Chrome, and the links work perfectly. As do they in IE. But Safari on my iPad and iPhone when using my home wi-fi also results in the links not working. When used via 3G though the links work on both.

bilboSNB: Do you have the same behaviour? What browser are you using?

PS Remember to completely close and reopen your browser between tests because the browser will cache a lot of the DNS queries.

 

[Zirescu]

I also tested this with Safari on a Mac and had the same issue. I reviewed the console log and noticed a bunch of javascript errors. I went and disabled javascript, reloaded the page, and all the links worked.

 

[davejsb]

I also tested this with Safari on a Mac and had the same issue. I reviewed the console log and noticed a bunch of javascript errors. I went and disabled javascript, reloaded the page, and all the links worked.

Disabling javascript in Chrome fixes this issue too.

 

[elorimer]

This isn't really about routers, but if you "inspect elements" on some of those links they are a little alarming. I don't speak webpagese but why do they need to know if I have Google Voice?

 

[bilboSNB]

Worked , thanks!

 

[thelonelycoder]

I use this method but I use a static file created the first time implemented and it works great in the most part.

This might be an old listing of blocked domains then. You may want to update it and use an domain exclusion file as posted here.

 

[Sharkus77]

Thanks for such a useful thread!

One question, I've found the hosts used to be lacking a bit. However, I can't seem to find any more host lists with 0.0.0.0 (Everyone seems to love 127.0.0.1).. Is there any way to update this to use a site with 127.0.0.1 instead of 0.0.0.0? Or would it be possible to use add a local hosts_blacklist.txt (/tmp/mnt/sda/hosts_blacklist.txt) into the mix for merging with the others?

Thanks for any help! Learning so much from all this

 

[Brouno]

Nice tips

Just a little thing :

After implementing this you can see that reverse resolution of 0.0.0.0 return the first line of the file hosts.clean
for example in the command netstat -a

To avoid this, just add this setting to the /jffs/configs/dnsmasq.conf.add

ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0

So , you got file /jffs/configs/dnsmasq.conf.add

address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts.clean
ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0

et voila !

 

[elorimer]

My N16 has been operating using about 98% of its memory, leaving only 2 mb free. I recently noticed that my hosts.clean file became 0 bytes and the ad blocking wasn't working, so something in my update isn't working right.

But, I also noted that my memory usage was down to around 70%. Am I right in assuming that these blocked addresses are read into memory? I might want to prune this file down, or use an external blocker.

 

[thelonelycoder]

Thanks for such a useful thread!

One question, I've found the hosts used to be lacking a bit. However, I can't seem to find any more host lists with 0.0.0.0 (Everyone seems to love 127.0.0.1).. Is there any way to update this to use a site with 127.0.0.1 instead of 0.0.0.0? Or would it be possible to use add a local hosts_blacklist.txt (/tmp/mnt/sda/hosts_blacklist.txt) into the mix for merging with the others?

Thanks for any help! Learning so much from all this

There is a big debate about the merits of 127.0.0.0 vs 0.0.0.0. In my opinion the latter wins as it is not answered by any host. 0.0.0.0 simply does not exist. 127.0.0.0 however is your local machine or your router. It will try to answer the request made to it and time out.

I have a whitelist addition in this thread.
If you want to add a blacklist just add your file (eg hosts_blacklist.txt) to /jffs/configs/dnsmasq.conf.add like so:

addn-hosts=/tmp/mnt/sda1/hosts_blacklist.txt

 

[thelonelycoder]

My N16 has been operating using about 98% of its memory, leaving only 2 mb free. I recently noticed that my hosts.clean file became 0 bytes and the ad blocking wasn't working, so something in my update isn't working right.

But, I also noted that my memory usage was down to around 70%. Am I right in assuming that these blocked addresses are read into memory? I might want to prune this file down, or use an external blocker.

My RT-AC66U runs at about 30% with a blocked hosts file of 690 KB.
You could limit the hosts files to only use pgl.yoyo.org of the three downloaded in /jffs/scripts/post-mount and /jffs/scripts/update-hosts.sh like so:

wget -qO- "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean

This would trim your file down considerably.

 

[grogi]

Create or add to /jffs/configs/dnsmasq.conf.add

address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts.clean

Why do we need to specify the address=/0.0.0.0/0.0.0.0 line?

 

[ColinTaylor]

Why do we need to specify the address=/0.0.0.0/0.0.0.0 line?

This was answered here: http://www.snbforums.com/threads/adblocking-with-combined-hosts-file.15309/page-4#post-119703

Although it might be interesting to see if this post (http://www.snbforums.com/threads/adblocking-with-combined-hosts-file.15309/page-6#post-171732) provides a different/better solution.

 

[ScottW]

This works GREAT.... But one thing I don't see discussed in this thread is IPV6. I'm still seeing quite a few successful resolutions to ad servers using IPV6 addresses.

Any easy way to fix this? Would there be a quick SED or AWK line that can be added to the pipe, that would read each "0.0.0.0 hostname" pair, then echo it back out, plus a line feed and ":: hostname" pair on the next line?

Would something like this after the "sort" pipe work?
awk '{print $1,$2,"\n::",$2}'

Or is there a better way that doesn't make such a large file?

 

[thelonelycoder]

This works GREAT....
Or is there a better way that doesn't make such a large file?

I am still waiting for my ISP to enable IPv6. Until then there's no use for me to look into that. I would not be able to test it.

 

[ScottW]

I am still waiting for my ISP to enable IPv6. Until then there's no use for me to look into that. I would not be able to test it.

I tested, using the AWK command in my previous post. It does work, creating a ":: domain" line for every "0.0.0.0 domain" line in the file. This obviously doubles the number of lines in the file, but dnsmasq seems to be fine with that and I don't see any obvious performance degradation.

Mar 16 13:59:41 dnsmasq[746]: read /tmp/mnt/ASUS/hosts/hosts.clean - 49374 addresses

I was hoping there was a better way -- without duplicating every line -- but if not, this seems to work for now!