What's new

AdBlocking with combined hosts file

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

1. Standard method (the original)

The files are stored on a USB disk, /tmp/mnt/sda1/ is the path. If you use /jffs/ edit accordingly.

Using a USB disk we need to delay the startscript.
This will create an up-to-date hosts file after every reboot.
Add to /jffs/scripts/post-mount
Code:
#!/bin/sh
sleep 12
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serve...tdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 5
service restart_dnsmasq
Tell dnsmasq to add the new hosts file and where to send the unwanted requests to (sending them to 0.0.0.0)
Create or add to /jffs/configs/dnsmasq.conf.add
Code:
address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts.clean
To update the hosts file (manually or as a cron job) create /jffs/scripts/update-hosts.sh
Code:
#!/bin/sh
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serve...tdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 5
service restart_dnsmasq
To run update-hosts.sh every Friday midnight as a cron job create or add to /jffs/scripts/init-start /jffs/scripts/services-start
Code:
#!/bin/sh

cru a UpdateHosts "00 00 * * 5 /jffs/scripts/update-hosts.sh"
Set all files in /jffs/scripts/ as executable
Code:
chmod a+rx /jffs/scripts/*
That's it, reboot and enjoy!
Tested on a RT-AC66U.

i have this installed on my ac68u....no errors on logfile but what files do i need to working this?
here is my log....http://pastebin.com/7N06W2tK

i see this but i do not know what is missing there?
Code:
dnsmasq[436]: failed to load names from /tmp/mnt/sda1/hosts.clean: No such file or directory
picture: http://imagizer.imageshack.com/img907/4698/a0i7V9.png
 
Last edited:
You can do it however you want @ChikkSpot.
i'm curious which way is faster, processing-power-wise (i.e., domain suffix matching vs going through tons of entries).
I believe you mix up the hosts file and the blacklist. My blacklist only has about 52 (manual) entries. The rest is handled by the hosts file.
 
i see this but i do not know what is missing there?
Code:
dnsmasq[436]: failed to load names from /tmp/mnt/sda1/hosts.clean: No such file or directory
It appears to be running according to your log: Nov 2 19:04:20 dnsmasq[853]: read /tmp/mnt/sda1/hosts.clean - 25248 addresses

It's just that the usb drive is not yet mounted at 19:01:48
 
You can do it however you want @ChikkSpot.

I believe you mix up the hosts file and the blacklist. My blacklist only has about 52 (manual) entries. The rest is handled by the hosts file.

no I was referring to the hosts.blocked file.

if you look at the entries from those 5 different sites in the script, the ones from mvps.org, someonewhocares.org and hosts-file.net contains all kinds of variants for each domain:

0.0.0.0 ad.ae.doubleclick.net
0.0.0.0 ad.ar.doubleclick.net
0.0.0.0 ad.at.doubleclick.net
0.0.0.0 ad.au.doubleclick.net
0.0.0.0 ad.be.doubleclick.net
0.0.0.0 ad.br.doubleclick.net
0.0.0.0 ad.ca.doubleclick.net
0.0.0.0 ad.ch.doubleclick.net
0.0.0.0 ad.cl.doubleclick.net
0.0.0.0 ad.cn.doubleclick.net
0.0.0.0 ad.de.doubleclick.net #[Tracking.Cookie]
0.0.0.0 ad.dk.doubleclick.net
0.0.0.0 ad.es.doubleclick.net
0.0.0.0 ad.fi.doubleclick.net
.... and lots and lots of those.

the ones from pgl.yoyo.org and malwaredomainlist.com only contain the domain names.

therefore, instead of putting hundred of those (for example) doubleclick.net entries in a /tmp/hosts.blocked file, could have simply do this with just 1 line in dnsmasq.conf.add:

address=/doubleclick.net/0.0.0.0

now here is the problem: dnsmasq.conf.add lives in /jffs, which means it's not wise to write to it frequently. putting the domain names while it's probably more efficient processing-wise it's going to wear out /jffs faster.

another observation is, with the script in its current form, the entries from pgl.yoyo.org and malwaredomainlist.com are having extremely limited usefulness, as any single variation of those isn't going to be blocked, even something as simple as putting a "www." in front of it. just try ping www.yougube.com instead of yougube.com, and you can't possibly manually blacklist them all.

from a strictly technical point of view, those sites produce 2 very different types of entries and really need to be processed differently.
 
It appears to be running according to your log: Nov 2 19:04:20 dnsmasq[853]: read /tmp/mnt/sda1/hosts.clean - 25248 addresses

It's just that the usb drive is not yet mounted at 19:01:48
are you saying that this working? how to test if this addresses realy blocked to create connection to my browser?
 
@bayern1975 By entering one of the hosts in the hosts.clean file into your browser and see what happens for example?
 
@bayern1975 By entering one of the hosts in the hosts.clean file into your browser and see what happens for example?
so, this addresses not open...this is correct? how to add more addresses or IP addresses to block? if i add address to hosts.clean file and reboot lost that address....

ok, i got it....this is advanced method....now i have just one problem to solve....blocking incoming conection to router....but i not have luck for now.....
 
Last edited:
so, this addresses not open...this is correct? how to add more addresses or IP addresses to block? if i add address to hosts.clean file and reboot lost that address....
Yes, you seem to have a working adblocking. To manually add your own blacklist you will need to use the advanced method in post #1. But you'll have to remove all files from the standard method first, as described in the post.
 
reSearchITeng, you could reduce the filesize using more hostnames per line.
So don't use "127.0.0.1 hostA". Better use "127.0.0.1 hostA hostB hostC" and so on. I don't know if you find a solution per script.

You can see a possible result on the folliwng website within the "hosts-optimized.zip". I didn't check if they use the same sources like you do.
http://hosts-file.net/?s=Download

It's probably also a good idea for the other solutions in this thread as it will reduce the used memory.
 
no I was referring to the hosts.blocked file.

if you look at the entries from those 5 different sites in the script, the ones from mvps.org, someonewhocares.org and hosts-file.net contains all kinds of variants for each domain:

0.0.0.0 ad.ae.doubleclick.net
0.0.0.0 ad.ar.doubleclick.net
0.0.0.0 ad.at.doubleclick.net
0.0.0.0 ad.au.doubleclick.net
0.0.0.0 ad.be.doubleclick.net
0.0.0.0 ad.br.doubleclick.net
0.0.0.0 ad.ca.doubleclick.net
0.0.0.0 ad.ch.doubleclick.net
0.0.0.0 ad.cl.doubleclick.net
0.0.0.0 ad.cn.doubleclick.net
0.0.0.0 ad.de.doubleclick.net #[Tracking.Cookie]
0.0.0.0 ad.dk.doubleclick.net
0.0.0.0 ad.es.doubleclick.net
0.0.0.0 ad.fi.doubleclick.net
.... and lots and lots of those.

the ones from pgl.yoyo.org and malwaredomainlist.com only contain the domain names.

therefore, instead of putting hundred of those (for example) doubleclick.net entries in a /tmp/hosts.blocked file, could have simply do this with just 1 line in dnsmasq.conf.add:

address=/doubleclick.net/0.0.0.0

now here is the problem: dnsmasq.conf.add lives in /jffs, which means it's not wise to write to it frequently. putting the domain names while it's probably more efficient processing-wise it's going to wear out /jffs faster.

another observation is, with the script in its current form, the entries from pgl.yoyo.org and malwaredomainlist.com are having extremely limited usefulness, as any single variation of those isn't going to be blocked, even something as simple as putting a "www." in front of it. just try ping www.yougube.com instead of yougube.com, and you can't possibly manually blacklist them all.

from a strictly technical point of view, those sites produce 2 very different types of entries and really need to be processed differently.

Good post. Now, my question: Can't we generate a file in a USB drive with the list of "address=/xxxx.xxx/0.0.0.0" lines and have that appended to dnsmasq.conf with dnsmasq.conf.postconf? Wouldn't that solve the whole "don't waste jffs write cycles"?

I've never done a postconf myself, but it doesn't seem to far fetched...
 
Good post. Now, my question: Can't we generate a file in a USB drive with the list of "address=/xxxx.xxx/0.0.0.0" lines and have that appended to dnsmasq.conf with dnsmasq.conf.postconf? Wouldn't that solve the whole "don't waste jffs write cycles"?
It would be fairly easy to set up the dnsmasq entries. pgl.yoyo.org even supply their list in dnsmasq format already. The jffs writes are a non-issue, as has been discussed at length elsewhere in this forum. Performance using the current "large" host file is also not an issue, even on my N66U.

So, the main question is "is it worth the effort?". You'd be adding another layer of complexity on top of something that some people struggle with already.

Unless you planned on removing the hosts.blocked file completely and using only dnsmasq.conf you'd still end up parsing the same size hosts file.

Using these pre-built lists is always going to be an imperfect method. Personally, I prefer to keep the scripts as simple as possible rather than add complexity just for the sake of a couple of extra domains.
 
dnsmasq.conf.postconf

So, the main question is "is it worth the effort?". You'd be adding another layer of complexity on top of something that some people struggle with already.

right now I'm testing a new script of my own.

in it i'm generating the entries in dnsmasq.conf format and at the end i do a cat /tmp/temphost >> /etc/dnsmasq.conf (i'm only using the entries from pgl.yoyo.org and www.malwaredomainlist.com, as the other 3 are like the expanded version of those. i'm sure they contain more stuff but just those 2 are good enough for me).

then all i really need is service restart_dnsmasq and i don't even need to call any script explicitly anywhere, as everything is handled nicely by .postconf autorun. then i set those up in wan-start:

Code:
#!/bin/sh

sleep 10
cru a UpdateHosts "00 00 * * 5 service restart_dnsmasq"

# re-launch dnsmasq because dnsmasq came up before wan was ready
service restart_dnsmasq

so it's actually less complex.
 
right now I'm testing a new script of my own.

in it i'm generating the entries in dnsmasq.conf format and at the end i do a cat /tmp/temphost >> /etc/dnsmasq.conf (i'm only using the entries from pgl.yoyo.org and www.malwaredomainlist.com, as the other 3 are like the expanded version of those. i'm sure they contain more stuff but just those 2 are good enough for me).

then all i really need is service restart_dnsmasq and i don't even need to call any script explicitly anywhere, as everything is handled nicely by .postconf autorun. then i set those up in wan-start:

Code:
#!/bin/sh

sleep 10
cru a UpdateHosts "00 00 * * 5 service restart_dnsmasq"

# re-launch dnsmasq because dnsmasq came up before wan was ready
service restart_dnsmasq

so it's actually less complex.
Took the idea of having everything set up using "address=/xxxx.xxx/0.0.0.0", your suggestion of using only those two sources and do not use any other script besides dnsmasq.postconf and got rid of the hosts file completely. Will see how it goes as for blocking ads...

This is my version of dnsmasq.postconf
Code:
#!/bin/sh

logger -t $(basename $0) "DNS Postconf Started!"

#Location of downloaded and working files
dir=/tmp/mnt/sda2/hosts
#IP to use to resolve hosts
DivertIP=10.99.99.99
#Set Timeout and Tries
tmout=30
tries=3

#Get Yoyo server list, ready in dnsmasq format using DivertIP
wget -qO- -T $tmout -t $tries "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip="$DivertIP > $dir/yoyo.txt

if [[ $? -eq 0 ]]; then
    #Download OK!
    logger -t $(basename $0) "Yoyo download OK"
    echo "Yoyo download OK"
    #Save Backup
    cp $dir/yoyo.txt $dir/yoyo_bkp.txt
else
    logger -t $(basename $0) "Yoyo download FAILED"
    echo "Yoyo download FAILED"
    #Use Backup
    cp $dir/yoyo_bkp.txt $dir/yoyo.txt
fi

#Get Malwaredomainlist hosts file, must be worked on
wget -q -T $tmout -t $tries -O $dir/mdl.txt  "http://www.malwaredomainlist.com/hostslist/hosts.txt"

if [[ $? -eq 0 ]]; then
    #Download OK!
    logger -t $(basename $0) "Malwaredomainlist download OK"
    echo "Malwaredomainlist download OK"
    #Save Backup
    cp $dir/mdl.txt $dir/mdl_bkp.txt
else
    logger -t $(basename $0) "Malwaredomainlist download FAILED"
    echo "Malwaredomainlist download FAILED"
    #Use Backup
    cp $dir/mdl_bkp.txt $dir/mdl.txt
fi

#Continue Processing of file
#Convert DOS->UNIX
sed -i 's/.$//' $dir/mdl.txt
#Remove all comments from file
sed -i 's/#.*//g' $dir/mdl.txt
#Remove al spaces from file
sed -i 's/\t//g' $dir/mdl.txt
#Remove all localhost entries
sed -i '/localhost/d' $dir/mdl.txt
#Changes the "127.0.0.1" at the beginning of each line with "address=/"
sed -i 's/^127.0.0.1  /address=\//g' $dir/mdl.txt
#Removes all lines not containing "address"
sed -i '/address/!d' $dir/mdl.txt
#adds "/$DivertIP" to the end of each line
sed -i 's/$/\/'$DivertIP'/' $dir/mdl.txt
#Remove non-printable chars
sed -i 's/[^a-zA-Z 0-9`~!@#$%^&*()_+\[\]\\{}|;'\'':",.\/<>?]//g' $dir/mdl.txt

#Move both files to a midway file
cat $dir/mdl.txt > $dir/midway.txt
cat $dir/yoyo.txt >> $dir/midway.txt

#Sort Midway file and remove duplicates
sort -u $dir/midway.txt

#Clean whitelist of white lines (can produce a blank end result otherwise)
grep -v '^$' $dir/whitelist.txt > $dir/whitelist_clean.txt

#Remove whitelisted entries from config file
cat $dir/whitelist_clean.txt | sed $'s/\r$//' | grep -vf - $dir/midway.txt > $dir/dnsmasq.conf.add

#Append our file to dnsmasq already created config
cat $dir/dnsmasq.conf.add >> /etc/dnsmasq.conf

logger -t $(basename $0) "DNS Postconf Completed!"

Inspiration for converting a regular host file from http://www.techist.com/forums/f43/using-sed-change-hosts-file-into-dnsmasq-config-form-231542/

I used IP 10.99.99.99 because I have an iptables filter (added by firewall-start) that redirects traffic to a pixelserv running on port 8080 of my router (10.124.56.1)
Code:
iptables -t nat -A PREROUTING --dest 10.99.99.99 -p tcp -m multiport --dport 80,443 -j DNAT --to-dest 10.124.56.1:8080

I added timeouts to wget to prevent the firmware from being locked much time in case no response, and I always save the last successful wget to use as a backup in those cases.

Changed the cron job from calling the update script to restart dnsmasq and I think that's it
 
I'm seeing mixed results with the advanced method, can anyone help verify I'm just crazy or if I did something wrong?

I have an RT-AC68U with firmware 378.55. I followed the advanced method and if I run /jffs/scripts/update-hosts.sh it reports "Done" and I see this in my router logs:

Nov 4 19:31:27 rc_service: service 11066:notify_rc restart_dnsmasq
Nov 4 19:31:27 dnsmasq[18756]: exiting on receipt of SIGTERM
Nov 4 19:31:27 custom config: Appending content of /jffs/configs/dnsmasq.conf.add.
Nov 4 19:31:27 dnsmasq[11069]: started, version 2.73rc9 cachesize 1500
Nov 4 19:31:27 dnsmasq[11069]: warning: interface ppp1* does not currently exist
Nov 4 19:31:27 dnsmasq[11069]: asynchronous logging enabled, queue limit is 5 messages
Nov 4 19:31:27 dnsmasq-dhcp[11069]: DHCP, IP range 192.168.0.2 -- 192.168.0.254, lease time 1d
Nov 4 19:31:27 dnsmasq[11069]: read /etc/hosts - 5 addresses
Nov 4 19:31:27 dnsmasq[11069]: read /tmp/mnt/sda1/hosts/blacklist.txt - 0 addresses
Nov 4 19:31:28 dnsmasq[11069]: read /tmp/mnt/sda1/hosts/hosts.blocked - 72267 addresses
Nov 4 19:31:28 dnsmasq[11069]: read /etc/hosts.dnsmasq - 0 addresses
Nov 4 19:31:28 dnsmasq-dhcp[11069]: read /etc/ethers - 3 addresses
Nov 4 19:31:28 dnsmasq[11069]: using nameserver ----#53 for domain local
Nov 4 19:31:28 dnsmasq[11069]: using nameserver ----#53 for domain ----.com
Nov 4 19:31:28 dnsmasq[11069]: using nameserver ----#53 for domain local
Nov 4 19:31:28 dnsmasq[11069]: using nameserver ----#53 for domain ----.com
Nov 4 19:31:28 dnsmasq[11069]: using nameserver ----#53
Nov 4 19:31:28 dnsmasq[11069]: using nameserver ----#53

So the logs would seem to indicate that it can read the files okay (except maybe that interface ppp1* warning?), but if I turn Adblock Plus off, I still see lots of ads, including on this site/page. This is after an ipconfig /flushdns in Windows 7 (not to mention I initially set this up on October 25th).

I turned on Firefox developer and watched the network and would see logs successfully contacting something like "cdn.flashtalking.com", then I grepped by hosts.blocked file and sure enough that exact line is in there.

If I ping on my router I see this:

/tmp/mnt/sda1/hosts# ping cdn.flashtalking.com
PING cdn.flashtalking.com (0.0.0.0): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.146 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.129 ms
64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.129 ms

and on my Windows desktop I'll see this:

>ping cdn.flashtalking.com

Pinging w3.b.cap-mii.net [216.38.162.128] with 32 bytes of data:
Reply from 216.38.162.128: bytes=32 time=47ms TTL=50
Reply from 216.38.162.128: bytes=32 time=44ms TTL=50
Reply from 216.38.162.128: bytes=32 time=44ms TTL=50
Reply from 216.38.162.128: bytes=32 time=45ms TTL=50

Ping statistics for 216.38.162.128:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 44ms, Maximum = 47ms, Average = 45ms

Any help is greatly appreciated, please let me know if I can provide any more information.
 
What do you get if you do an nslookup from your Windows PC?
Code:
nslookup cdn.flashtalking.com
>nslookup cdn.flashtalking.com
Server: google-public-dns-a.google.com
Address: 8.8.8.8

Non-authoritative answer:
Name: e2236.g.akamaiedge.net
Address: 96.6.157.139
Aliases: cdn.flashtalking.com
geo.flashtalking.com.akadns.net
us.wlb.flashtalking.com.akadns.net
cdn.flashtalking.com.edgekey.net

For what it's worth I see this on my router:

/tmp/mnt/sda1/hosts# nslookup cdn.flashtalking.com
Server: 127.0.0.1
Address 1: 127.0.0.1 localhost.localdomain

Name: cdn.flashtalking.com
Address 1: 0.0.0.0 0.0.0.0

And this on a debian box:
$ nslookup cdn.flashtalking.com
Server: 8.8.8.8
Address: 8.8.8.8#53

Non-authoritative answer:
cdn.flashtalking.com canonical name = geo.flashtalking.com.akadns.net.
geo.flashtalking.com.akadns.net canonical name = us.wlb.flashtalking.com.akadns.net.
us.wlb.flashtalking.com.akadns.net canonical name = cdn.flashtalking.com.edgekey.net.
cdn.flashtalking.com.edgekey.net canonical name = e2236.g.akamaiedge.net.
Name: e2236.g.akamaiedge.net
Address: 96.6.157.139

Edit: okay, while looking at the output I kept seeing the Google DNS server at 8.8.8.8 so I removed that entry and 8.8.4.4 and both the Windows machine and the debian box work as expected. I know the first post mentions it won't work with DNS filtering but I assumed that meant the parental controls stuff. Huh, thanks for helping, but I am curious why they don't/can't work in tandem?
 
Last edited:
Edit: okay, while looking at the output I kept seeing the Google DNS server at 8.8.8.8 so I removed that entry and 8.8.4.4 and both the Windows machine and the debian box work as expected. I know the first post mentions it won't work with DNS filtering but I assumed that meant the parental controls stuff. Huh, thanks for helping, but I am curious why they don't/can't work in tandem?
Your client PC was configured to go directly to Google's DNS servers and was therefore bypassing dnsmasq on the router (where the ad blocking takes place).
 
Hi @thelonelycoder Like I got the N66U, just upgraded to the 56-2 Merlin, formatted the usb drive, did not name it, and the dnsmasq.postconf does not work, gives me the same errors that you were helping me troubleshoot last week, but the dnsmasq.conf.add does work with the dnsmasq logging and dnscrypt just like when I had the 55, everything is working, VPN, adblocking, dnscrypt, but no way to get working the postconf file, I though it would be the naming of the USB drive, but did everything from scratch and even factory defaulted the router, and the same errors from last time... that it could not mount non existing drive/
But now everything is standard sda1.
Just to let you know.,
Thanks

Mine looks about the same.
Logging is the reason why we have to move contents from dnsmasq.conf.add to dnsmasq.postconf in the latest release of Merlin's firmware. As there is no mention of it in Post #1 I am not going to change it.
 
Just want to thanks the OP for this add blocker.
The only thing I miss is that I would love to see not loaded adds replaced by a fixed picture in order to see what was blocked.
 
Similar threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Staff online

Top