AdBlocking with combined hosts file

[thelonelycoder]

AB-Solution is now Diversion
Please check the Diversion website for latest version and discussion link.

These snippets of code posted here are now a automated install script with control center character under the name of AB-Solution.

This is a simple but effective way to block ads on all your devices
------------------------------------------------------------
Changelog
------------------------------------------------------------
September 06, 2015: Added Quick Enable / Disable Adblocking, TOC added
August 31, 2015: Removed requirement for Entware, it was never really needed.
April 10, 2015: Typo, changed addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked.txt to [...]/hosts.blocked in script dnsmasq.conf.add, thanks tijaune.
April 08, 2015: Added Advanced method
March 31, 2015: Added preparatory steps for jffs
March 21, 2015: Simplified install steps.
March 20, 2015: Adding the cron job with /jffs/scripts/services-start instead of /jffs/scripts/init-start for compatibility with latest firmware.
March 17, 2015: If DNS-Filtering is enabled in your routers interface AdBlocking will not work. The DNS Filter bypasses Dnsmasq settings. See this thread for details.
April 30, 2014: If you want to remove some domains appearing in the hosts.clean you can do this with a whitelist file. See post 65 in this thread.
If you want to go even further with my ad-blocking solution, e2301010 has posted AdBlocking with combined hosts file with Lighttpd.
March 14, 2014: wget scripts are modified to reflect the much better and smarter findings of decedion.

------------------------------------------------------------
Table of content
------------------------------------------------------------
Indroduction
1. Standard method (the original)
2. Advanced method
3. Notes
4. Quick Enable / Disable Adblocking
------------------------------------------------------------

Introduction

This combines well maintained hosts files into one, with no duplicate entries.
The resulting file is about 728 KB (1450 KB Advanced method) containing more than 26000 (52600) lines of blocked domains.

Preparatory steps for both methods:
In Administration / System set to Yes:
- Enable JFFS partition*
- Enable JFFS custom scripts and configs
*) If this has not been set before set 'Format JFFS partition at next boot' to Yes and Reboot.

In this post two ways to use the combined hosts file are explained, the Standard and Advanced method. The Standard method is the original method, updated as per the Changelog. On April 04, 2015 the Advanced method was added with additional hosts files and support for a black- and whitelist of domains.
Note: The two methods are not compatible, use only one in your router.

1. Standard method (the original)

The files are stored on a USB disk, /tmp/mnt/sda1/ is the path. If you use /jffs/ edit accordingly.

Using a USB disk we need to delay the startscript.
This will create an up-to-date hosts file after every reboot.
Add to /jffs/scripts/post-mount

#!/bin/sh
sleep 12
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 5
service restart_dnsmasq

Tell dnsmasq to add the new hosts file and where to send the unwanted requests to (sending them to 0.0.0.0)
Create or add to /jffs/configs/dnsmasq.conf.add

address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts.clean

To update the hosts file (manually or as a cron job) create /jffs/scripts/update-hosts.sh

#!/bin/sh
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 5
service restart_dnsmasq

To run update-hosts.sh every Friday midnight as a cron job create or add to /jffs/scripts/init-start /jffs/scripts/services-start

#!/bin/sh

cru a UpdateHosts "00 00 * * 5 /jffs/scripts/update-hosts.sh"

Set all files in /jffs/scripts/ as executable

chmod a+rx /jffs/scripts/*

That's it, reboot and enjoy!
Tested on a RT-AC66U.

2. Advanced method

This method supports a black- and a whitelist for domains you want to include/exclude in the hosts file. It uses 5 well maintained hosts files (Standard method uses 3). Thank you Weble for the input in post #143.
Note: The code below is NOT compatible with the Standard method!

This will create an up-to-date hosts file after every reboot and update the hosts file with cron every Friday at midnight.
Note that the files are stored in /tmp/mnt/sda1/hosts/. Edit all references in the code if you use a different path.
Add this to /jffs/scripts/services-start

#!/bin/sh

sleep 10
cru a UpdateHosts "00 00 * * 5 /jffs/scripts/update-hosts.sh"

sleep 5
source  /jffs/scripts/update-hosts.sh

Create or add to /jffs/configs/dnsmasq.conf.add

# AdBlocking
address=/0.0.0.0/0.0.0.0
ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked
addn-hosts=/tmp/mnt/sda1/hosts/blacklist.txt

To update the hosts file (to run it manually, and automatically as a cron job and when rebooting the rooter) create /jffs/scripts/update-hosts.sh

#!/bin/sh

# set directory
dir=/tmp/mnt/sda1/hosts

# get hosts files and combine and sort, write to temp file
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > $dir/temp

#get hosts files combine and convert 127.0.0.1 to 0.0.0.0 and output to temp
wget -qO- "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://hosts-file.net/ad_servers.txt" | grep -w ^127.0.0.1 | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' >> $dir/temp

# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked

#remove temp file
rm $dir/temp

#restart dnsmasq to apply changes
sleep 1
service restart_dnsmasq

Set all files in /jffs/scripts/ as executable

chmod a+rx /jffs/scripts/*

Create /tmp/mnt/sda1/hosts/whitelist.txt and fill it with domains you want to whitelist, one domain per line. An example would look like so:

clients2.google.com
something.other.com
anotherdomain.com

Create /tmp/mnt/sda1/hosts/blacklist.txt and fill it with domains you want to blacklist that are not in the combined hosts file, one domain per line. Add 0.0.0.0 to every line as in this example:

0.0.0.0 www.rivalo.com
0.0.0.0 rivalo.com
0.0.0.0 madpandatv.net
0.0.0.0 www.snapdo.com

These two files (whitelist.txt and blacklist.txt) need to be present for the scripts to work. Leave the files empty if you have no white/blacklist domains.

Reboot and enjoy!

3. Notes

A note for Windows users: You need to flush the DNS Cache when you tinker with your Router.
In a command prompt enter ipconfig/flushdns and hit Enter.

On Windows 7 or 8 Systems this generally needs Administrator privileges (hit Windowskey and type cmd. In the list, right-click on cmd (.exe) and choose 'run as Administrator').
Also, clear your browsers cache after this.

4. Quick Disable / Enable Adblocking

To temporarily disable the hosts files remove the following, depending on which method you use:

1. Standard method (the original)
in /jffs/configs/dnsmasq.conf.add remove this:

address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts.clean

2. Advanced method
in /jffs/configs/dnsmasq.conf.add remove this:

# AdBlocking
address=/0.0.0.0/0.0.0.0
ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked
addn-hosts=/tmp/mnt/sda1/hosts/blacklist.txt

And then restart dnsmasq (or reboot the router):

service restart_dnsmasq

 

[r00t4rd3d]

So this blocks for all clients connected to the router? Wouldnt it slow everything down considerably?

 

[sinshiva]

just tried out the scripts, everything works great. thank you!

 

[thelonelycoder]

So this blocks for all clients connected to the router? Wouldnt it slow everything down considerably?

As this is a simple hosts file and dnsmasq handles it, yes it affects all clients connected to the router. This is exactly what I wanted it for. Almost free of ads on all devices in my household. I use a varied mix of Linux, Windows, Mac, iOS and Android devices. No need to install AdBlocker on any of them as it conveniently blocks almost everything, even InApp Ads. I love it!

As for the performance I am happy as it is. Woud be great to have a free performance tool to stress the routers dns resolving. I am not aware of one.

just tried out the scripts, everything works great. thank you!

You are welcome. Glad it works for you.

 

[dancemonkey]

Working well on N66U here. Thanks!

 

[L&LD]

I would love to try this myself, but is this as step by step as can be?

I'm not usually as dumb as I feel when I look at post #1, but I really don't know where to begin to get this going on my RT-N66U with the RMerlin 374.39_0-em firmware installed.

Any baby steps walkthrough for a coding newbie?

 

[sinshiva]

sure.

it'll be easiest to do this from an ssh client, so be sure you have ssh enabled and are using something such as PuTTY.

setting up entware is very simple, once logged in, just run;

entware-setup.sh

i always used nano when i was first using linux, but i've since forced myself to learn vi. to keep it simple; after installing entware, you use entware's packagemanager 'opkg' to install it.

opkg install nano

you can also check out what other things are available through the package manager by using 'opkg list' or 'opkg search blah'. you can also do something like 'opkg list | grep whatever' as a means of searching. to make it easier to read the whole list, you can use 'opkg list | less'

to use nano, you run the command like thus to create/edit a file;

nano /jffs/scripts/init-start

to exit and save, you first use CTRL + X and confirm by using SHIFT + Y

anyway, you can copy/paste the scripts into nano. if you get any whitespace at the beginning of any of the lines in the script, remove it, so that they look more or less exactly like the examples above.

entware creates some scripts in /jffs/scripts when it's installed, so when you are adding to the /jffs/scripts/post-mount script, for example, either replace the whole thing with thelonelycoder's script, or be sure to remove the following if you see it in the script twice; this is what's generated by the entware install:

if [ $1 = "/tmp/mnt/sda1" ]
then
  ln -sf $1/entware /tmp/opt
fi

i think that covers all the important stuff.

oh, and not sure if you know about tab completion, but when you're at the command line, you can type 'ent' and hit TAB and it'll replace it with entware-setup.sh

 

[L&LD]

sinshiva,

I still feel just as 'duh' as ever looking at this stuff - but now, I have enough power to be dangerous!

I hope to get a chance to try this in the next few days.

Thanks again.

 

[thelonelycoder]

I still feel just as 'duh' as ever looking at this stuff - but now, I have enough power to be dangerous!

Relax, there is a human oriented way just for the lazy butt types. Forget about the nano stuff and PuTTY.

Lean back and read first.

Instead of PuTTY use mRemoteNG, an Open Source SSH, VNC ,Telnet, what have you client. It even stores ssh passwords and has a clean usable Interface. The PuTTY dev's never get that these are a must have these days. I will not budge. Key files are supported as well as Pageant (for the hard core guys).

To transfer and edit files on the router use WinSCP so you don't have to use vi or nano on a ssh terminal. WinSCP runs over ssh but needs SFTP Server on the router installed. So install it after logging in with ssh:

opkg install openssh-sftp-server

Then connect with WinSCP and suddenly you feel at home.
Now you can create and edit files more easily.

While we're at it, use Notepad++ as a replacement of the standard Editor in Windows. In WinSCP under Preferences / Editor add Notepad++.

Hope this helps.

 

[Martineau]

I would love to try this myself, but is this as step by step as can be?

I'm not usually as dumb as I feel when I look at post #1, but I really don't know where to begin to get this going on my RT-N66U with the RMerlin 374.39_0-em firmware installed.

Any baby steps walkthrough for a coding newbie?

Sinshiva's post is excellent, but if you don't want to initially go anywhere near the arcane unix editors and you are on a Wintel box....

1. Install WINSCP and configure it for an SCP connection to the router and save the admin/password credentials (unless you want to type them for every connection!)

2. Navigate in the right-hand pane to /jff/scripts

3. If the file you want to edit already exists then simply double-click to open the file in edit mode

   otherwise SHIFT+F4 will ask you for a new name for your script to edit.

4. Paste (or tediously manually type) in the new scipt lines as required.

5. Simply close the file 'intuitively' none of this weird CTRL and SHIFT key combo nonsense! :-)

6. If this is a new file, then right click and select 'properties',and make sure all 9 R/W/X boxes are ticked - should show octal=0777.

7. Test script by right-clicking file, select 'custom commands' then click  'execute'

    or to see the results in a window select 'custom commands' then click 'enter....'
        and tick the box 'show results in terminal'  then type name of the script you want to run prefixed with the two characters  './'

Learning vi/nano and the unix shell is rewarding, but if you prefer point and click (and use Windows) then this is the way to get started!

Regards,

 

[r00t4rd3d]

WinSCP runs over ssh but needs SFTP Server on the router installed.

You can use WinSCP with the SCP protocol and not have to install anything to use it.

 

[jakey]

I'd like try this but I can't install entware, I have put a small SD card in the internal slot of my RT-N66U and it auto mounts as sda1, the problem is it's formatted as fat32 and I know it needs to be ext2/3.

I umount the card and try to format it with mkfs.ext3 /mnt/sda1 but it doesn't work.

Where am I messing up ?

thanks

 

[thelonelycoder]

Learning vi/nano and the unix shell is rewarding, but if you prefer point and click (and use Windows) then this is the way to get started! Regards,

We are not all bearded *nix Gurus here. Some want to work in a more relaxed environment. But we still code.

Looking through the generated hosts.clean file I noticed some of Dan Pollock's (of soneonewhocares.org fame) comments in his hosts file have a different formatting. Some comments are not stripped out.

There's no need to but if you want to replace

sed -i '/::1/d;/^#/d;/127.0.0.1    .*local/d;/255./d;/fe80/d;s/# .*//;s/127.0.0.1/0.0.0.0/' /tmp/mnt/sda1/whocares.org.hosts

with

sed -i '/::1/d;/^#/d;/127.0.0.1    .*local/d;/255./d;/fe80/d;s/# .*//;s/#.*//;s/127.0.0.1/0.0.0.0/' /tmp/mnt/sda1/whocares.org.hosts

The original post in this thread (#1) is updated.

 

[thelonelycoder]

I umount the card and try to format it with mkfs.ext3 /mnt/sda1 but it doesn't work.

Where am I messing up ?

You're very close to success. mkfs.ext2 will work.
It will only work with ext2 file system.

 

[jakey]

You're very close to success. mkfs.ext2 will work.
It will only work with ext2 file system.

thanks for reply m8 but the result is the same

admin@RT-N66U:/tmp/home/root# mkfs.ext2 /mnt/sda1
mke2fs 1.42.8 (20-Jun-2013)
/mnt/sda1 is not a block special device.
Proceed anyway? (y,n) y
mkfs.ext2: Device size reported to be zero.  Invalid partition specified, or
        partition table wasn't reread after running fdisk, due to
        a modified partition being busy and in use.  You may need to reboot
        to re-read your partition table.

Should I being doing something first with fdisk ?

The problem seems to be that once sda1 is unmounted sda1 just becomes a folder and not a device.

 

[sinshiva]

You're very close to success. mkfs.ext2 will work.
It will only work with ext2 file system.

though ext2 is probably better for an sdcard, you can use;

mke2fs -t ext3 /dev/sda1

for ext3.

 

[sinshiva]

thanks for reply m8 but the result is the same

admin@RT-N66U:/tmp/home/root# mkfs.ext2 /mnt/sda1
mke2fs 1.42.8 (20-Jun-2013)
/mnt/sda1 is not a block special device.
Proceed anyway? (y,n) y
mkfs.ext2: Device size reported to be zero.  Invalid partition specified, or
        partition table wasn't reread after running fdisk, due to
        a modified partition being busy and in use.  You may need to reboot
        to re-read your partition table.

Should I being doing something first with fdisk ?

ah, use /dev/sda1 rather than /mnt/sda1

[edit1] forgot to say; only for formatting the partition. when it's mounted, it'll be accessed via /mnt/sda1 [/edit1]
[edit2/]
you may need to repartition to set the partition to linux type 83, etc.

fdisk /dev/sda (no number)

and just use the help menu, it's pretty easy if you are only creating a single partition on the drive. delete the existing partition, create a new one with type 83 using all the available space and be sure to write the partition table when you're done

 

[jakey]

Thanks both for your help, got there in the end and entware installed.

Will finish the adblocking part tomorrow, need some shuteye now

 

[Zirescu]

Just finished deploying this to my RT-N66U but using the SD card and some minor changes to the mount points and works like a charm. No more annoying ads showing up when I run Skype. And most of the web sites I visit are now ad free on all of the mobile devices. Thank you for sharing theses instructions!

 

[jakey]

I'd also like to thank thelonelycoder for doing the work and sharing this, all up and running now on my RT-N66, all my browsing seems to ad free now but the 'in app' ads on my Lumia920 Windows phone 8 remain.

Thanks also to sinshiva for helping me sort out the SD card issue I had, perhaps you could help me with one thing more

I always have to power off and on the router to reboot otherwise if using the CLI or GUI to reboot the SD card always mounts as sdb1, now with my Linux STB I would use the fstab to make sure devices mount the same every time.

As the fstab is created dynamically on boot I'm not sure how to go about this so that the SD card always mounts as sda1 regardless of how the Asus is rebooted.

BTW I always use WinSCP for creating files and use it's internal editor which is quite adequate for simple jobs like this, WinSCP can also be configured to automatically run and login with Putty so with a double click i'm into the file system with Putty loaded and logged in.