AdBlocking with combined hosts file

[Smacs]

First of all, thanks, this works really good. Nice way to get rid of ads especially on tablets or phones, lacking an adblock extension.

But to be somewhat paranoid - what are the chances that any of the three lists could be used for injecting a couple of malicious entries into our routers, redirecting some valid URL's (PayPal etc.) to a phishing server?

 

[thelonelycoder]

First of all, thanks, this works really good. Nice way to get rid of ads especially on tablets or phones, lacking an adblock extension.

But to be somewhat paranoid - what are the chances that any of the three lists could be used for injecting a couple of malicious entries into our routers, redirecting some valid URL's (PayPal etc.) to a phishing server?

None. Look at the script where it says 'grep -w ^0.0.0.0'
It will only add lines starting with 0.0.0.0. to the hosts.clean file.

 

[Weble]

Hey guys I have added a little extra into my script to grab host files from other sites that use the 127.0.0.1 standard. It will grab the file, convert the local host ip to 0.0.0.0 and the rest is the same. Warning this makes your host file quite large at 24mb ish but its working fine on my n66u

#!/bin/sh

# set directory
dir=/mnt/sda1
# get hosts files and combine and sort, write to temp file
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' > $dir/temp
#get hosts files combine and convert 127.0.0.1 to 0.0.0.0 and output to temp
wget -qO- "http://adaway.org/hosts.txt" "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://support.it-mate.co.uk/downloads/hosts.txt" "http://hostsfile.mine.nu/Hosts" | grep -w ^127.0.0.1 | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' >> $dir/temp
# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/temp | sort -u | fgrep -vf $dir/whitelist.txt > $dir/hosts.clean
rm $dir/temp

#restart dnsmasq to apply changes
sleep 3
service restart_dnsmasq

I also made a script to update the whitelist without having to rerun the whole script.

#!/bin/sh

# set directory
dir=/tmp/mnt/sda1
# remove whitelisted entries in temp and write final file, remove temp file
mv $dir/hosts.clean $dir/temp
cat $dir/temp | fgrep -vf $dir/whitelist.txt > $dir/hosts.clean
rm $dir/temp

#restart dnsmasq to apply changes
sleep 1
service restart_dnsmasq

 

[thelonelycoder]

Hey guys I have added a little extra into my script to grab host files from other sites that use the 127.0.0.1 standard. It will grab the file, convert the local host ip to 0.0.0.0 and the rest is the same. Warning this makes your host file quite large at 24mb ish but its working fine on my n66u

Thanks for the addition. I have done this before, and added my 2¢ to it in post 122.
I am going to run my router with this for a few days to see how it performs.
It uses quite a bit of RAM, some tests reveal this:

RAM readings on my RT-AC66U:
Fresh reboot, w/o scripts (standard merlin settings): 61MB
Fresh reboot, with standard hosts scripts as in post 1: 64MB
Fresh reboot, with standard hosts scripts, then ran updatehosts.sh with Weble's addition: 171MB
Fresh reboot, with Weble's addition: 136MB

Looking good so far, will report back after a few days use.

Edit: Just to clarify:
The 'http://support.it-mate.co.uk/downloads/hosts.txt' file is a mirror of hpHosts (http://hosts-file.net/download/hosts.txt)

Edit 2: To make Google Drive work again I had to add these to the whitelist:
#allow google drive
clients2.google.com
clients3.google.com
clients4.google.com
clients5.google.com

 

[enr00ted]

Any news with the last modifications/additions ?

 

[thelonelycoder]

Any news with the last modifications/additions ?

Are you referring to the one in the post above yours?
A follow up post is planned for today.

 

[thelonelycoder]

I have tested Weble's findings (in a post above) and have added an Advanced method in post #1 in this thread.
Since this thread is about adblocking I have only added pure Adblock lists and removed the outdated hosts files in his script.
The Advanced method supports a whitelist and a blacklist. Note that the original (Standard) and Advanced method are not compatible. You will have to delete the old scripts to run the Advanced method.

 

[tijaune]

Hi thelonelycoder,

In post 1, Advanced method of update-hosts.sh, there is a slight error missing .txt at the end, should be as follow...

# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked.txt

... this will match with /jffs/config/dnsmasq.conf.add.

 

[thelonelycoder]

Hi thelonelycoder,

In post 1, Advanced method of update-hosts.sh, there is a slight error missing .txt at the end, should be as follow...

... this will match with /jffs/config/dnsmasq.conf.add.

Thank you for reporting, it now matches the script, without the .txt extension as intended.

 

[enr00ted]

Are you referring to the one in the post above yours?
A follow up post is planned for today.

Yes, I'll give it a shot later

 

[decedion]

I have done this modifications:

/jffs/scripts/post-mount

#!/bin/sh

if [ $1 = "/tmp/mnt/usb" ]
then
  ln -sf $1/entware /tmp/opt

  wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext&useip=0.0.0.0" "http://hosts-file.net/ad_servers.asp" "http://adaway.org/hosts.txt" "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://hosts-file.net/download/hosts.txt" "http://hostsfile.mine.nu/Hosts"  | egrep -w "^(127.0.0.1)|^(0.0.0.0)" | sed '/0.0.0.0 0.0.0.0/d' | sed 's/\r$//' | sed 's/127.0.0.1/0.0.0.0/' | sed '/localhost/d' | LC_ALL=C sort -fiu -k 2 > $1/hosts.clean
  sleep 5
  service restart_dnsmasq
fi

NOTE: The file created is big (25.7M).

There is no need of sleep before creating the adblock file if we include that process in 'if [ $1 = "/tmp/mnt/sda1" ]...'. In addition thus we ensure that the unit is mounted and, in case of several connected units, the process only it executes one time.
I don't sure if it's required the 'sleep' after 'wget...'.

/jffs/scripts/services-start

#!/bin/sh....
cru a reboot "0 6 */5 * * reboot"
....

With this change every 5 days at 6 o'clock in the morning the router is rebooted and the adblock file is regenerated.
-------------------------------------
If you don't want reboot the router every n days but you want update the file every n days, you have to create a file with the next code:
/jffs/scripts/create-adblock-file

#!/bin/sh
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext&useip=0.0.0.0" "http://hosts-file.net/ad_servers.asp" "http://adaway.org/hosts.txt" "http://www.malwaredomainlist.com/hostslist/hosts.txt" "http://hosts-file.net/download/hosts.txt" "http://hostsfile.mine.nu/Hosts"  | egrep -w "^(127.0.0.1)|^(0.0.0.0)" | sed '/0.0.0.0 0.0.0.0/d' | sed 's/\r$//' | sed 's/127.0.0.1/0.0.0.0/' | sed '/localhost/d' | LC_ALL=C sort -fiu -k 2 > $1/hosts.clean
sleep 5
service restart_dnsmasq

And you have to change
/jffs/scripts/post-mount

#!/bin/sh

if [ $1 = "/tmp/mnt/usb" ]
then
  ln -sf $1/entware /tmp/opt

  . /jffs/scripts/create-adblock-file
fi

And you have to change
/jffs/scripts/services-start

#!/bin/sh....
cru a update-adblock "0 6 */5 * * /jffs/scripts/create-adblock-file"
....

You can change the first 2 numbers by the minutes and hours respectively and the third number by number of days to update.
Regards.

 

[thelonelycoder]

NOTE: The file created is big (25.7M).

There is no need of sleep before creating the adblock file if we include that process in 'if [ $1 = "/tmp/mnt/sda1" ]...'.
I don't sure if it's required the 'sleep' after 'wget...'.

Thanks for that decedion. I am no fan of that big of a hosts file. It introduces too many blockages for sites that I never even visit, even as a third party by cookies. I did do a test for about a week with the HP hosts file and some others (see earlier post in this thread). I had to constantly add hosts to the whitelist to get sites and services working in my LAN. I am now running the Advanced method as described in Post #1 and have had no issues so far.
The sleep times are there for another reason. I want to give some breathing time for the cpu while it is starting up. Downloading, sed, combining and sort are quite memory and cpu heavy operations. That is the case especially with the 25MB hosts file you have. Give it some slack so it can do it's other jobs as well.

 

[decedion]

The sleep times are there for another reason. I want to give some breathing time for the cpu while it is starting up. Downloading, sed, combining and sort are quite memory and cpu heavy operations. That is the case especially with the 25MB hosts file you have. Give it some slack so it can do it's other jobs as well.

ok for the sleep after the process of sort, combine, etc. but ¿before?

 

[thelonelycoder]

ok for the sleep after the process of sort, combine, etc. but ¿before?

I agree with you, the sleep before running the scripts is not really needed, I just have it there because I've had scripts not running at reboot without it and just left it in for the time being. I may remove it after tests with the next merlin release (.53).

 

[tijaune]

I just used the advanced method on rt-n16 running with the last supported merlin 376.49_5. Normal method has been running for some weeks already. Tools page still show 64 MB free memory. Will need feedback from my old man if he experience slowdown or not.

 

[thelonelycoder]

Will need feedback from my old man if he experience slowdown or not.

If he does, let me know. I'll handle the complaints.

 

[thelonelycoder]

rt-n16 running with the last supported merlin 376.49_5

How does the reboot with the new scripts work? I am asking because update-hosts.sh is started in the services-start file at boot.

 

[tijaune]

How does the reboot with the new scripts work? I am asking because update-hosts.sh is started in the services-start file at boot.

oh, I did not reboot his router because I was using TeamViewer to help him clean remotely. However I expected it to work as on mine (ac66u), I remember it works correctly.

 

[GregS]

I'm on an Asus RT-AC68 and for some reason this line just produces a blank hosts.blocked file:
# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked

There's no error and the rest of the script works fine. If I go through the process line by line I can see the temp file does exists and has tons of normal looking entries. I do have a whitelist.txt as specified in post #1 with a few dummy domains.

Any ideas?

 

[andyxm715]

Big thanks to thelonelycoder & decedion got ad blocking working on my RT-AC56U