• SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

AdBlocking with combined hosts file

Thanx so much lonelycoder, I followed your easy to follow advanced tutorial, I think it works. If I ping a a random host in the list after running update-hosts.sh, the ping fails. So I think its working. I flushed the router's dnsmasq, also I had to do this in my OS terminal- Ubuntu or Kubuntu
Code: 
sudo /etc/init.d/dns-clean start
and flush the browser cache before http://ads-blocker.com/testing/ came back clean.
Edit: Thanks for including a link in the script that gets the "Windows 10 reporting domains". I don't have such devices on my network yet as far as I know, but its only a matter of time.
 
Last edited:
Thanks, lonelycoder for putting this together.

I've made some changes to the advanced version of /jffs/scripts/update-hosts.sh so I thought I'd share them.

This lets you easily add or remove block lists by commenting or uncommenting them. Also, it merges in your blacklist with the rest of the blocked sites and ensures uniqueness.

Code: 
#!/bin/sh

# set directory
dir=/tmp/mnt/sda1/hosts

### Sources with 0.0.0.0(uncomment desired blocklists)
#SOURCESA="$SOURCESA http://winhelp2002.mvps.org/hosts.txt" 
#SOURCESA="$SOURCESA http://someonewhocares.org/hosts/zero/hosts" 
SOURCESA="$SOURCESA http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0"

### Sources with 127.0.0.1(uncomment desired blocklists)
# MalwareDomainList.com Hosts List
SOURCESB="$SOURCESB http://www.malwaredomainlist.com/hostslist/hosts.txt" 
# hpHosts - Ad and Tracking servers only
SOURCESB="$SOURCESB http://hosts-file.net/ad_servers.txt"
# AdAway default blocklist
# Blocking mobile ad providers and some analytics providers
SOURCESB="$SOURCESB http://adaway.org/hosts.txt"


### OPTIONAL: I keep my whitelist and black list on an external site so I can edit them from anywhere without having to go into router settings again
#wget -O- http://url_to_my/whitelist.txt > $dir/whitelist
#wget -O- http://url_to_my/blacklist.txt > $dir/blacklist

# get hosts files and combine and sort, write unique list to temp file
wget -O- $SOURCESA | grep -w ^0.0.0.0 | sed s/0.0.0.0/0.0.0.0/g |sed $'s/\r$//' | sort -u > $dir/temp

# get hosts files combine and convert 127.0.0.1 to 0.0.0.0 and add unique list to temp
wget -O- $SOURCESB | grep -w ^127.0.0.1 | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sort -u >> $dir/temp

# Get personal blacklist and combine with the others
cat $dir/blacklist | sed $'s/\r$//' >> $dir/temp

# Be sure entire blocklist is free of double whitespace and unique
cat $dir/temp | sed 's/  */\ /g' | sort -u > $dir/tempblack

# remove whitelisted entries in tempblack and write final file, remove temp and tempblack files
cat $dir/whitelist | sed $'s/\r$//' | sort -u | grep -vf - $dir/tempblack > $dir/hosts.blocked

#remove temp files
rm $dir/temp
rm $dir/tempblack

#restart dnsmasq to apply changes
sleep 1
service restart_dnsmasq

If you use this method, you can remove the line about your blacklist in /jffs/configs/dnsmasq.conf.add.
It should now look like:
Code: 
# AdBlocking
address=/0.0.0.0/0.0.0.0
ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked

Cheers
 
Code: 
### OPTIONAL: I keep my whitelist and black list on an external site so I can edit them from anywhere without having to go into router settings again
#wget -O- http://url_to_my/whitelist.txt > $dir/whitelist
#wget -O- http://url_to_my/blacklist.txt > $dir/blacklist


# Get personal blacklist and combine with the others
cat $dir/blacklist | sed $'s/\r$//' >> $dir/temp

# remove whitelisted entries in tempblack and write final file, remove temp and tempblack files
cat $dir/whitelist | sed $'s/\r$//' | sort -u | grep -vf - $dir/tempblack > $dir/hosts.blocked

I think you may need to change the reference to whitelist and blacklist to whitelist.txt and blacklist.txt if you already have those in your hosts directory?
 
elorimer said:
Code: 
### OPTIONAL: I keep my whitelist and black list on an external site so I can edit them from anywhere without having to go into router settings again
#wget -O- http://url_to_my/whitelist.txt > $dir/whitelist
#wget -O- http://url_to_my/blacklist.txt > $dir/blacklist

I think you may need to change the reference to whitelist and blacklist to whitelist.txt and blacklist.txt if you already have those in your hosts directory?
Click to expand...
I keep them as whitelist.txt and blacklist.txt on my ftp server. WGET pulls them down as whitelist and blacklist. It overwrites the ones that are there or creates new ones if not.
 
I'm sorry, I am very new to this, please help me how to do it, I just flashed the ASUS Merlin and dont know how to make hosts file works.
thanks
TechLifeWeb said:
Thanks, lonelycoder for putting this together.

I've made some changes to the advanced version of /jffs/scripts/update-hosts.sh so I thought I'd share them.

This lets you easily add or remove block lists by commenting or uncommenting them. Also, it merges in your blacklist with the rest of the blocked sites and ensures uniqueness.

Code: 
#!/bin/sh

# set directory
dir=/tmp/mnt/sda1/hosts

### Sources with 0.0.0.0(uncomment desired blocklists)
#SOURCESA="$SOURCESA http://winhelp2002.mvps.org/hosts.txt"
#SOURCESA="$SOURCESA http://someonewhocares.org/hosts/zero/hosts"
SOURCESA="$SOURCESA http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0"

### Sources with 127.0.0.1(uncomment desired blocklists)
# MalwareDomainList.com Hosts List
SOURCESB="$SOURCESB http://www.malwaredomainlist.com/hostslist/hosts.txt"
# hpHosts - Ad and Tracking servers only
SOURCESB="$SOURCESB http://hosts-file.net/ad_servers.txt"
# AdAway default blocklist
# Blocking mobile ad providers and some analytics providers
SOURCESB="$SOURCESB http://adaway.org/hosts.txt"


### OPTIONAL: I keep my whitelist and black list on an external site so I can edit them from anywhere without having to go into router settings again
#wget -O- http://url_to_my/whitelist.txt > $dir/whitelist
#wget -O- http://url_to_my/blacklist.txt > $dir/blacklist

# get hosts files and combine and sort, write unique list to temp file
wget -O- $SOURCESA | grep -w ^0.0.0.0 | sed s/0.0.0.0/0.0.0.0/g |sed $'s/\r$//' | sort -u > $dir/temp

# get hosts files combine and convert 127.0.0.1 to 0.0.0.0 and add unique list to temp
wget -O- $SOURCESB | grep -w ^127.0.0.1 | sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sort -u >> $dir/temp

# Get personal blacklist and combine with the others
cat $dir/blacklist | sed $'s/\r$//' >> $dir/temp

# Be sure entire blocklist is free of double whitespace and unique
cat $dir/temp | sed 's/  */\ /g' | sort -u > $dir/tempblack

# remove whitelisted entries in tempblack and write final file, remove temp and tempblack files
cat $dir/whitelist | sed $'s/\r$//' | sort -u | grep -vf - $dir/tempblack > $dir/hosts.blocked

#remove temp files
rm $dir/temp
rm $dir/tempblack

#restart dnsmasq to apply changes
sleep 1
service restart_dnsmasq

If you use this method, you can remove the line about your blacklist in /jffs/configs/dnsmasq.conf.add.
It should now look like:
Code: 
# AdBlocking
address=/0.0.0.0/0.0.0.0
ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked

Cheers
Click to expand...
 
@TechLifeWeb What is the advantage of having them stored in a publicly accessible place vs the routers own file system?
You'd have to log into your ftp server to change your files, just as you have to on your router. Then log into the router to run an update or reboot as restarting dnsmasq will not pull the files.
I fail to see why that's more convenient for you.
 
after I create the hosts.clean file, how can I apply it as hosts file of the router, because I still cannot block ads.
I run this
address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/N/hosts.clean
and the error was -sh: addn-hosts=/tmp/mnt/N/hosts.clean: not found
thanks
(run code with terminal in OSX)
 
nhan72nn said:
after I create the hosts.clean file, how can I apply it as hosts file of the router, because I still cannot block ads.
I run this
address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/N/hosts.clean
and the error was -sh: addn-hosts=/tmp/mnt/N/hosts.clean: not found
thanks
(run code with terminal in OSX)
Click to expand...
You do it completely wrong. The code you mention above goes into the files on the router. They are not meant to be run in a terminal. Read the instructions carefully. You need to download a sftp client like FileZilla, they have a Mac version. With it you log into the router and create the files and insert the code into them.
 
I did use putty to make it, but it seems doesnt work at all cuz the pages I locked still go thru...
i download the filezilla to use on mac, type the IP, port 22 (ssh), but cannot establish the connect...
I relly dont know how to make it work, please help

thelonelycoder said:
You do it completely wrong. The code you mention above goes into the files on the router. They are not meant to be run in a terminal. Read the instructions carefully. You need to download a sftp client like FileZilla, they have a Mac version. With it you log into the router and create the files and insert the code into them.
Click to expand...
 
nhan72nn said:
I did use putty to make it, but it seems doesnt work at all cuz the pages I locked still go thru...
i download the filezilla to use on mac, type the IP, port 22 (ssh), but cannot establish the connect...
I relly dont know how to make it work, please help
Click to expand...

Did you enable SSH under the Administration->System tab?
 
Zirescu said:
Did you enable SSH under the Administration->System tab?
Click to expand...
Yes, I did...if you have done it, would you like to show me after I edit file in jffs and etc, usb...how can I apply Adblock, I read the article and tried to do the same but I still get ads and I checked by goto a url which is listed in hosts.clean but the connection still go thru
 
thelonelycoder said:
You need to download a sftp client like FileZilla, they have a Mac version. With it you log into the router and create the files and insert the code into them.
Click to expand...
AFAIK sftp is not supported, only ssh and scp. Filezilla won't help there. Use PuTTY or WinSCP.
 
@ColinTaylor I read that Filezilla falls back to scp if it cannot connect with sftp. I can't check it atm as I'm away from home.
I use WinSCP and connect with sftp. Entware has the package.
 
So I have been trying to get this to work for 6 hours now, and still nothing, ether nothing happens or I can't connect to the internet.

I have the files in "/tmp/mnt/TEST/jffs" ("sda1" does not exist, my USB name is "TEST") with "dnsmasq.conf.add" file in config folder and "update-hosts.sh", "services-start", "post-mount" in scrips folder, I replace sda1 with TEST. run "chmod a+rx /jffs/scripts/*" and restarted the router.

If I try to run "sh /jffs/scripts/update-hosts.sh" I get,

admin@RT-N66U/# sh /jffs/scripts/update-hosts.sh
: Invalid argumentte-hosts.sh: line 2: can't create /tmp/mnt/TEST/hosts.clean

Done.
admin@RT-N66U:/#

I tried with both the name of my USB (TEST) and sda1 neither work.
 
ColinTaylor said:
Wrong. Look again at post #1. The scripts and config files go in /jffs/scripts and /jffs/config. The USB drive (/tmp/mnt/TEST) is used to store the downloaded block lists.
Click to expand...

I read "The files are stored on a USB disk" as where the files I created should be.

Anyway, I moved the files to /jffs/ and did "chmod a+rx /jffs/scripts/*" restarted router and I wasn't able to connect to the internet. I created the files with Putty with text editor vi and just copy pasted the correct text.

Edit: The exact files I have.

/jffs/scripts/post-mount

Code: 
#!/bin/sh
sleep 12
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/TEST/hosts.clean
sleep 5
service restart_dnsmasq


/jffs/configs/dnsmasq.conf.add
Code: 
address=/0.0.0.0/0.0.0.0
addn-hosts=/tmp/mnt/TEST/hosts.clean

/jffs/scripts/update-hosts.sh
Code: 
#!/bin/sh
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/TEST/hosts.clean
sleep 5
service restart_dnsmasq

/jffs/scripts/services-start
Code: 
#!/bin/sh

cru a UpdateHosts "00 00 1 * * /jffs/scripts/update-hosts.sh"

and I did run "chmod a+rx /jffs/scripts/*" before I rebooted.
 
Last edited:
OK. That looks correct. What happens if you try and run the script from the command line?
Code: 
/jffs/scripts/update-hosts.sh
 
You must log in or register to reply here.

Similar threads

MegaMango
dnsmasq-surrogate for RT-BE88U: Ad Blocking & Security Enhancement
Replies
2
Views
726
MegaMango
MegaMango
M
Diversion Diversion failed to update. Check the Diversion-web-update.log with sf in the SSH UI.
Replies
4
Views
350
mikefrommanchester
M
P
ASUSwrt dnsmasq addn-hosts entry limit
Replies
0
Views
595
pseu_asus
P
S
Wireguard VPN change routes ip
Replies
0
Views
726
S75
S
JGrana
chatai - version two - have a chat session with either Googles Gemini AI or Anthropics Claude (or both at once)
Replies
4
Views
1K
JGrana
JGrana

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Members online

Total: 603 (members: 35, guests: 568)
Top