AdBlocking with combined hosts file

[emNewB]

I have no idea, what websites are we talking about (if they are nsfw pm me...)?

I've fixed it on my router and it now works on my OpenVPN'd computer and her home computer. I like your thinking but unfortunately they were SWF. lol

coupons.com -> Whitelist -> cdn.cpnscdn.com & coupons.com
weeklyad.target.com -> Whitelist -> weeklyad.target.com

My main point of asking is to find out if this adblock script is supposed to apply to OpenVPN connections too? Or is it only being applied since I have the "Respond to DNS" & "Advertise DNS to Clients" enabled during my OpenVPN session?

 

[thelonelycoder]

@emNewB your assumtion and DNS settings are correct, without these a VPN circumvents the local dns.

 

[emNewB]

@emNewB your assumtion and DNS settings are correct, without these a VPN circumvents the local dns.

Thank you!

 

[mcfrojd]

Is there any way to whitelist for specific sites only?
If I want to whitelist bs.serving-sys.com on only one site (tv3play.se)
But not on other sites?

 

[thelonelycoder]

Is there any way to whitelist for specific sites only?
If I want to whitelist bs.serving-sys.com on only one site (tv3play.se)
But not on other sites?

Nope

 

[zekesdad]

can someone try drudgereport.com and see if there are ads? im watching the log, and it seems to be blocking some content on a couple other pages, but drudgereport seem to have ads all over the place.

 

[ColinTaylor]

can someone try drudgereport.com and see if there are ads? im watching the log, and it seems to be blocking some content on a couple other pages, but drudgereport seem to have ads all over the place.

I'm not seeing any ads at all.

 

[kenhlan]

Just wanted to say thank you. Using the advanced version. For some reason an earlier version of the scripts stopped working on my latest FW upgrade, file not found, etc. Might have been a version of ColinTaylor's I used/modified a few months ago which served me well until the compatibility issue.

Changing over was pretty much cut and paste/copy and save with WinSCP.

In any case the advance version is working on my AC68 with Merlin 378.55

Happy Camper.

 

[adrian oprea]

i have a n18u
and use this script to generate hosts file

#!/bin/sh
dir=/mnt/sda1/adblock
outlist=$dir/hosts.blocked
tempoutlist=$dir/list.tmp
echo "Getting ad list files quiet..."
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" \
| sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sed 's/  */\ /g' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u >> $tempoutlist
wget -qO- "http://adaway.org/hosts.txt" \
| sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sed 's/  */\ /g' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u >> $tempoutlist
wget -qO- "http://someonewhocares.org/hosts/zero/hosts" \
| sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sed 's/  */\ /g' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u >> $tempoutlist
wget -qO- "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext&useip=0.0.0.0" \
| sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sed 's/  */\ /g' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u >> $tempoutlist
wget -qO- "http://hosts-file.net/ad_servers.txt" \
| sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sed 's/  */\ /g' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u >> $tempoutlist
wget -qO- "http://hosts-file.net/download/hosts.txt" \
| sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sed 's/  */\ /g' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u >> $tempoutlist
wget -qO- "http://hostsfile.mine.nu/Hosts" \
| sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sed 's/  */\ /g' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u >> $tempoutlist

#echo "Getting Mother of All Ad Blocks list..."
#wget -qO- "http://adblock.mahakala.is/hosts" --user-agent="Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0" \
#| sed s/127.0.0.1/0.0.0.0/g | sed $'s/\r$//' | sed 's/  */\ /g' | grep -w ^0.0.0.0 | awk '{print $1 " " $2}' | sort -u >> $tempoutlist

# remove whitelisted entries in tempblack and write final file, remove temp and tempblack files
cat $tempoutlist | sort -u | fgrep -vf $dir/whitelist > $outlist

# Count how many domains/whitelists were added so it can be displayed to the user
numberOfAdsBlocked=$(cat $outlist | wc -l | sed 's/^[ \t]*//')
echo "$numberOfAdsBlocked ad domains blocked."
#echo "Deleting temp file ..."
rm $tempoutlist

sleep 1
echo "Restart dnsmasq..."
service restart_dnsmasq
echo "All done !!!"

 

[bayern1975]

how to test if this adblock script working correct? i installed this it`s looking good, no errors at log file but i do not see any changes when browsing interent?

 

[adrian oprea]

You read the first post ?
You have to make changes and to /jffs/scripts and /jffs/configs to work this script

This script is only for generate hosts blocked file

 

[bayern1975]

You read the first post ?
You have to make changes and to /jffs/scripts and /jffs/configs to work this script

This script is only for generate hosts blocked file

ok, but i do not know how to add, create in jffs/config? is there any sample for this?

 

[thelonelycoder]

ok, but i do not know how to add, create in jffs/config? is there any sample for this?

Best you start reading the thread from the beginning. A lot of the things you need to do it are asked in the first page or two.

 

[bayern1975]

i installed standard method but my english is not good to understand everything....if i have special and valid tutorial then i install and works everything like dnscrypt over entware.....there was super tutorial and i got working that without any problems.....
doesn`t matter, I will continue to use adblock plugin for firefox...

 

[Takoru]

Hello hervon,

is this script everything you have configured to get a proper adblocking working? I don't see any links to hosts files, except a black and white list.

Could you please provide a little tutorial how your adblocking is working? your script seems very nice.

 

[thelonelycoder]

thelonelycoder, since firmware 378.56 is out of beta. Here is the code I use for the script dnsmasq.postconf. As suggested by another forum member, I only rely on this file now. No need for the config dnsmasq.add anymore. Might be useful to update first post for the advanced method. Logging is inactive on purpose. Thanks for your great work.

#!/bin/sh
CONFIG=$1
source /usr/sbin/helper.sh

if [ -d "/mnt/sda1" ]; then
        pc_append $'\n' $CONFIG
        pc_append "# AdBlocking" $CONFIG
        pc_append "address=/0.0.0.0/0.0.0.0" $CONFIG
        pc_append "ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0" $CONFIG
        pc_append "addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked" $CONFIG
        pc_append "addn-hosts=/tmp/mnt/sda1/hosts/blacklist.txt" $CONFIG
        pc_append $'\n' $CONFIG
        pc_append "# Logging" $CONFIG
        pc_append "# log-facility=/mnt/sda1/logs/dnsmasq.log" $CONFIG
        pc_append "# log-async" $CONFIG
        pc_append "# log-queries" $CONFIG
fi

Mine looks about the same.
Logging is the reason why we have to move contents from dnsmasq.conf.add to dnsmasq.postconf in the latest release of Merlin's firmware. As there is no mention of it in Post #1 I am not going to change it.

 

[ChikkSpot]

Something rather weird. Not sure if it's new with .56 or it has been like this since day #1.

For example, say your hosts.blocked has only 1 entry in it:

0.0.0.0 doubleclick.net

Now if you try to ping doubleclick.net, it returns 0.0.0.0 as expected.

But if you try to ping www.doubleclick.net, it returns the real IP address instead of 0.0.0.0 (i.e., not blocked).

So seems like the entries in hosts.blocked does not imply the entire domain? And you need to spell out every single ad-server under doubleclick.net?

I used to put the entries in address=/<hostname>/0.0.0.0 format in dnsmasq.conf.add and never had this issue before. Wonder if it's specific to doing it via addn-hosts.

 

[thelonelycoder]

Something rather weird. Not sure if it's new with .56 or it has been like this since day #1.

For example, say your hosts.blocked has only 1 entry in it:

0.0.0.0 doubleclick.net

Now if you try to ping doubleclick.net, it returns 0.0.0.0 as expected.

But if you try to ping www.doubleclick.net, it returns the real IP address instead of 0.0.0.0 (i.e., not blocked).

So seems like the entries in hosts.blocked does not imply the entire domain? And you need to spell out every single ad-server under doubleclick.net?

I used to put the entries in address=/<hostname>/0.0.0.0 format in dnsmasq.conf.add and never had this issue before. Wonder if it's specific to doing it via addn-hosts.

That is why I made the examples for the blacklist.txt look like this in the post:

0.0.0.0 www.rivalo.com
0.0.0.0 rivalo.com
0.0.0.0 madpandatv.net
0.0.0.0 www.snapdo.com

 

[reSearchITeng]

Hi All,
I've been playing this week-end with ad-blocking hosts files. I've created the "grandmother" of all hosts files .
curl -sL "https://github.com/ReSearchITEng/grandmother-hosts-ad-blocking/blob/master/hosts.blocked0?raw=true" >hosts.blocked0
curl -sL "https://github.com/ReSearchITEng/grandmother-hosts-ad-blocking/blob/master/hosts.blocked127?raw=true" >hosts.blocked127

(if curl is not working for you, replace "curl -sL" above with: "wget -O- --no-check-certificate")

On github I've put also the script to generate the files (generateBrandNewHosts0List.sh), anyone is invited to take a look, improve, comment, here it is: https://github.com/ReSearchITEng/grandmother-hosts-ad-blocking . It collects the files from all these:

• hosts.eladkarako.com
• winhelp2002.mvps.org
• adaway.org/hosts.txt
• someonewhocares.org/hosts/zero/hosts
• pgl.yoyo.org/adservers
• hosts-file.net/ad_servers.txt
• hosts-file.net/download/hosts.txt
• hostsfile.mine.nu/Hosts
• adblock.mahakala.is/hosts

All together is 27 Mb of unique&sorted entries.
I am using it on RaspberryPi1 : I've set dnsmasq there (RPi) and I've pointed the DNS of my router to it (with the secondary DNS pointing to my ISP or google or openDns, just in case I need to reboot Pi1 ).
I've tried to put this 27Mb file on the Asus N 16 RT, apparently it's too much for this old AP. I can see in the logs "cannot allocate xxx bytes", and nslookup fails to get anything resolved. When I put the same hosts file on RPi1 or Asus N 18 RT, everything is perfect. So it's clearly old N 16 cannot handle such big hosts file apparently.

If anyone has ideas/comments on this topic, please put in the message the keyword: #grandmother

 

[ChikkSpot]

That is why I made the examples for the blacklist.txt look like this in the post:

0.0.0.0 www.rivalo.com
0.0.0.0 rivalo.com
0.0.0.0 madpandatv.net
0.0.0.0 www.snapdo.com

the problem then becomes those files become unnecessarily huge, as the alternative address=/<domain>/0.0.0.0 can handle the entire domain.

i'm curious which way is faster, processing-power-wise (i.e., domain suffix matching vs going through tons of entries).