AdBlocking with combined hosts file

[thelonelycoder]

I'm on an Asus RT-AC68 and for some reason this line just produces a blank hosts.blocked file:
# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked

There's no error and the rest of the script works fine. If I go through the process line by line I can see the temp file does exists and has tons of normal looking entries. I do have a whitelist.txt as specified in post #1 with a few dummy domains.

Any ideas?

I am aware of it, it just happens at boot on my router, have not had the time to look into it, but will asap.

 

[elorimer]

Same here, on an N66U. Just at boot. Running the script afterwards it works fine.

 

[thelonelycoder]

Same here, on an N66U. Just at boot. Running the script afterwards it works fine.

I believe it has to to with the new firmware (currently on 378.53, but it happened on .52 as well). I'll run a few tests when I get time to do it. The cron job seems to run, so all is well exept at boot. Might have to ask RMerlin about it.

 

[thelonelycoder]

I am at my wits end what the problem is with the custom scripts, just made a post to get help from others.
FYI: http://hosts-file.net/ is down for unknown reasons, the hosts file (http://hosts-file.net/ad_servers.txt) will not be inculded.

I got the scripts working like thus for the Advanced method:
In post-mount add this:

#!/bin/sh

if [ $1 = "/tmp/mnt/sda1" ]
then
  ln -sf $1/entware /tmp/opt
fi

#!/bin/bash
sleep 5
source  /jffs/scripts/update-hosts.sh

It's a double calling for the update-hosts.sh and it works that way for the time being.

 

[thelonelycoder]

I'm on an Asus RT-AC68 and for some reason this line just produces a blank hosts.blocked file:
# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked

There's no error and the rest of the script works fine. If I go through the process line by line I can see the temp file does exists and has tons of normal looking entries. I do have a whitelist.txt as specified in post #1 with a few dummy domains.

Any ideas?

Try this:
cat $dir/tmp | sort -u | fgrep -vf $dir/whitelist.txt > $dir/hosts.blocked

 

[elorimer]

In my system log, an error is logged early in the boot sequence that blacklist.txt and whitelist.txt aren't found, and I never see the entry for dnsmasq reading the hosts.clean file. In my case the hosts directory is on a USB stick with a label, so instead of sda1 I use the label. But the USB label doesn't show in the log until much later. Perhaps services-start is too early, and post-mount calls the script after the USB stick is mounted?

 

[thelonelycoder]

In my system log, an error is logged early in the boot sequence that blacklist.txt and whitelist.txt aren't found, and I never see the entry for dnsmasq reading the hosts.clean file. In my case the hosts directory is on a USB stick with a label, so instead of sda1 I use the label. But the USB label doesn't show in the log until much later. Perhaps services-start is too early, and post-mount calls the script after the USB stick is mounted?

I have only recently had failures with the services-start script, still looking for a solution that works and makes sense.
Never logs an error as yours in mine. Try calling it in the post-mount file and see what happens.

 

[GregS]

Try this:
cat $dir/temp | sort -u | fgrep -vf $dir/whitelist.txt > $dir/hosts.blocked

This also produces a blank file.
I found the issue; having a blank line in the whitelist.txt file.

 

[tim28taft]

Hello all I'm running RT-N66U using 374.43_2-11E1j9527 (Merlin fork). I formated 15 gig USB stick to EXT2 i was able to install Entware however how on earth do you get this script to execute work you enter this script inside putty right ?
I just want to block some malware sites I have a list i made.

#!/bin/sh

if [ $1 = "/tmp/mnt/sda1" ]
then
  ln -sf $1/entware /tmp/opt
fi

sleep 12
wget -qO- "http://sizzleradio.com/djrequests/hostfile.txt" | grep -w ^0.0.0.0 | sed $'s/\r$//' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 5
service restart_dnsmasq

 

[tim28taft]

Nevermind after installing a program called "WinSCP" I was able to login and follow the instructions on the first page everything made much more since thanks

 

[shooter40sw]

Works great, just installed it on my N66U, dont notice performance impact, using the latest merlin build 378.53, using dnscrypt also, just noticed on the ipad and android device there are no ads, I used the advanced method, total RAM used is 70MB. After I booted it just started to work. Thanks!

 

[ColinTaylor]

Looks like it might be a timing issue to me. Increase the value of the first "sleep" command to something like 60 seconds and see if that fixes it.

UPDATE: Actually, thinking about it... That message is expected early in the boot process because dnsmasq is started before the services-start script is run. What you should see is your script running and dnsmasq restarting later in the log. If that's not happening then it sounds like the services-start isn't running.

 

[GregS]

Looks like it might be a timing issue to me. Increase the value of the first "sleep" command to something like 60 seconds and see if that fixes it.

My bad, my services-start file was missing all the important stuff. Likely due to my recent entware re-install. Restoring it resolved my issue.

 

[shooter40sw]

Strange but, Im doing an SSH tunnel to my home and when I try to access for example pcword I get stats.g.doublick.net is asking for username and password, through a dialog box, and says RT66U thats my router. Like I posted above ad block is working on my devices, but dont know why its working like this, I suppose its the adblocker, in other pages it gives me google analitics, and so on, each time it freezes the web surfing asking for this,,, In a quick test at home it did not do this, it seems to be doing this when using the ssh tunnel. thanks

 

[tim28taft]

Works great, just installed it on my N66U, dont notice performance impact, using the latest merlin build 378.53, using dnscrypt also, just noticed on the ipad and android device there are no ads, I used the advanced method, total RAM used is 70MB. After I booted it just started to work. Thanks!

Can you post your adblock code and your dnscrypt

 

[blunt_blade]

Hi there,

I'm trying to set this up on my RT-AC87U using the first simple method that uses the one hosts.clean file, but I'm running into some trouble.

Once dnsmasq has restarted with the new config no address will resolve, every request just returns "ping: bad address 'google.com'" for every address.

I've confirmed that the hosts.clean file does indeed contain entries and that all the paths are correct etc.

I'm running Asuswrt-merlin 378.53.

Any help would be greatly appreciated.

Just note that I am a bit of a Linux noob, I know just enough to find my way around

 

[shooter40sw]

Can you post your adblock code and your dnscrypt

Hi, Install Entware, then use dnscrypt guide that you can find in the forums, and in github, then I followed the instructions on the advanced method here, I dont know if what I did is the most efficient way, if anyone can point out anything else it would be nice, I just added in the first lines of the dnmasq.conf.add file, thats it

/jffs/configs# cat dnsmasq.conf.add
# AdBlocking
address=/0.0.0.0/0.0.0.0
ptr-record=0.0.0.0.in-addr.arpa,0.0.0.0
addn-hosts=/tmp/mnt/sda1/hosts/hosts.blocked
addn-hosts=/tmp/mnt/sda1/hosts/blacklist.txt

#Dnscrypt
no-resolv
server=127.0.0.1#65053

 

[blunt_blade]

I managed to fix the issue I was experiencing.

It was my mistake, I was accidentally writing the dnsmasq config to /jffs/configs/dnsmasq.conf instead of /jffs/configs/dnsmasq.conf.add.

My mistake, all seems to be working well now.

Thanks for the great tutorial!

 

[tijaune]

Advise to anyone using http://www.expedia.ca for their next trip hotel reservation and using the advanced method. I had to remove http://hosts-file.net/ad_servers.txt provider for hotel search to work properly, otherwise the site keep spinning. I guess expedia fetch hotel list from ads servers such as googleads ...

 

[Chrysalis]

more likely it was a tracking domain.

The subscribe button on extremetech is dead with these lists loaded, and it was down to ziffdavis domains which they use for some tracking and marketing.

Ideally we should see adverts hosts files that dont have all tracking stuff integrated, but seems they all combined lists.

I also dont use one of these lists entirely as I found over 600 domains had nothing to do with tracking or ads. A list with that many domains as false positives is far from credible.