AdBlocking with combined hosts file

[zekesdad]

That is missing a + inbetween. For some reason that was omitted in an edit I've done recently. Maybe a XenForo bug or something.
The line now again correctly reads chmod a+rx /jffs/scripts/* in the first post.


You have to create the directory /hosts in the root directory of your stick, you will then have a path like this: /tmp/mnt/sda1/hosts/.
Then place the whitelist and blacklist files in it.

Thank you again for your time and help.

I think it seems to be working, in that it creates a hosts.block(ed? - had to go to work so I don't exactly remember the name) in /tmp/mnt/sda1/hosts/ and that file is filled with 0.0.0.0 stuff.here.com, but even after rebooting the router and the computers, and ipconfig/flushdns'ing, i couldn't actually get it to block any ads. Again, im sure it's something I did, but going back over your tutorial, I can't pin it down.

 

[thelonelycoder]

Thank you again for your time and help.

I think it seems to be working, in that it creates a hosts.block(ed? - had to go to work so I don't exactly remember the name) in /tmp/mnt/sda1/hosts/ and that file is filled with 0.0.0.0 stuff.here.com, but even after rebooting the router and the computers, and ipconfig/flushdns'ing, i couldn't actually get it to block any ads. Again, im sure it's something I did, but going back over your tutorial, I can't pin it down.

Are you sure you've created the file /jffs/configs/dnsmasq.conf.add and added the the entries in the box as per step by step instructions on post #1?

 

[zekesdad]

Are you sure you've created the file /jffs/configs/dnsmasq.conf.add and added the the entries in the box as per step by step instructions on post #1?

Yes, I just checked. Also, I think that's what created the file hosts.blocked.

 

[thelonelycoder]

Yes, I just checked. Also, I think that's what created the file hosts.blocked.

No, the hosts file is invoked by the services-start script and executed by update-hosts.sh.
Try enable logging as explained in post #218 and do the following:
Restart dnsmasq

service restart_dnsmasq

and then look at the logfile by calling

tail -f /tmp/mnt/sda1/logs/dnsmasq.log

and look for the startup entries from dnsmasq. You should see something like this:
dnsmasq[1252]: read /etc/hosts - 5 addresses
dnsmasq[1252]: read /tmp/mnt/sda1/hosts/blacklist.txt - 51 addresses
dnsmasq[1252]: read /tmp/mnt/sda1/hosts/hosts.blocked.txt - 376349 addresses

If not your hosts file is not part of dnsmasq.

 

[zekesdad]

No, the hosts file is invoked by the services-start script and executed by update-hosts.sh.
Try enable logging as explained in post #218 and do the following:
Restart dnsmasq

service restart_dnsmasq

and then look at the logfile by calling

tail -f /tmp/mnt/sda1/logs/dnsmasq.log

and look for the startup entries from dnsmasq. You should see something like this:
dnsmasq[1252]: read /etc/hosts - 5 addresses
dnsmasq[1252]: read /tmp/mnt/sda1/hosts/blacklist.txt - 51 addresses
dnsmasq[1252]: read /tmp/mnt/sda1/hosts/hosts.blocked.txt - 376349 addresses

If not your hosts file is not part of dnsmasq.

ok, so I did all that, and when I try to go to a "blocked site" from hosts.blocked I dont get it redirecting to 0.0.0.0.

i picked http://cqcounter.com/ and this: http://pastebin.com/qJ6TZQdD was the outcome.

 

[thelonelycoder]

@zekesdad Looks like you have no DNS Filering enabled. What is the output of the dnsmasq log when you restart it? Im interested in the read.... part as stated in the previous reply.

 

[zekesdad]

@zekesdad Looks like you have no DNS Filering enabled. What is the output of the dnsmasq log when you restart it? Im interested in the read.... part as stated in the previous reply.

youre right, i forgot to edit the path of the usb stick in the config file. seems to be working awesomely now. thank you for all your help and patience!

 

[mcfrojd]

Hello, im pretty new to this, but after a lot of effort i managed to get the "Advance" version to work.
Now i only have one problem.
When i start the 888 poker (windows app) it wount start when i have ad-blocking in my router.
How do i find out what adresses the 888 poker app uses so i can whitelist them?

Or how do i modify the "Advance" version scripts to not fetch all the adresses from the (5?) diffrent sources, and only use the adresses i manually add to the blacklist.txt (might be a tought task to add manually, but i would get full controll of whats get blocked)

 

[thelonelycoder]

@mcfrojd Add these to your whitelist.txt and restart your router.

www.888poker.com
www.pacificpoker.com
script.crazyegg.com
www.googletagmanager.com

 

[mcfrojd]

@mcfrojd Add these to your whitelist.txt and restart your router.

www.888poker.com
www.pacificpoker.com
script.crazyegg.com
www.googletagmanager.com

Ok thanks, I'll try that, but how do you find out those url:s?
Is there a program to analyse connections, or could you find it in the router logs or under router tab "connections"?

 

[mcfrojd]

On the site https://www.flashback.org/ (swedish) there is a lot of ads that not get filtered.
I have tried to add these url's to blacklist.txt but the ads still keep coming on that site.
"added to blacklist.txt"
0.0.0.0 https://www.flashback.org/ads/
0.0.0.0 https://static.flashback.org/

How do i analyse the site to find what to add to blacklist.txt?

 

[thelonelycoder]

@mcfrojd Look at the formatting of the blacklist entries again, it should read:
0.0.0.0 www.flashback.org
0.0.0.0 static.flashback.org

Only a full domain name is supported, no subdirectories like www.flashback.org/ads/
But be careful, that site serves the ads and the (real) content from the same domain. If you block them you'll end up with a blank page....

To analyse what is happening look at the post #218

 

[shooter40sw]

Hi @thelonelycoder , love your work with the adblocker, I implemented a while ago your advanced method but I´m looking to the log you recently posted
-----------------
dnsmasq[1252]: read /tmp/mnt/sda1/hosts/blacklist.txt - 51 addresses
dnsmasq[1252]: read /tmp/mnt/sda1/hosts/hosts.blocked.txt - 376349 addresses
------------------
where the number of the host.blocked.txt is a big number compared to my logs, that when I reboot the router or when the cron job starts, I only get in my logs between 21000 to 71000 addresses, what can I check out on this issue?
Thanks

 

[thelonelycoder]

@shooter40sw Because you can do with it whatever you want! I have added this rather large hosts file: http://support.it-mate.co.uk/downloads/hosts.txt
Be careful though, I have disabled the update when rebooting. It's just too big a job to do when booting for the router. Mine updates as everyone elses on Friday. That'll do for me.

 

[thelonelycoder]

Ok thanks, I'll try that, but how do you find out those url:s?
Is there a program to analyse connections, or could you find it in the router logs or under router tab "connections"?

Experience at looking at logs helps a lot in doing that.

 

[ChikkSpot]

Thanks for your input, I just tested your modified version and noticed a significant increase of time to process for the replace part of it.
In addintion to the 5 listed host files I include http://support.it-mate.co.uk/downloads/hosts.txt, a rather large file.
With the current script on my RT-AC66U I get these times:

real    0m55.992s
user    0m22.610s
sys    0m6.250s

While yours I had these times:

real    3m45.761s
user    1m31.790s
sys    1m42.590s

Not a big problem exept when rebooting, it takes longer for the router to come up. What I like is the hosts.blocked file decreases in size from 13524 KB down to 11751 KB with your refinements. That means less memory usage on the router.
For the time being I'll leave your script running.

I also had to go forth and back with the service-start and wan-start placement of the calling line for the script. I activated the logger to see where it failed but never really found the source of the problem. It eventally worked everytime with the service-start so I left it in there.

Yeah. It's definitely going to take more time to create the hosts.blocked file, but I think 3 mins or so once every week is OK.

However the bigger concern that I have is, would a big hosts.blocked file impact throughput. One thing that I'm not sure if how efficient dnsmasq comes up with the 0.0.0.0 with such a giant list to go through. I'm hoping it's not too much of a concern (e.g., it's cached somehow, etc.)

 

[thelonelycoder]

@ChikkSpot I have no performance issue with the large hosts file.
However, I have disabled the creation of a new combined file when booting. Auto update once a week takes care of that.

 

[shooter40sw]

Hi @thelonelycoder to stop the creation of the new file did you just eliminate the last three lines on the services-start file of the tutorial? I know the first lines make the cron job, but if you did a different approach? please share!, thanks a lot. regards

@ChikkSpot I have no performance issue with the large hosts file.
However, I have disabled the creation of a new combined file when booting. Auto update once a week takes care of that.

 

[thelonelycoder]

@shooter40sw
In /jffs/scripts/services-start comment out the code like this:

#source /jffs/scripts/update-hosts.sh

 

[mikkelnl]

I tried setting up method 2 on my AC68U, but it's not blocking adds... is there any way to check what went wrong?

/never mind... I found that after the reboot the USB stick wasn't recognized somehow... a 2nd reboot fixed that and it seems to be working now.