AdBlocking with combined hosts file

[zekesdad]

Thanks for that, clears one more thing up for me.

Now regarding the scripts in NP++, what is the correct file extension to save as? I see lots of choices like "*", *txt, and a unix script file. I do have EOL set to unix.

there shouldn't be any extensions.

 

[Joe C]

OK, This is what I have done..

Reformat jffs as John said, I have by default /scripts and /configs

Started fresh with copying the 2 scripts over from the first post of this thread. The first script was named services-start, NP++ added the .txt, so I had to rename it, wich just entailed deleting the .txt extension.

The second script, update-hosts.sh, NP++ saved that without the .txt extension.

Using WinScp, I moved both scripts to /jffs/scripts. In WinScp they sit side by side, no other folder, is that correct?

I have the file dnsmasq.conf.add.txt (note the .txt extension) in /jffs/configs

My usb stick is at /tmp/mnt/sda1/hosts I can navigate there and see blacklist.txt and whitelist.txt. There is also a hosts file there.

To test the script services-start, in WinScp I right clicked and selected execute to see it run. I get a dialog saying host not responding for more then 15 seconds with a countdown timer, then the box closes. The script did not run, correct?

I have not done the chmod command yet as I was thinking I should have seen some type of confirmation that the scripts are working first, yes?

 

[thelonelycoder]

Run the commands in a proper ssh client like PuTTY and not via WinSCP. And ideally you would not need to as the router does it all by himself.
If you follow the instructions step by step and reboot the router at the end of it everything should be working properly.

 

[Joe C]

In Putty I navigated to /jffs/scripts and issued ./services-start.
The result is: cat:read error:is a directory

 

[ColinTaylor]

Thanks for the response, zekesdad.

I have a folder /jffs/scripts/services-start and I copied the file and named it services-start. In WinScp I right click on the script , select custom commands and execute to see it run. The progress bar does not leave 0% complete then the dialog box closes a few seconds later.

I have issued the chmod command, but where should it do it from ? In /temp/home/root# or in /jffs/scripts/services-start?

UPDATE: Sorry, I hadn't spotted that others have already replied to this. Please ignore this post.

Hi Joe C,

1) Your folder should be called /jffs/scripts not /jffs/scripts/services-start.
2) Within the /jffs/scripts folder should be all your scripts. i.e. services-start and update-hosts.sh
3) You should have a folder called /jffs/configs. Within it should be your file dnsmasq.conf.add
4) You should have a folder on your USB drive called hosts (mounted on the router as /tmp/mnt/sda1/hosts)
5) This folder must contain the files whitelist.txt and blacklist.txt

To run the scripts you need to be logged onto the router (using something like PuTTY) or you can just reboot the router.

 

[ColinTaylor]

In Putty I navigated to /jffs/scripts and issued ./services-start.
The result is: cat:read error:is a directory

Try the following instead:

/jffs/scripts/update-hosts.sh

 

[elorimer]

Joe C:

1. If you still have dnsmasq.conf.add.txt, That's a problem. It needs to be dnsmasq.conf.add, and it needs to be in unix format. In it you need the line pointing to your hosts.clean (or whatever you call it). Whenever dnsmasq restarts, it goes to the dnsmasq.conf.add file, sees the line pointing to hosts.clean, and reads all those hosts in. If it doesn't, nothing will happen. You can see it do that in the system log.
2. If you don't do the changemod, the scripts won't run in the first place. One way to do it is to select the scripts in WinSCP, right click, go to properties, and change the rights by clicking the checkboxes until they add to 755.
3. If you execute a script in WinSCP, in my experience, you get that popup that just shows 0%, until the script finishes. If it doesn't run, you get an error, but 0% is as good as it gets. Putty gives you more feedback.
4. Somewhere, you need to run update-hosts.sh first, to generate the file. Some do it from services-start. I prefer to follow what thelonelycoder did and do it in a chron job once a week. Then services-start just restarts dnsmasq. In my case, my USB drive is not mounting early enough, so I put the dnsmasq restart instruction in post-mount instead.
5. whitelist.txt and blacklist.txt both need to exist (even if they are empty). But whitelist.txt can't have empty lines or your hosts.clean file will be empty.

PS. Hope that helps. I've found this whole thread a marvelous way to go from complete ignorance to some awareness of how the scripts work with Merlin's firmware, and Linux commands. Even more useful than adblocking!

 

[Joe C]

Try the following instead:

/jffs/scripts/update-hosts.sh

Same result cat:read error: is a directory.

I removed the .txt extension from dnsmasq.conf.add, reran and same error

 

[Joe C]

Joe C:

1. If you still have dnsmasq.conf.add.txt, That's a problem. It needs to be dnsmasq.conf.add, and it needs to be in unix format. In it you need the line pointing to your hosts.clean (or whatever you call it). Whenever dnsmasq restarts, it goes to the dnsmasq.conf.add file, sees the line pointing to hosts.clean, and reads all those hosts in. If it doesn't, nothing will happen. You can see it do that in the system log.

Can you explain this further? I do not have a hosts clean that I'm aware of. I just copied the whole dnsmasq file and did not add anything to it. That's not correct?

 

[thelonelycoder]

Can you explain this further? I do not have a hosts clean that I'm aware of.

The advanced method's name for the generated hosts file is hosts.blocked. The standard method uses the name hosts.clean. Yours should generate a hosts.blocked file.

 

[elorimer]

Because of the cat:read error, perhaps you have an error in this part of update-hosts:

# remove whitelisted entries in temp and write final file, remove temp file
cat $dir/whitelist.txt | sed $'s/\r$//' | grep -vf - $dir/temp > $dir/hosts.blocked

If you have a hosts.blocked file that looks ok, perhaps you just restart dnsmasq and see if
you've got something that gets read in.

 

[Joe C]

Update.. Looking over my configuration, I realized that blacklist.txt and whitelist.txt were created as folders and not files. Fixed that and both service-start and update-hosts.sh will run in putty. After 20 or so seconds of the prompt returns with "done", nothing else, that's good right ?

I executed chmod a+rx /jffs/scripts/* then reboot from putty. Device restarts, I close my browser, flush dns because I'm using a Win7 computer and when everything is back up there are still ads.

Heres the log file:

Dec 11 14:42:18 miniupnpd[773]: remove port mapping 8083 TCP because it has expired
Dec 11 14:42:18 miniupnpd[773]: remove port mapping 8082 TCP because it has expired
Dec 11 14:55:51 login[945]: root login on 'pts/0'
Dec 11 14:56:30 rc_service: service 965:notify_rc restart_dnsmasq
Dec 11 14:56:30 dnsmasq[804]: exiting on receipt of SIGTERM
Dec 11 14:56:30 custom config: Appending content of /jffs/configs/dnsmasq.conf.add.
Dec 11 14:56:30 dnsmasq[968]: started, version 2.75 cachesize 1500
Dec 11 14:56:30 dnsmasq[968]: asynchronous logging enabled, queue limit is 5 messages
Dec 11 14:56:30 dnsmasq-dhcp[968]: DHCP, IP range 192.168.1.115 -- 192.168.1.254, lease time 1d
Dec 11 14:56:30 dnsmasq-dhcp[968]: DHCP, sockets bound exclusively to interface br0
Dec 11 14:56:30 dnsmasq[968]: read /etc/hosts - 6 addresses
Dec 11 14:56:30 dnsmasq[968]: read /tmp/mnt/sda1/hosts/blacklist.txt - 0 addresses
Dec 11 14:56:31 dnsmasq[968]: read /tmp/mnt/sda1/hosts/hosts.blocked - 73433 addresses
Dec 11 14:56:31 dnsmasq[968]: read /etc/hosts.dnsmasq - 9 addresses
Dec 11 14:56:31 dnsmasq-dhcp[968]: read /etc/ethers - 9 addresses
Dec 11 14:56:31 dnsmasq[968]: using nameserver 24.229.54.212#53
Dec 11 14:56:31 dnsmasq[968]: using nameserver 216.144.187.199#53
Dec 11 14:56:31 dnsmasq[968]: using nameserver 204.186.0.180#53

 

[ColinTaylor]

It looks like it should be working

Dec 11 14:56:31 dnsmasq[968]: read /tmp/mnt/sda1/hosts/hosts.blocked - 73433 addresses

What do you get if you type the following on your Windows PC?

nslookup doubleclick.com

 

[Joe C]

nslookup doubleclick.com entered into cmd prompt returns:


Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\Administrator>nslookup doubleclick.com
Server: google-public-dns-a.google.com
Address: 8.8.8.8

Non-authoritative answer:
Name: doubleclick.com
Addresses: 2607:f8b0:4006:80a::1000
173.194.123.8
173.194.123.3
173.194.123.0
173.194.123.2
173.194.123.6
173.194.123.14
173.194.123.5
173.194.123.4
173.194.123.7
173.194.123.1
173.194.123.9




In a browser it returns can't find server.

 

[john9527]

Did you flush your windows dns cache?

ipconfig /flushdns

 

[elorimer]

In addition to the ipconfig /flushdns, you probably also have to clear your browser cache. I find in chrome, for example, unless I go to settings/advanced settings/delete cached images, the ads are still there--it is showing you what it had saved from before. The browser looks to see if it has the ad already, and if it does, it shows the ad. If it doesn't, it will look up the ip name in the windows cache, and if it findsit, goes there. If it isn't in the cache, then it goes out to dnsmasq, where your host.blocked returns 0.0.0.0 and it stops.

The part of your syslog that shows the 73000+ names being read in shows that the router part is working.

 

[ColinTaylor]

C:\Users\Administrator>nslookup doubleclick.com
Server: google-public-dns-a.google.com
Address: 8.8.8.8

Non-authoritative answer:
Name: doubleclick.com
Addresses: 2607:f8b0:4006:80a::1000
173.194.123.8
173.194.123.3
173.194.123.0
173.194.123.2
173.194.123.6
173.194.123.14
173.194.123.5
173.194.123.4
173.194.123.7
173.194.123.1
173.194.123.9




In a browser it returns can't find server.

The problem is that you are not using the routers DNS server, which is where the ad blocking happens. Your PC is going directly to Google's DNS.

You want to see something like this:

C:\Users\Colin>nslookup doubleclick.com
Server: router.asus.com
Address: 192.168.1.1

Name: doubleclick.com
Addresses: 2a00:1450:4009:80a::200e
0.0.0.0

Make sure your PC is getting its DNS server address from DHCP and that on the router "LAN > DHCP Server > DNS Server 1 & 2" is left blank.

 

[john9527]

The problem is that you are not using the routers DNS server, which is where the ad blocking happens. Your PC is going directly to Google's DNS.

Good catch!

 

[Joe C]

[QUOTE
Make sure your PC is getting its DNS server address from DHCP and that on the router "LAN > DHCP Server > DNS Server 1 & 2" is left blank.[/QUOTE]

The router was and still is set exactly as described above. I did flush dns again and did clear cache from the browser. Still have ads.

I assign IP addresses to all my devices through the router, could that be doing it?

 

[ColinTaylor]

Make sure your PC is getting its DNS server address from DHCP and that on the router "LAN > DHCP Server > DNS Server 1 & 2" is left blank.

The router was and still is set exactly as described above. I did flush dns again and did clear cache from the browser. Still have ads.

I assign IP addresses to all my devices through the router, could that be doing it?

The problem is none of the above. You need to look at your PC and find out how/why it's getting the wrong DNS server address. Have you set it manually?