What's new

AdBlocking with combined hosts file with Lighttpd

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

e2301010

New Around Here
Hi everyone,

First post here so bare with me.

This builds on thelonelycoder's script for host based domain blocking. The main difference is a customizable HTML or 1x1 transparent GIF file will resolve and replace the blocked section of the page, instead of a 404-type message. This is essentially emulating pixelserv if you use a GIF file.

Prerequisites:
- Merlin's latest firmware
- Entware installed
- USB drive formatted to EXT2 (I used EASEUS Partition Master for Windows)
- Running thelonelycoder's script

For these instructions, I'm going to use 192.168.0.1 as my router's IP address. Replace with your router's IP as necessary.

FIRST STEP: Freeing port 80 on the router
1. Login to the your router via web browser

2. Go to Administration > System
- Enabled SSH > Checked to Yes (if not already done)
- Authentication Method > Change to HTTPS
- HTTPS Lan port > 8443

3. Click Apply

4. Verify you are able to login to your router by going to https://192.168.0.1:8443

5. Depending on the browser, it will give you a security warning, simply bypass or add the address to the exceptions list. There's also a possibility that you will get a long message that the IP/port has changed and will not let you login. Do not panic. Firefox or Chrome has enabled offline cache, you can simply turn that off (just do a search) or use Internet Explorer.

SECOND STEP: Modifying the hosts.clean file to route to 192.168.0.1
1. Login to the route via WinSCP > browse to /jffs/configs/ > right-click, edit dnsmasq.conf.add with changes:
remove this line as modifying it doesn't do anything:
Code:
address=/0.0.0.0/0.0.0.0
and ensure this shows as the first line:
Code:
addn-hosts=/tmp/mnt/sda1/hosts.clean
click the save icon

2. Browse to /jffs/scripts/ > right-click, edit post-mount file > copy and paste the code below:
Code:
#!/bin/sh
if [ $1 = "/tmp/mnt/SD2" ]
then
  ln -sf $1/entware /tmp/opt
fi
sleep 12
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//;s/0.0.0.0/192.168.0.1/' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 5
service restart_dnsmasq

click the save icon

3. In the same directory > right -click, edit update-hosts.sh file > copy and paste the code below:
Code:
#!/bin/sh
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//;s/0.0.0.0/192.168.0.1/' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 5
service restart_dnsmasq

click the save icon

THIRD STEP: Install and configure Lighttpd
1. Login to the router via PuTTY and enter the following commands:
Code:
opkg install lighttpd
opkg install lighttpd-mod-redirect
/opt/etc/init.d/lighttpd stop

2. Login to the router via WinSCP > browse to /opt/etc/lighttpd/ > right-click, edit lighttpd.conf with changes:
between the parenthesis of server.modules, add "mod_redirect" so it looks like:
Code:
server.modules = ( "mod_redirect" )
change server.port from 81 to 80
Code:
server.port = 80
comment out server.bind by adding # in front
Code:
#server.bind = "0.0.0.0"
change server.document-root path to /mnt/sda1 where sda1 is the name of your USB stick
Code:
server.document-root = "/mnt/sda1"
modify line to:
Code:
index-file.names = ( "index.html" )
add new line:
Code:
url.redirect = ( "^(.*?)(?<!\/index\.html)$" => "/index.html" )
click the save icon

3. Optional: the url.redirect is essentially a regex that redirects any blocked URL to /index.html. Once it reaches /index.html, it will stop redirecting. You can change index.html to blank.gif (for example) by replacing all instances of index with blank and all instance of html with gif for the url.redirect line only.
Code:
url.redirect = ( "^(.*?)(?<!\/blank\.gif)$" => "/blank.gif" )

4. Browse to /mnt/sda1/ and upload a simple index.html or blank.gif file. Ensure this file matches entries on lighttpd.conf

5. Go back to PuTTY and type reboot or click Reboot button from the browser.

6. Use a device connected to the router via WiFi or LAN port and browse to adchimp dot com and see if it redirects to your index.html or blank.gif file.

DONE!
 
Last edited:
Thanks you! I was about to look into that as well. Will try it out after some much needed pillow time.
 
Just installed this, works great!
In another thread decedion came up with a better script than I did to get the hosts files. Here's what my scripts look now:

This creates hosts.clean in one fell swoop and replaces 0.0.0.0 with 192.168.0.1.
The part in /jffs/scripts/post-mount
Code:
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//;s/0.0.0.0/192.168.0.1/' | sort -u > /tmp/mnt/sda1/hosts.clean
then the update-hosts.sh looks like so:
Code:
#!/bin/sh
wget -qO- "http://winhelp2002.mvps.org/hosts.txt" "http://someonewhocares.org/hosts/zero/hosts" "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate[day]=&startdate[month]=&startdate[year]=&mimetype=plaintext&useip=0.0.0.0" | grep -w ^0.0.0.0 | sed $'s/\r$//;s/0.0.0.0/192.168.0.1/' | sort -u > /tmp/mnt/sda1/hosts.clean
sleep 1
service restart_dnsmasq
 
Last edited:
Very nice. Just one line and one output file. I've gone ahead and updated the first post.

Can you confirm the address=/0.0.0.0/0.0.0.0 doesn't do anything? I initially tried to simply modify that line with the router's IP, leaving the hosts.clean file to resolve to 0.0.0.0 but it didn't work.
 
Last edited:
Can you confirm the address=/0.0.0.0/0.0.0.0 doesn't do anything?
I can confirm this will not work. Just did two tests adding address=/0.0.0.0/192.168.0.1 before and then after the addn-hosts entry. Apparently this does not replace 0.0.0.0 with the Routers IP for all the hosts file. Maybe a variable would do it.
 
Well it doesn't matter much, the hosts.clean file points to the right IP, which is why I've removed it from the dnsmasq.conf.add file

I can confirm this will not work. Just did two tests adding address=/0.0.0.0/192.168.0.1 before and then after the addn-hosts entry. Apparently this does not replace 0.0.0.0 with the Routers IP for all the hosts file. Maybe a variable would do it.
 
Is there a benefit to serving up a simple index.html over a pixel image (or vice versa)?

Can I try to come up with a special url.redirect directive that delivers pixel images for image/media requests, and index.html for everything else? (I say, "can I," because this is uncharted water for me....


---
baltosml@RT-N66U
 
Is there a benefit to serving up a simple index.html over a pixel image (or vice versa)?
One "benefit" is you can add text or whatever you like to fill the void. For testing I added "*uck this add!" to index.html just to enjoy it!
But I use the pixel alone. Some Sites are better preserved that way.

Can I try to come up with a special url.redirect directive that delivers pixel images for image/media requests, and index.html for everything else? (I say, "can I," because this is uncharted water for me....
Regex is your friend, the possibilities are almost endless. Look up .htaccess url rewriting
 
Thanks. I am trying out the transparent pixel. Works well.

A recipe site my wife and I use was broken because of some blocked sites. I created a simple list of white listed sites (one per line), and added this after the line that creates the hosts.clean file:

cat /mnt/sda1/whitelist | while read line; do sed -i /${line}/d /mnt/sda1/hosts.clean ; done


---
baltosml@RT-N66U
 
Last edited:
This is very helpful. Thanks for putting it together. How hard would it be to automate this entire configuration, to make setting it up after firmware upgrades that much easier?


Sent from my iPad using Tapatalk HD
 
Thanks for this all seems to be working as it should, however

it is now impossible to connect to the RT-N66 (https://192.168.1.1:8443/) using IE, I run windows 8.1 and have added the address to trusted zones.

Any ideas please ?
 
it is now impossible to connect to the RT-N66 (https://192.168.1.1:8443/) using IE, I run windows 8.1 and have added the address to trusted zones.
Just helpful hints:
Did you clear the browsers cache, the Windows DNS cache (ipconfig/flushdns , run as administrator in IE 8)?
Do you get an error or a timeout or the page where it says the Routers settings have changed?
Is your routers IP 192.168.1.1?
 
How hard would it be to automate this entire configuration, to make setting it up after firmware upgrades that much easier?

As long as you don't touch /jffs/* and any other mounted devices there is no need to set it up again after a firmware upgrade. Just reenable /jffs/ if it is disabled after the upgrade.
 
Yes, was using your method before this so just made these adjustments.

The error I get is "There is a problem with this website’s security certificate. We recommend that you close this webpage and do not continue to this website." blah blah

btw, what is the quickest way to turn on/off ad blocking ?

thanks
 
The error I get is "There is a problem with this website’s security certificate. We recommend that you close this webpage and do not continue to this website." blah blah

IE can be persistent with self signed certificates. For whatever reasons...
Try using Firefox for once, it's a great browser. At least for us power users and webcoders!

btw, what is the quickest way to turn on/off ad blocking ?

In /jffs/configs/dnsmasq.conf.add remove the entry
Code:
addn-hosts=/tmp/mnt/sda1/hosts.clean
and restart dnsmasq:
Code:
service restart_dnsmasq
Clear browser- and DNS cache and all adds show up again.
 
guide is wrong.

the file /opt/etc/init.d/lighthttpd doesnt exist

instead it is

/opt/etc/init.d/S80lighttpd

why the S80 at start?
 
guide is wrong.
the file /opt/etc/init.d/lighthttpd doesnt exist
instead it is
/opt/etc/init.d/S80lighttpd
why the S80 at start?
The package lighttpd has been updated and now uses this name for the startscript. S80 is an ordering thing I believe, in what order to start the service.

Ha! Just noticed 200 posts makes you a "Very Senior Member" . Well I do have gray hair so this is well deserved.
 
Last edited:
Couple of things noticing so far.

All dns lookups that hit a hostname in the hosts file take just under 300ms. (info found using firefox dev tools). It be great if we can make dnsmasq cache results from the hosts file.

On a yahoo answers page I got a unable to connect to sb.scorecardresearch.com error, yet testing http://sb.scorecardresearch.com/ in my browser loads my blank image, so that seems to suggest either it was a ssl link as some adservers now use https or my router (AC66) is not handling the demand put on it.
 
The 300ms is the timeout for the host not found. You cannot change this.And yes https sites are excluded. Thats something for more advanced systems. Blocking with a hosts file is very basic stuff. If you want a better adblocking look into squid or other proxy servers. opkg has two or three available. As for my need I am extremely happy how this works.
 
The 300ms is the timeout for the host not found. You cannot change this.And yes https sites are excluded. Thats something for more advanced systems. Blocking with a hosts file is very basic stuff. If you want a better adblocking look into squid or other proxy servers. opkg has two or three available. As for my need I am extremely happy how this works.

It is good, this will be staying on my lan, just seeing if there is ways to improve it, adding https is doable with basic config change but the issue would be prompts in the browser to approve the certs, so I wont bother the same as yourself as it adds too much complication.

the 300ms is dns lookup time not page loading time, the developer tool breaks down the time periods to different parts of the request although I think usually when not using the developer tools firefox will allow cached lookups from the OS/browser so this will only be an issue on the first lookup.
 

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top