AB-Solution - The Ad Blocking Solution

[thelonelycoder]

Hi there @thelonelycoder I have a small problem with absolution 3.8 on my ac68u running 380.65_4. The log shows this:

Apr 19 02:04:58 : AB-Solution: file 5 download failed or file is not hosts file, using backup file
Apr 19 02:05:05 : AB-Solution updated blocking file, 69075 domains are now blocked
Apr 19 02:05:09 : AB-Solution counted ads before log files reset (triggered by update-hosts.add)
Apr 19 02:05:09 : AB-Solution blocked 9,299 total 9,299 week 702 new ads
Apr 19 02:05:09 : AB-Solution rotated dnsmasq log files

User name deleted for privacy.
The line I'm worried about is the first one saying the 5th file is not a hosts file. When I run ab-solution.sh I see that I have 5 hosts files in the header a the top.
Any ideas what is going on?
Is it normal to have 5 hosts files?

Your blocking file is amalgamated from 5 hosts files. The fifth likely timed out while updating. You can enter c into AB and look at the hosts_list variable.

 hosts_list="http://hosts-file.net/ad_servers.txt
 http://hosts-file.net/emd.txt
 http://hosts-file.net/grm.txt
 http://hosts-file.net/mmt.txt
 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
 http://adblock.mahakala.is"

In the above example, six hosts files are used, http://adblock.mahakala.is is the last one.
If your blocking file is a standard file from the selection in b then there's nothing to worry about. These servers may timeout every now and then.
That is why the backup is built in.
Run u to manually update the blocking file, most likely it will update without any errors.

 

[skeal]

Your blocking file is amalgamated from 5 hosts files. The fifth likely timed out while updating. You can enter c into AB and look at the hosts_list variable.

 hosts_list="http://hosts-file.net/ad_servers.txt
 http://hosts-file.net/emd.txt
 http://hosts-file.net/grm.txt
 http://hosts-file.net/mmt.txt
 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
 http://adblock.mahakala.is"

In the above example, six hosts files are used, http://adblock.mahakala.is is the last one.
If your blocking file is a standard file from the selection in b then there's nothing to worry about. These servers may timeout every now and then.
That is why the backup is built in.
Run u to manually update the blocking file, most likely it will update without any errors.

Thank you for your prompt reply. Thanks for the info bud!

 

[Bamsefar]

To code it, this would be something like this:
- add wildcard domain(s) to file wildcard_blocklist
- find all queries in dnsmasq log files for first entry in wildcard_blocklist and write it to a temp file
- repeat for every entry in wildcard_blocklist
Then:
- remove duplicates in temp file
- remove whitelist entries in temp file
- add it to auto_blacklist
- remove duplicates from auto_blacklist
- reload the auto_blacklist in dnsmasq

As you can see, this would be doable, but time consuming with a lot of entries in the wildcard_blocklist.
Is this about ad-blocking? Practically all ad domains are blocked by one of the hosts files in use in the blocking file.
And I wonder if it's worth the effort.

I guess a few awk commands?

I understand what you are trying to say, however your solutions is wonderful as it is, and I could always write the script myself (it's not that hard) - is there any way to reload blacklist from outside, say with a parameter to ab_solution.sh?

 

[thelonelycoder]

I guess a few awk commands?

I understand what you are trying to say, however your solutions is wonderful as it is, and I could always write the script myself (it's not that hard) - is there any way to reload blacklist from outside, say with a parameter to ab_solution.sh?

Just issue a dnsmasq restart:

service restart_dnsmasq

But if you want to outright wildcard block them, use what was mentioned before.
Add or create /jffs/configs/dnsmasq.conf.add
Then add one blocking domain per line, as below:

address=/domain.com/0.0.0.0
address=/otherdomain.com/0.0.0.0

Substitute the blocking IP with the pixelserv IP and you're good to go.
Every time you change this file, restart dnsmasq for the changes to take effect.
Remember that you added them manually, this is outside the control of AB.

 

[HawkInOz]

I recently had the USB drive fail that was connected to my router. There goes AB-Solution... no problem, I just re-installed. There goes my whitelist... oh boy... big problem. Months of adding this and that to the file and documenting all the additions with ' # descriptions ' so I'd know why they were there. Gone. Well, that got me thinking. What are the chances that the whitelist could be emailed as either an attachment or in the message body of an email, as a way to back up its contents? This could even be included with the weekly 'Router Stats' email. Sure would be a great addition for version 4.0!

 

[skeal]

Your blocking file is amalgamated from 5 hosts files. The fifth likely timed out while updating. You can enter c into AB and look at the hosts_list variable.

 hosts_list="http://hosts-file.net/ad_servers.txt
 http://hosts-file.net/emd.txt
 http://hosts-file.net/grm.txt
 http://hosts-file.net/mmt.txt
 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
 http://adblock.mahakala.is"

In the above example, six hosts files are used, http://adblock.mahakala.is is the last one.
If your blocking file is a standard file from the selection in b then there's nothing to worry about. These servers may timeout every now and then.
That is why the backup is built in.
Run u to manually update the blocking file, most likely it will update without any errors.

I ran the u command in absolution and it fails. Message is:

Your blocking file is amalgamated from 5 hosts files. The fifth likely timed out while updating. You can enter c into AB and look at the hosts_list variable.

 hosts_list="http://hosts-file.net/ad_servers.txt
 http://hosts-file.net/emd.txt
 http://hosts-file.net/grm.txt
 http://hosts-file.net/mmt.txt
 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
 http://adblock.mahakala.is"

In the above example, six hosts files are used, http://adblock.mahakala.is is the last one.
If your blocking file is a standard file from the selection in b then there's nothing to worry about. These servers may timeout every now and then.
That is why the backup is built in.
Run u to manually update the blocking file, most likely it will update without any errors.

The only way I could fix this was raise my protectonic level. My thoughts are that the program rewrote the host files to the new configuration when applied. Seems to work well using the protection level you use. I think I'll go with this. Everything works!

 

[Xentrk]

I recently had the USB drive fail that was connected to my router. There goes AB-Solution... no problem, I just re-installed. There goes my whitelist... oh boy... big problem. Months of adding this and that to the file and documenting all the additions with ' # descriptions ' so I'd know why they were there. Gone. Well, that got me thinking. What are the chances that the whitelist could be emailed as either an attachment or in the message body of an email, as a way to back up its contents? This could even be included with the weekly 'Router Stats' email. Sure would be a great addition for version 4.0!

Another way is to use an SSH client software to drag and drop the files to your PC. I use MobaXterm. I create an SFTP terminal session to the router and navigate to the folder where the blacklist and whitelist files are located. For SFTP session to work, you need to install entware package openssh-sftp-server. In the pane with my PC directories, I then navigate to the folder on my pc where I want to copy the files to. I then drap and drop the files.

I also recommend you check the ASCII Mode box to prevent DOS characters from getting loaded in the file upon a drag and drop from the PC to the Router.

 

[thelonelycoder]

I recently had the USB drive fail that was connected to my router. There goes AB-Solution... no problem, I just re-installed. There goes my whitelist... oh boy... big problem. Months of adding this and that to the file and documenting all the additions with ' # descriptions ' so I'd know why they were there. Gone. Well, that got me thinking. What are the chances that the whitelist could be emailed as either an attachment or in the message body of an email, as a way to back up its contents? This could even be included with the weekly 'Router Stats' email. Sure would be a great addition for version 4.0!

Sure, I'll add it to the to do list.
A backup is always good.
Use WinSCP to save them on a Windows computer.

 

[thelonelycoder]

Another way is to use an SSH client software to drag and drop the files to your PC. I use MobaXterm. I create an SFTP terminal session to the router and navigate to the folder where the blacklist and whitelist files are located. For SFTP session to work, you need to install entware package openssh-sftp-server. In the pane with my PC directories, I then navigate to the folder on my pc where I want to copy the files to. I then drap and drop the files.

View attachment 9064

I also recommend you check the ASCII Mode box to prevent DOS characters from getting loaded in the file upon a drag and drop from the PC to the Router.

View attachment 9065

Most SSH clients support the SCP protocol, WinSCP does, you just have to select it and it works just as well as SFTP.
This way, you don't need to rely on the device being mounted. Without the device, the SFTP protocol is not available. SCP always works as it is built in the router firmware.

 

[Twiglets]

thelonelycoder,

(Please excuse explaining your own code but it is for the other readers of this post.)
I have a suggested change to the tail_logfile() subroutine [in 'functions.add'] you may find of interest.
I find that the change is useful when 'Following' an unfiltered log as it highlights the 'Blocked' entries in the context of the other log entries.

(New code in Red) (Original code in Blue)
The changes enable the lines that have been blocked either by 'pixelserv-tls' or redirection to address '0.0.0.0', to be highlighted in the listing to screen.
The highlighting is inline with the Color Theme you select via the 'ct' option.
(I have used your own variables as defined elsewhere to ensure nothing is inadvertently broken !!! )

(If you replace $igreen with $yellow the color is not changed by any 'ct' option changes excluding the 'Black & White option' which ignores all color settings.)

tail_logfile(){
trap : INT
if [ $1 == standard ];then
echo -e " --> following the logfile now (tail -F):\n"
tail -F $abSolutionPath/$logsDir/dnsmasq.log | \
while read line; do
if echo "$line" | grep -q "$custom_ipV4"; then
echo -e "$igreen$line$nc"
else
echo "$line"
fi
done
elif [ $1 == blocking ];then​

...

 

[thelonelycoder]

thelonelycoder,

(Please excuse explaining your own code but it is for the other readers of this post.)
I have a suggested change to the tail_logfile() subroutine [in 'functions.add'] you may find of interest.
I find that the change is useful when 'Following' an unfiltered log as it highlights the 'Blocked' entries in the context of the other log entries.

(New code in Red) (Original code in Blue)
The changes enable the lines that have been blocked either by 'pixelserv-tls' or redirection to address '0.0.0.0', to be highlighted in the listing to screen.
The highlighting is inline with the Color Theme you select via the 'ct' option.
(I have used your own variables as defined elsewhere to ensure nothing is inadvertently broken !!! )

(If you replace $igreen with $yellow the color is not changed by any 'ct' option changes excluding the 'Black & White option' which ignores all color settings.)

tail_logfile(){
trap : INT
if [ $1 == standard ];then
echo -e " --> following the logfile now (tail -F):\n"
tail -F $abSolutionPath/$logsDir/dnsmasq.log | \
while read line; do
if echo "$line" | grep -q "$custom_ipV4"; then
echo -e "$igreen$line$nc"
else
echo "$line"
fi
done
elif [ $1 == blocking ];then​

...

Thanks, and it does look prettier and better readable.
I'm going to add it to tail options 1-3, making it consistent with option 4, where colored output is already built in.

 

[Tastannin]

Hi @thelonelycoder! Thank you for AB-Solution! It's been nice to work from home and have no ads bugging me.

I have a question about adding pr0n blocking - I have manually added some pr0n blocklists to the custom blocking file. However, I'd like to be able to have it automatically update like the regular ad blocklists do (Options 1-6) - is there any way you could add a menu option for adding pr0n or other options from StevenBlack, for example?

Thanks again - keep up the good work!

 

[thelonelycoder]

Hi @thelonelycoder! Thank you for AB-Solution! It's been nice to work from home and have no ads bugging me.

I have a question about adding pr0n blocking - I have manually added some pr0n blocklists to the custom blocking file. However, I'd like to be able to have it automatically update like the regular ad blocklists do (Options 1-6) - is there any way you could add a menu option for adding pr0n or other options from StevenBlack, for example?

Thanks again - keep up the good work!

How did you add the unmentionable hosts files?
The custom blocking file updates the same way as the preset ones, once per week.

 

[Twiglets]

Thanks, and it does look prettier and better readable.
I'm going to add it to tail options 1-3, making it consistent with option 4, where colored output is already built in.

Thanks.

I did not look at the other options .... (Duh!)
I only ever use tail option 1 so stopped reading when i got to the code I needed to change.

 

[thelonelycoder]

Thanks.

I did not look at the other options .... (Duh!)
I only ever use tail option 1 so stopped reading when i got to the code I needed to change.

Haha, good to know AB is ahead of users wishes...

The code part your're looking for is in the tail_domain() function right below.
A bit of trickery going on in the option 4 tailing to make it work, but I'm sure you'll get how it's done.

 

[mike37]

Greetings all. Trying to install on Asuswrt-Merlin AC-68; command:

curl -O ab-solution.inf
(gives 2 error messages)
curl: Remote file name has no length!
curl: try 'curl --help' or 'curl --manual' for more information

 

[thelonelycoder]

Greetings all. Trying to install on Asuswrt-Merlin AC-68; command:

curl -O ab-solution.inf
(gives 2 error messages)
curl: Remote file name has no length!
curl: try 'curl --help' or 'curl --manual' for more information

Where did you get these commands from?
Copy and paste the whole line into your SSH terminal, not just part of it.
Just as it says on the webste, it even has pictures.
Please read the instructions, manual and FAQ if you are unfamiliar with terminal commands.

 

[mike37]

Thank you the lonelycoder and cybrnook for the quick replies.

The first command I see on http://www.ab-solution.info/install/install.html is "curl -O ab-solution.inf" which is what I entered, and which prompted my posting.

cybrnook, thanks for the copy-able posting (I couldn't copy it from the web page) of the correct command which appears farther down the page.

Installation works well; VERY NICE script IMHO!
As you likely know, the medium block list seems a good level; quite comprehensive and seems to meet my needs.

TU

 

[thelonelycoder]

Thank you the lonelycoder and cybrnook for the quick replies.

The first command I see on http://www.ab-solution.info/install/install.html is "curl -O ab-solution.inf" which is what I entered, and which prompted my posting.

cybrnook, thanks for the copy-able posting (I couldn't copy it from the web page) of the correct command which appears farther down the page.

Installation works well; VERY NICE script IMHO!
As you likely know, the medium block list seems a good level; quite comprehensive and seems to meet my needs.

TU

What browser/OS are you using that the line is broken?

 

[mike37]

Firefox on QubesOS; fedora-20 X64 vm

(yes, old OS, but not usually problematic)

(NoScript and Ublock Origin are installed, but I deactivated each.)