AB-Solution - The Ad Blocking Solution

[thelonelycoder]

I am using 5 and set it up in the blacklist as it was not being blocked.

What @visortgw means: The blocking file and the blacklist are two different things, just to make this clear.
The blocking file is managed with (b) and auto-updated weekly.
Entries in the blacklist are manually edited by the user and have no update function.

 

[elorimer]

There are a whole pile of useful things that run off aws. I run a cheap instance for myself as a backup, too. And, of course, some kindle/echo stuff in addition to example.net. So I wouldn't want to block aws without understanding it better.

 

[ndtemple417]

Anyway to block youtube ads, or add a custom list for youtube ads? Sorry I'm new to this and I hate seeing ads on my vizio smart tv on youtube. Thanks so much for this great tool also by the way.

 

[thelonelycoder]

Anyway to block youtube ads, or add a custom list for youtube ads? Sorry I'm new to this and I hate seeing ads on my vizio smart tv on youtube. Thanks so much for this great tool also by the way.

If it were that simple, blocking these ads would be built in.

 

[ndtemple417]

If it were that simple, blocking these ads would be built in.

LOL sorry for asking

 

[jack901]

so there is no way to block by name if it has multiple ips for the same name?
I am using it from the blacklist that you manually enter. and even tho it is the same name it will pull the new ip and go thru instead of blocking by name.

What @visortgw means: The blocking file and the blacklist are two different things, just to make this clear.
The blocking file is managed with (b) and auto-updated weekly.
Entries in the blacklist are manually edited by the user and have no update function.

 

[jtp10181]

You have to each add them as a separate entry.
The wildcard * DOES NOT WORK in white- and blacklist.

I tested this before I saw this post and it does work (sort of), or does it cause some other issues?

Before adding to whitelist.txt

 grep 'google-analytics.com' blocking_file
0.0.0.0 google-analytics.com
0.0.0.0 ssl.google-analytics.com
0.0.0.0 www.google-analytics.com

Then I added this to whitelist.txt

*.google-analytics.com

And it filters out ALL 3 of the entries above, which is not what I would expect at all.

So I did a little investigation and found that the whitelist entries are being removed using egrep -v (-v echos everything NOT matched)
And there is a space padded before and an end line anchor added after, i guess the space will match the space between the 0.0.0.0 and the domain.
So what is getting checked after processing is ' *.google-analytics.com$'

So using this knowledge, I realized that the '.' will match "anything" since it is a regex match.
For some reason the ' *' (thats: [space]*) is matching anything, any number of times, not sure why because that is not normal for regex. The space should only match a space. And it only seems to work at the start of the match.

Anyway I have some ideas that would make it more robust and also allow a standard wildcard if you are interested.
As it is right now, the '.' used in the domains is matching "anything" and has a very SMALL potential for false matches as intended.

For example:

Whitelist entry:
googl..com

Would match:
google.com
googly.com
googlr.com
googl.zcom

etc....

 

[thelonelycoder]

so there is no way to block by name if it has multiple ips for the same name?
I am using it from the blacklist that you manually enter. and even tho it is the same name it will pull the new ip and go thru instead of blocking by name.

What exact domain name are we talking about/want to block?

 

[thelonelycoder]

I tested this before I saw this post and it does work (sort of), or does it cause some other issues?

Before adding to whitelist.txt

 grep 'google-analytics.com' blocking_file
0.0.0.0 google-analytics.com
0.0.0.0 ssl.google-analytics.com
0.0.0.0 www.google-analytics.com

Then I added this to whitelist.txt

*.google-analytics.com

And it filters out ALL 3 of the entries above, which is not what I would expect at all.

So I did a little investigation and found that the whitelist entries are being removed using egrep -v (-v echos everything NOT matched)
And there is a space padded before and an end line anchor added after, i guess the space will match the space between the 0.0.0.0 and the domain.
So what is getting checked after processing is ' *.google-analytics.com$'

So using this knowledge, I realized that the '.' will match "anything" since it is a regex match.
For some reason the ' *' (thats: [space]*) is matching anything, any number of times, not sure why because that is not normal for regex. The space should only match a space. And it only seems to work at the start of the match.

Anyway I have some ideas that would make it more robust and also allow a standard wildcard if you are interested.
As it is right now, the '.' used in the domains is matching "anything" and has a very SMALL potential for false matches as intended.

For example:

Whitelist entry:
googl..com

Would match:
google.com
googly.com
googlr.com
googl.zcom

etc....

IF IT SAYS do not use wildcard. THEN I MEAN IT. Dnsmasq CANNOT handle that. Simple as that.

 

[thelonelycoder]

So using this knowledge, I realized that the '.' will match "anything" since it is a regex match.
For some reason the ' *' (thats: [space]*) is matching anything, any number of times, not sure why because that is not normal for regex. The space should only match a space. And it only seems to work at the start of the match.

Anyway I have some ideas that would make it more robust and also allow a standard wildcard if you are interested.
As it is right now, the '.' used in the domains is matching "anything" and has a very SMALL potential for false matches as intended.

That is fixed in my dev version.

 

[tomsk]

so there is no way to block by name if it has multiple ips for the same name?
I am using it from the blacklist that you manually enter. and even tho it is the same name it will pull the new ip and go thru instead of blocking by name.

AB-solution blocks by hostname.Are you talking about canonical (CNAME) name records?... example below

Feb 25 19:26:09 dnsmasq[22195]: query[A] white.ish.instagram.com from 192.168.2.221
Feb 25 19:26:09 dnsmasq[22195]: forwarded white.ish.instagram.com to 10.0.0.1
Feb 25 19:26:09 dnsmasq[22195]: reply white.ish.instagram.com is <CNAME>
Feb 25 19:26:09 dnsmasq[22195]: reply instagram.com is 34.199.72.183
Feb 25 19:26:09 dnsmasq[22195]: reply instagram.com is 52.200.144.118
Feb 25 19:26:09 dnsmasq[22195]: reply instagram.com is 52.0.143.223
Feb 25 19:26:09 dnsmasq[22195]: reply instagram.com is 52.1.164.9
Feb 25 19:26:09 dnsmasq[22195]: reply instagram.com is 34.198.80.66
Feb 25 19:26:09 dnsmasq[22195]: reply instagram.com is 52.2.182.190
Feb 25 19:26:09 dnsmasq[22195]: reply instagram.com is 52.20.169.144
Feb 25 19:26:09 dnsmasq[22195]: reply instagram.com is 52.0.251.243

the client machine queries white.ish.instagram.com which is not blocked by AB and then passed by dnsmasq to the upstream DNS resolver ( in this case my VPN provider DNS). white.ish.instagram.com is actually just an alias for a bunch of other domains the IP addresses of which will now be returned to the client machine.

For a blocked domain the dnsmasq log looks like this

Feb 25 19:31:42 dnsmasq[22195]: query[A] fls-na.amazon-adsystem.com from 192.168.2.60
Feb 25 19:31:42 dnsmasq[22195]: /tmp/mnt/data/adblocking/blocking_file fls-na.amazon-adsystem.com is 192.168.2.3

the client machine queries fls-na.amazon-adsystem.com which is a domain listed on the AB hostfile. The query is redirected ( in this case to my pixelserv-tls server, but could also be to the 0.0.0.0 null address if pixelserv not enabled) and the query never makes it to the upstream resolver whether it is a CNAME or not.

So blocking the one CNAME domain stops all those canonical domains from returning multiple IPs to the client.

 

[thelonelycoder]

Then I added this to whitelist.txt

*.google-analytics.com

The main script is not the only script using the whitelist, what might work here may not work in the other scripts.
Next thing users are going to add it to the blacklist, which will fail miserably. So, just don't fckn mention it. I will have to support the users, not you.

NO WILDCARD (*) IN THE WHITE- AND BLACKLIST. PERIOD.

 

[tomsk]

 

[jack901]

it catches and blocks the first one , then it is called again and is accepted as shown in the log below. first one blocked then called again (same name different ip) I would think it would just use the blacklist again but for some reason it does not.

/tmp/mnt/ASUS/adblocking/blacklist.txt ec2-54-225-122-86.compute-1.amazonaws.com is 0.0.0.0
Feb 25 05:23:11 dnsmasq[4253]: query[A] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.15.2
Feb 25 05:23:11 dnsmasq[4253]: /tmp/mnt/ASUS/adblocking/blacklist.txt ec2-54-225-122-86.compute-1.amazonaws.com is 0.0.0.0
Feb 25 05:23:11 dnsmasq[4253]: query[AAAA] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.15.2
Feb 25 05:23:11 dnsmasq[4253]: forwarded ec2-54-225-122-86.compute-1.amazonaws.com to 84.200.69.80
Feb 25 05:23:11 dnsmasq[4253]: forwarded ec2-54-225-122-86.compute-1.amazonaws.com to 84.200.70.40
Feb 25 05:23:11 dnsmasq[4253]: query[AAAA] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.15.2
Feb 25 05:23:11 dnsmasq[4253]: forwarded ec2-54-225-122-86.compute-1.amazonaws.com to 84.200.69.80
Feb 25 05:23:11 dnsmasq[4253]: forwarded ec2-54-225-122-86.compute-1.amazonaws.com to 84.200.70.40

What exact domain name are we talking about/want to block?

 

[visortgw]

What exact domain name are we talking about/want to block?

He still wants to block amazonaws.com -- good luck!

 

[thelonelycoder]

I'm just pssd that now we have no longer wildcard removal in the whitelist, users want it back.
Coding can be a bich at times...

 

[jtp10181]

IF IT SAYS do not use wildcard. THEN I MEAN IT. Dnsmasq CANNOT handle that. Simple as that.

EDIT: While I was putting this up you posted some more things, and I see you just dont want to support it anymore. Understandable and I am cool with it.
---------------------------------------
Well Dnsmasq never sees it, since its in the whitelist and it processed by egrep. But yes, I understand it cannot be in the blacklist file.

I found a little magic string using regex that will actually work to filter out anything ending in a domain, like 'google.com' or 'www.google.com' but not 'not-google.com' using the current public version.

But I don't expect most people to understand regex so I wont share it here, plus it sounds like your dev version is already making some more changes.

Any way to get a peek at it? Just one dev to another. I just think it would be really nice (for all users) to support a basic wildcard in the whitelist file. Unless you want to support full blown regular expressions. That would work too, advanced users could use regex, and basic users could have a single line for every subdomain. Unless there is a reason to not support it at all in the whitelist file.

The biggest thing where it is nice, is if you have example of domain.com that serves ads but you need to see them for some game you are playing, ads1.domain.com and ads2.domain.com are in the host file so you whitelist them. Lets say 2 weeks later ads3.domain.com gets added to the hosts file. You would have to manually catch this and whitelist it, where as a whitelist of *.domain.com would already catch it.

Please understand, I know this is your tool, and it is awesome. I am not making demands, I am just tossing out suggestions. If you don't want to do any of that, or do something totally different I would still support it.

 

[tomsk]

I'm just pssd that now we have no longer wildcard removal in the whitelist, users want it back.
Coding can be a bich at times...

Just no pleasing some people ...hehe .... the wildcard is a sledgehammer to crack a nut IMHO, Of course individual domain matching is more tedious to setup, but i think it pays off in less unwanted blocks later on.

"Block all near matches" option for the impatient?

 

[thelonelycoder]

it catches and blocks the first one , then it is called again and is accepted as shown in the log below. first one blocked then called again (same name different ip) I would think it would just use the blacklist again but for some reason it does not.

/tmp/mnt/ASUS/adblocking/blacklist.txt ec2-54-225-122-86.compute-1.amazonaws.com is 0.0.0.0
Feb 25 05:23:11 dnsmasq[4253]: query[A] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.15.2
Feb 25 05:23:11 dnsmasq[4253]: /tmp/mnt/ASUS/adblocking/blacklist.txt ec2-54-225-122-86.compute-1.amazonaws.com is 0.0.0.0
Feb 25 05:23:11 dnsmasq[4253]: query[AAAA] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.15.2
Feb 25 05:23:11 dnsmasq[4253]: forwarded ec2-54-225-122-86.compute-1.amazonaws.com to 84.200.69.80
Feb 25 05:23:11 dnsmasq[4253]: forwarded ec2-54-225-122-86.compute-1.amazonaws.com to 84.200.70.40
Feb 25 05:23:11 dnsmasq[4253]: query[AAAA] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.15.2
Feb 25 05:23:11 dnsmasq[4253]: forwarded ec2-54-225-122-86.compute-1.amazonaws.com to 84.200.69.80
Feb 25 05:23:11 dnsmasq[4253]: forwarded ec2-54-225-122-86.compute-1.amazonaws.com to 84.200.70.40

I'm not seeing that here. This is my output with and without pixelserv-tls on. Something is bypassing your query.

Feb 25 17:21:44 dnsmasq[14167]: /tmp/mnt/absolution/adblocking/blacklist.txt ec2-54-225-122-86.compute-1.amazonaws.com is 0.0.0.0
Feb 25 17:21:44 dnsmasq[14167]: query[A] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.2.200
Feb 25 17:21:44 dnsmasq[14167]: /tmp/mnt/absolution/adblocking/blacklist.txt ec2-54-225-122-86.compute-1.amazonaws.com is 0.0.0.0
Feb 25 17:21:49 dnsmasq[14167]: query[A] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.2.200
Feb 25 17:21:49 dnsmasq[14167]: /tmp/mnt/absolution/adblocking/blacklist.txt ec2-54-225-122-86.compute-1.amazonaws.com is 0.0.0.0

Feb 25 17:21:02 dnsmasq[9813]: query[A] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.2.200
Feb 25 17:21:02 dnsmasq[9813]: /tmp/mnt/absolution/adblocking/blacklist.txt ec2-54-225-122-86.compute-1.amazonaws.com is 192.168.2.2
Feb 25 17:21:02 dnsmasq[9813]: query[A] ec2-54-225-122-86.compute-1.amazonaws.com from 192.168.2.200
Feb 25 17:21:02 dnsmasq[9813]: /tmp/mnt/absolution/adblocking/blacklist.txt ec2-54-225-122-86.compute-1.amazonaws.com is 192.168.2.2

 

[tomsk]

I'm not seeing that here. This is my output with and without pixelserv-tls on. Something is bypassing your query.

The IPv4 [A] record query seems to be blocked but the IPv6 [AAAA] record query is forwarded upstream.... strange....