AB-Solution - The Ad Blocking Solution

[Xentrk]

A (possibly premature) warning regarding the list above: Over the past few days I discovered some issues with several Google services, and basically the only thing I've changed (manually) is adding this list to blacklist.txt and process it.

Youtube stopped working on Chromecast and on AppleTV (3rd gen)

Google Keep doesn't work anymore as it did. I can still save snippets, pages or bookmarks to Google Keep, but opening keep.google.com gives me a crippled page without my 'kept' content and no usable buttons.

@thelonelycoder: If I hash out the listed IP's above en re-run 'Process blacklist' will that be sufficient to test whether that will restore functionality? Or do I have remove them and add them to whitelist.txt and process that?

I think I would have to test them one by one to see whether the functionality I need isn't broken.

Thanks for the feedback.

Last night, I encountered the issue with the gmail user icon again. When clicking on it, the box does not display with the option to logout. It was getting late and did not have time to work on it.

Before this change, I never saw YouTube ads when watching on my Roku. I would see them on the iPad and Windows 10 browser. But I could select to skip the ad withing 5 seconds. So, ads were not that big of an issue for e.

I can resume testing in a few days. I have to spend time preparing for a presentation on Tuesday.

 

[MarCoMLXXV]

Thanks for the feedback.

Last night, I encountered the issue with the gmail user icon again. When clicking on it, the box does not display with the option to logout. It was getting late and did not have time to work on it.

Most welcome @Xentrk, and thank you for sharing the list .

I have the same issue with the Gmail icon, I just realized. Not using it to log out, but I often use it to login to multiple Gmail-accounts simultaneously. I tried that a few days ago, thought it was a temporal glitch and didn't even think about the hosts we added to blacklist.txt.

I will probably remove them for now and start adding them one by one and see for a day or two if it effects any functionality I (or the wife or the kiddo) use. They'll definitely let me know if 'the Internet is broken again' .

I can imagine devices like Chromecast and AppleTV use different streams, not setting cookies (checkout the domain names in the list, doesn't seem unlikely those are actually for streaming to these kind of devices), so I think we just blocked a bit too rigorous. Luckily, you'll definitely find out at some point and remember the list of hosts we added... Let's just keep sharing our results and I'm sure in the end we'll have a clear image of what breaks things and what stops ads.

 

[MarCoMLXXV]

My preliminary findings:

Adding a hash in front the lines in blacklist.txt (with a text editor like nano in a ssh terminal) and then reprocessing the blacklist by running ab-solution, entering 'el' followed by ´3' to reprocess the black- (and white)list, apparently unblocks the lines you just commented out with a hash.

When adding a hash at beginning of the lines www.youtube-nocookie.com and youtube-nocookie.com, like this

#192.168.1.2 youtube-nocookie.com
#192.168.1.2 www.youtube-nocookie.com

Google Chromecast and AppleTV were able to connect and play Youtube videos again. Even though youtube-nocookie.com is not directly registered by Google, it's registered by Markmonitor.com, apparantly a company protecting the large brands and the domain(s) is/are used to provide embedded Youtube-videos, for example on devices like AppleTV, ChromeCast but also Smart TV's and so on.

Gstatic.com is a whole different story, but I had no to choice (at least for now) then to unblock them for now:

#192.168.1.2 www.gstatic.com
#192.168.1.2 gstatic.com

Owned by Google, causing a tremendous amount of annoyance by the masses as it slows down many (if not all) of Googles Services, yet blocking the entire domain ain't that simple and not recommended if you use any of Google's services. It serves many purposes, good and bad. Google uses it to to push it's static content throughout the world, many of the icons of Google's services and other static content are stored there, yet Google uses it for it's telemetry too, Googles reCaptcha won't work anymore (script+css). It also provides Googles Webfonts to many of Google's services and according to some, which I haven't tried, Google Images stops displaying images when gstatic.com is blocked too. So whether you hate it or like it, if you use one or more Google Services, it'll be basically impossible to block it. Contrary to most others, this is a smart move of Google to combine services they need to find out more about you and your activities on the web, and content you need to have their services function properly. Just Google it (no pun intended), there's many more to it...

I'll look into the others tomorrow. At least the kiddo can watch Youtube again... yay (mostly for me, that is)

Edit: interesting article about Markmonitor.com:
http://www.abovetopsecret.com/forum/thread837847/pg1 (it's a copy of the original at http://www.pastie.org/3867284, which is blocked so I posted this link instead).

 

[FirewallFaucet]

I came across this post about blocking youtube ads:

Some of the hosts already exist in ab-solution. Following are the ones I found that do not. I added them to my blacklist.

[/CODE]

I had to remove "gstatic.com" and "clients1.google.com" as it was blocking me from seeing the " ✔ Are you a human?" security check for some important sites (not google sites). The sites you recommeneded to blacklist did seem to work on my end, will see if it continues to be effective with those two removed as well. (It was working fine on my roku though on occasion it would say "Youtube cannot run right now: "OK" "Exit" and i'd hit "OK" to keep going with no problems)

 

[MarCoMLXXV]

@FirewallFaucet: That sounds like about the same sort of issues I summarized above your post, except clients1.google.com is still blocked at my side. Will see if I can find a Google ReCaptcha today to try out.

 

[Przem]

Hello all,
I am currently using Pihole/RaspberryPi3 as adblocker and my XWRT-Vortex flashed R7000, also YA-Malware-Blocker, and I was wondering what are differences/advantages of using AB-Solution vs Pihole.
I have not found any comparison/user experiences in here.
P.

 

[MarCoMLXXV]

In addition to my previous post, I had to unblock

clients1.google.com
s.ytimg.com

this morning as well, as Youtube refused to play on AppleTV and these two came up when following AB-Solution's unfiltered log ( from within AB-Solution's main menu, you can follow by 'f', '1' (for unfiltered i.e. all dnsmasq output) and Enter to confirm). Lines showing up in green are redirected to the Pixelsrv IP, in other words: being blocked. So, from the original list,

These are still being blocked:

0.0.0.0 clients.l.google.com
0.0.0.0 www-googletagmanager.l.google.com
0.0.0.0 googleadapis.l.google.com
0.0.0.0 ade.googlesyndication.com

The first host listed is questionable, lots of info can be found about it, for example here. Apparently that host is used for many different purposes as well, as the list at the bottom of the page lists just about every Google service I heard off. I'll see how long it takes before I find out blocking that host cripples some service I use as well. As for the remainder, looking at the hostnames, these will most likely be actual ad (-related) servers.

To be continued...

 

[MarCoMLXXV]

Hello all,
I am currently using Pihole/RaspberryPi3 as adblocker and my XWRT-Vortex flashed R7000, also YA-Malware-Blocker, and I was wondering what are differences/advantages of using AB-Solution vs Pihole.
I have not found any comparison/user experiences in here.
P.

I have no experience with Pihole myself, but I'm sure @thelonelycoder or @Jack Yaz can explain the differences to you.

 

[thelonelycoder]

@thelonelycoder: If I hash out the listed IP's above en re-run 'Process blacklist' will that be sufficient to test whether that will restore functionality? Or do I have remove them and add them to whitelist.txt and process that?
I think I would have to test them one by one to see whether the functionality I need isn't broken.

To clarify this again: The blacklist is loaded separately from the blocking file into Dnsmasq. So, whatever you put into blacklist.txt is treated completely separate from what is in the blocking file.
The whitelist distracts it's content from the blocking file. If you remove an entry in the whitelist through the el function, it is re-added to the blocking file.

Manually out-commenting entries in the whitelist and blacklist is basically the same as removing them. But manually removing entries in the whitelist does not automatically add them back to the blocking file. This only works if you do it through the AB-UI.
Be aware that running the sort and confirm function in el will remove ALL out-commented entries except the explanatory ones AB places at the top of the files.

 

[thelonelycoder]

In addition to my previous post, I had to unblock

clients1.google.com
s.ytimg.com

this morning as well, as Youtube refused to play on AppleTV and these two came up when following AB-Solution's unfiltered log ( from within AB-Solution's main menu, you can follow by 'f', '1' (for unfiltered i.e. all dnsmasq output) and Enter to confirm). Lines showing up in green are redirected to the Pixelsrv IP, in other words: being blocked. So, from the original list,

These are still being blocked:

0.0.0.0 clients.l.google.com
0.0.0.0 www-googletagmanager.l.google.com
0.0.0.0 googleadapis.l.google.com
0.0.0.0 ade.googlesyndication.com

The first host listed is questionable, lots of info can be found about it, for example here. Apparently that host is used for many different purposes as well, as the list at the bottom of the page lists just about every Google service I heard off. I'll see how long it takes before I find out blocking that host cripples some service I use as well. As for the remainder, looking at the hostnames, these will most likely be actual ad (-related) servers.

To be continued...

I'm down to none of the blacklist entries @Xentrk posted earlier.
Erratic behaviors in browsers and various apps was the result. So I removed all of them.

 

[thelonelycoder]

Hello all,
I am currently using Pihole/RaspberryPi3 as adblocker and my XWRT-Vortex flashed R7000, also YA-Malware-Blocker, and I was wondering what are differences/advantages of using AB-Solution vs Pihole.
I have not found any comparison/user experiences in here.
P.

Advantages: No extra device, use your Raspberry for something more fun and let your router do a bit of extra work. The ad-blocking workload for the router when running AB-Solution is minimal, even if pixelserv-tls is used.
With pixelserv-tls installed, AB-Solution properly blocks https / SSL ad-servers and may even completely remove the space the ad would fill on the web page. This depends on the form the ad is served. Pi-Hole cannot do any of these the way AB-Solution/pixelserv-tls does.
AB-Solution can send stats, backups, update notifications for router firmware and AB-Solution.

Differences: AB-Solution "only" has a shell script user interface at the moment.

 

[MarCoMLXXV]

To clarify this again (..)

Sorry, had no intention to annoy you again. I checked the first post, checked the manual and the FAQ on the website, couldn't find an answer (or might have just missed it, as it has quite a lot of documentation with many features added over time), so thought that the best way of finding out was asking it here. I sincerely apologize for not having read 72 pages with posts prior to asking. I fully understand that to you this is all plain and clear, for some (or maybe many?) others, it's not, as it's a feature-packed ad-blocking solution. If it was plain and clear, I wouldn't have asked it. I only have basic knowledge, so I need some help from time to time.

I had one more question, which I basically used nearly all morning for, to find an answer to it, but I haven't (yet). Instead of asking here whether it will affect the generation of AB-Solutions' weekly stats, I'll have to learn myself how to interpret your code to see whether it will or won't affect it.

I'm down to none of the blacklist entries @Xentrk posted earlier.
Erratic behaviors in browsers and various apps was the result. So I removed all of them.

Thanks for sharing that info. Would have saved me several hours of searching for answers why several issues occurred, time which I had rather spent on other things. Luckily, Youtube works again, so dad's gonna a take a nap, otherwise there'll be not enough energy left for the rest of the day.

@Xentrk: thanks again for sharing the list you found on Reddit, but given the findings of @thelonelycoder, I'm gonna remove them too. If you're experiencing issues, start by remove the hosts of the list you posted from blacklist.txt, it'll save you a bunch of time.

 

[kingskib]

I will check those settings tonight. I have a feeling the wireless client was connected to their work VPN and was using that route for accessing the web.

Thank you for the reply and great tool.

It was indeed the

Are those two the same device? If so, make sure the wireless adapter does not have any other entries for the Gateway and DNS Server in the settings (ipconfig/all).
Also, if you have IPv6 disabled on the router, make sure to not enable it on the devices adapters.
On Firefox, you can disable IPv6 in the about:config, just enter ipv6 into the search field, there are two settings that are set to on by default.

I will check those settings tonight. I have a feeling the wireless client was connected to their work VPN and was using that route for accessing the web.

Thank you for the reply and great tool.

It was indeed the VPN connection that is causing the issue.

 

[Xentrk]

@Xentrk: thanks again for sharing the list you found on Reddit, but given the findings of @thelonelycoder, I'm gonna remove them too. If you're experiencing issues, start by remove the hosts of the list you posted from blacklist.txt, it'll save you a bunch of time.

It was worth a try. Thanks for letting me know. The only issue I found was with the gmail user icon. I'll remove them based on your testing. I don't see that many ads anyway on YouTube.

I did spend some time today debugging why a certain media streaming channel (starts with a C and ends with an S) was not working for me. I changed to AB Max recently. I started working backwards until settling on shooter40sw list. I also made the corresponding change in pfBlockerNG on pfSense. No more commercials. I gave trying to id the offending host(s) a try but had no luck. I tried many that I saw listed. Glad it finally worked.

 

[thelonelycoder]

Sorry, had no intention to annoy you again

Now I know where this is coming from: Don't take this personal, it's just my writing style and the way I express myself.
I am in no way annoyed, so, it's no big deal. In fact, I am OK.

I had one more question, which I basically used nearly all morning for, to find an answer to it, but I haven't (yet). Instead of asking here whether it will affect the generation of AB-Solutions' weekly stats, I'll have to learn myself how to interpret your code to see whether it will or won't affect it.

Ask away. This is a discussion forum.

 

[thelonelycoder]

MarCoMLXXV is now a Guest here on the board.
Can anyone enlighten me what I did wrong this time?

 

[Jack Yaz]

MarCoMLXXV is now a Guest here on the board.
Can anyone enlighten me what I did wrong this time?

Nothing!

 

[thelonelycoder]

Nothing!

I do miss your old avatar.
Nothing as in "I did not do anything" or "I can't read between the lines"?

 

[Jack Yaz]

I do miss your old avatar.
Nothing as in "I did not do anything" or "I can't read between the lines"?

I changed it on Gravatar not knowing this board also used it.

I can't see that you did anything wrong, I think Marco thought you were annoyed when you were just trying to state things for clarity and completeness for others following the thread. That's my take anyway.

Now what's the digital equivalent of getting a pint and waiting for it to blow over?

 

[thelonelycoder]

I changed it on Gravatar not knowing this board also used it.

I can't see that you did anything wrong, I think Marco thought you were annoyed when you were just trying to state things for clarity and completeness for others following the thread. That's my take anyway.

Now what's the digital equivalent of getting a pint and waiting for it to blow over?

Move the browser window to the left monitor, move Notepad++ to the main center one, have file explorer and WinSCP open on the right and Xshell 5 at the top monitor, dim brightness and turn up the volume and start coding.