AB-Solution - The Ad Blocking Solution

[iManuB]

Hi @thelonelycoder . I'm sorry for the trouble, but i've a question: the best swap size for my RT-AC86U? Now, i've installed Skynet+Ab-Solution with 2GB of Swap.
Thanks so much!

 

[M@rco]

Hi @thelonelycoder . I'm sorry for the trouble, but i've a question: the best swap size for my RT-AC86U?

Seems like the answer to your question is in the post above yours...

 

[sentinelvdx]

Hi again! Question I've noticed Amazon App was not working, so checking the unfiltered log, all connections from this app was under "forwarded xxxxxx.amazon.com to 1.0.0.1" (forwarded to upstream dns).

What does this exactly means? Because the app it's unusuable, but amazon website works just perfect, so don't know how to solve a thing like this

Thanks,

Sent from S.G. S9+ Duos

 

[thelonelycoder]

Hi again! Question I've noticed Amazon App was not working, so checking the unfiltered log, all connections from this app was under "forwarded xxxxxx.amazon.com to 1.0.0.1" (forwarded to upstream dns).

That's a Cloudfare DNS Server IP.
Cloudfare uses 1.1.1.1 and 1.0.0.1, so check your WAN/DNS Server setting.
https://1.1.1.1/

 

[sentinelvdx]

That's a Cloudfare DNS Server IP.
Cloudfare uses 1.1.1.1 and 1.0.0.1, so check your WAN/DNS Server setting.
https://1.1.1.1/

Yes, I use Cloudfare's DNS at home. You say Cloudfare is kind of blocking Amazon's app?

Sent from S.G. S9+ Duos

 

[TheLyppardMan]

I'm not sure if I've explained this properly. I didn't make any changes on my PC's DNS settings, but rather, changed the settings for my PC on the router from "Router" to "OpenDNS". I think I have got this figured now anyway. I've reinstalled AB Solution, set the global setting to "OpenDNS" and any devices I want AB Solution to filter to "Router". I can't use it on my PC because it breaks some sites as explained in my other thread (I've checked the log and can't see any reason why that should happen, but it does), so I'm just using it on mobiles at the moment.View attachment 13952 View attachment 13953

I hit a problem with this method. I found that for devices that were still being filtered through A B Solution, i.e., our mobile phones (via the Router settings), everything was fine, but for the rest of the devices which were by-passing A B Solution via the Global Setting (OpenDNS in our case), the script to enforce Google Safe Search was also being by-passed. It came down to either losing A B Solution or Safe Search, so the former had to go. We’ll just have to put up with advertisements on our mobiles and use uBlock Origin or equivalent on our PCs/Laptops.

 

[JasonReck]

OKay I had AB-solution working fine for weeks on my RT-AC68U, using firmware 384.5, until recently. I looked and the router showed the USB drive wasn't detected. I turned the router off and back on. It then detected the USB 3.0 drive, but I noticed both CPU cores were pegged at 100%. I used putty to connect to the router, but it was very slow and when I tried to open ab-solution, suddenly the drive was not found again. Any help troubleshooting this please?

 

[bengalih]

Hey @thelonelycoder - continuing some conversation over from the pixelserv thread to some questions more germane to just ab-solution.

I've got a feature request as it were regarding logging as well as some other related questions. Maybe the feature isn't needed and there is an easier way to find what I'm doing. Basically, I'm trying to find out what is being blocked for a particular client IP. Using the log to just show blocked domains isn't effective because it doesn't show any client info, like so:

Hit CTRL-C to show options while following log file

 --> following the logfile now (tail -F | grep "blocking_file\|blacklist.txt"):

Aug 20 12:15:26 dnsmasq[23451]: /tmp/mnt/flash_drive/adblocking/blocking_file app.appsflyer.com is 10.10.10.2
Aug 20 12:15:26 dnsmasq[23451]: /tmp/mnt/flash_drive/adblocking/blocking_file events.appsflyer.com is 10.10.10.2
Aug 20 12:15:36 dnsmasq[23451]: /tmp/mnt/flash_drive/adblocking/blocking_file app.appsflyer.com is 10.10.10.2
Aug 20 12:15:36 dnsmasq[23451]: /tmp/mnt/flash_drive/adblocking/blocking_file events.appsflyer.com is 10.10.10.2

If you use option 3 and enter in the IP for your client you get *everything* blocked or not:

Hit CTRL-C to show options while following log file

 --> following the logfile now (tail -F | grep 10.10.10.123):

Aug 20 12:16:47 dnsmasq[23451]: query[A] teleport.soom.la from 10.10.10.123
Aug 20 12:16:58 dnsmasq[23451]: query[A] www.google.com from 10.10.10.123
Aug 20 12:16:59 dnsmasq[23451]: query[A] e.crashlytics.com from 10.10.10.123
...
Aug 20 12:17:19 dnsmasq[23451]: query[A] www.toyota.com from 10.10.10.123
...
Aug 20 12:17:31 dnsmasq[23451]: query[A] dpm.demdex.net from 10.10.10.123

I can of course use an unfiltered log but that is a lot to weed through with multiple clients.
Instead, I can seemingly easily find what I need with this:

admin@RT-AC68U-4C30:/tmp/mnt/flash_drive/adblocking/logs# tail -f dnsmasq.log | grep -A 1 10.10.10.123
Aug 20 12:19:47 dnsmasq[23451]: query[A] app.appsflyer.com from 10.10.10.123
Aug 20 12:19:47 dnsmasq[23451]: /tmp/mnt/flash_drive/adblocking/blocking_file app.appsflyer.com is 10.10.10.2
Aug 20 12:19:47 dnsmasq[23451]: query[A] events.appsflyer.com from 10.10.10.123
Aug 20 12:19:47 dnsmasq[23451]: /tmp/mnt/flash_drive/adblocking/blocking_file events.appsflyer.com is 10.10.10.2

As you can see I simply tell grep to search for my IP term followed by the next 1 line. Theoretically it's possible that the block won't come directly after the query (either due to traffic or possibly race condition), but it seems likely. I could always increase that to 2 lines and still have a much easier time finding what I need.

Would you consider putting an option like this into the logging?

Also, I have a question about the experimental option (4). How exactly is supposed to work? When I use it entering the domain "appsflyer.com" this is what I see:

--> waiting for appsflyer.com to appear in log file:

Terminated
 (ignore 'Terminated', this cannot be suppressed)
 ---------------------------------------------------

 --> found appsflyer.com in log,
 showing requests and served blocked domains
 from device with ip: 10.10.10.123

I see this entry only once...regardless of how many times the client is trying to access that resource. Compare that to my grep statement above where I am getting an entry for every click on the resource. Also it is only stating "appsflyer.com" found in log whereas the raw log (and my grep) are showing hits to events. and app. names. I partly confused by the description in the option:

When domain appears in log file, all requests
and served blocked domains for the device are shown

I was taking that to mean that it would show me every domain/host that is being blocked.
Since I found out I can't block/whitelist on a second level domain (more on this below) this would be helpful to find *all* the hosts required for a particular function.

-------------------------------------

Next part is that about blocking second level domains. For instance I can't add "appsflyer.com" to my whitelist and get through since there are also specific entries for app.appsflyer.com, events.appsflyer.com, etc in the blocking file.
I realize/believe the problem here is the how the hosts file works/is being implemented with ab-solution.
The hosts file doesn't support full domain names (I suppose we could say wildcard here, but to clarify *.domain.com vs *domain.com. It doesn't even support the former). In the hosts file "domain.com" is totally different than "host.domain.com" and having an entry for one doesn't influence what a query for the other might do.

I'm just thinking out loud here, but have you ever considered trying to implement the "server=" and/or "address=" features in dnsmasq instead of (or in addition to) the host file? With its ability to support entire domains and the ability to return NXDOMAIN where required it might be able to provide some additional power to the application. Note I'm not at all aware of what many server/address entries might do performance wise compared to using them in a hosts file, but performance aside I think it is an interesting though exercise.

Short of this, would it not be possible (and relatively trivial) to support wildcard (*.domain.com) in the blacklists by simply allowing a user to specify this wildcard and then using grep to pull out every domain that matches? I think a lot of the work is already done as the whitelist procedure can already display all similar hosts for a domain...it would simply be a batch to add these to the whitelist.

Is this something you have ever considered?

(continued in next post...)

 

[bengalih]

-----------------------------

Last part is just an observation I made - and I bring it up just to discuss it - not sure if it is really a problem (or just an annoyance) or if there can be a great solution.

Just look at the following:

C:\Users\Ben>ping doubleclick.net
Ping request could not find host doubleclick.net. Please check the name and try again.

C:\Users\Ben>ping wer.adsfsadfsadf.com
Ping request could not find host wer.adsfsadfsadf.com. Please check the name and try again.

C:\Users\Ben>nslookup
Default Server:  router.asus.com
Address:  10.10.10.1

> doubleclick.net
Server:  router.asus.com
Address:  10.10.10.1

Name:    doubleclick.net
Addresses:  2607:f8b0:4000:813::200e
          0.0.0.0

> wer.adsfsadfsadf.com
Server:  router.asus.com
Address:  10.10.10.1

*** router.asus.com can't find wer.adsfsadfsadf.com: Non-existent domain
>

Note that a ping to either domain results in a host not found. There is literally no way for me to know as an end-user if doubleclick.net is in fact a valid domain or if it simply being blocked by ab-solution. Clearly wer.adsfsadfsadf.com is bogus, but maybe it's not? Perhaps it too is an actual domain and just being blocked. Only the nslookups give me a bit more insight where I see that 0.0.0.0 is being returned instead of NXDOMAIN and I can assume (based on my knowledge of how ab-solution works) that this is a blocked domain.

I think - based on this configuration this is a huge motivator to implement pixelserv-tls....even if you don't want to set up certificates or mess with it at all. With pixelserv set up I get the following:

C:\Users\Ben>ping doubleclick.net

Pinging doubleclick.net [10.10.10.2] with 32 bytes of data:
Reply from 10.10.10.2: bytes=32 time=494ms TTL=64
Reply from 10.10.10.2: bytes=32 time<1ms TTL=64
Reply from 10.10.10.2: bytes=32 time<1ms TTL=64
Reply from 10.10.10.2: bytes=32 time=1ms TTL=64

And know immediately this is a blocked domain because the IP of pixelserv is returned.

Anyway - just an interesting observation.

Thanks for all your work and willingness to help!

 

[Twiglets]

Re: your feature request ...... already done as suggested new filter per https://www.snbforums.com/threads/ab-solution-the-ad-blocking-solution.37511/page-151#post-401963

 

[bengalih]

Re: your feature request ...... already done as suggested new filter per https://www.snbforums.com/threads/ab-solution-the-ad-blocking-solution.37511/page-151#post-401963

Nice, thanks I'll give that a try. Just curious how that stuff works. Are you an *official*collaborator? I see that your post is 4+ months old so just curious if there hasn't been a new release since then and/or if @thelonelycoder actually plans to include this.
Not that I don't trust your work (the patch will be easy to validate), I'm just curious to know how the dev cycle is working and if this is planning to go into an official release or I will need to keep track and integrated it back into future versions.

 

[Twiglets]

Nice, thanks I'll give that a try. Just curious how that stuff works. Are you an *official*collaborator? I see that your post is 4+ months old so just curious if there hasn't been a new release since then and/or if @thelonelycoder actually plans to include this.
Not that I don't trust your work (the patch will be easy to validate), I'm just curious to know how the dev cycle is working and if this is planning to go into an official release or I will need to keep track and integrated it back into future versions.

It was a suggested new filter that I created as I needed it and thought it might be useful for others.
It is up to thelonelycoder if he wants to include it in the official release.
As he has been busy in 'real life' I have not chased.

 

[FadgewackeR]

It was a suggested new filter that I created as I needed it and thought it might be useful for others.
It is up to thelonelycoder if he wants to include it in the official release.
As he has been busy in 'real life' I have not chased.

That would indeed be useful

 

[swole]

I have a 250 MB download speed with cable company. Once enabling AB the download speed goes from about 270MB down to less that a hundred WTH. Anybody having similar issues? Turn off AB and my speeds go back to normal. Does this somehow disable the NAT HW acceleration feature?

 

[bengalih]

I have a 250 MB download speed with cable company. Once enabling AB the download speed goes from about 270MB down to less that a hundred WTH. Anybody having similar issues? Turn off AB and my speeds go back to normal. Does this somehow disable the NAT HW acceleration feature?

I'm sure @thelonelycoder and others will reply. But I'll just say that it seems pretty unlikely that AB is the culprit.

The way AB works is simply by modifying the hosts files on your router which provide DNS entries back to your client. It does this by redirecting them to 0.0.0.0, which effectively stops your computer from going there. So, for instance let's say you visit www.facebook.com and within facebook are some ads from ads.scuzzy.com. What will happen is that your router will give your PC the correct address for facebook (since it likely isn't filtered), such as 157.240.13.38, but when facebook tries to load the ad at scuzzy.com your router will tell your PC 0.0.0.0 and it won't load the ad.

Generally, this will always introduce speed *improvements* since we aren't downloading the unnecessary ads. In some cases a page might load slower if the DNS response is a little slower (usually this is in milliseconds).

However, DNS (and thus AB) are only involved in that initial loading of a resource. So, once you have the IP address and a download has started, DNS is no longer involved. Therefore if you are measuring pure download speed, it is incredibly unlikely a DNS solution is going to alter that.

Now, I suppose it is possible that AB running on your router consumes more resources (CPU) and could possibly slow down your your devices actual routing. That is theoretical and I don't think anyone has indicated that a properly working AB consumes enough CPU for this to happen. In fact, once again, AB is mostly passive as your router is just serving out the hosts records it modifies. The only time AB is really doing anything active is when it is updating lists or you are configuring it.

Why don't you provide some more info...like your router, f/w version, and anything else you did to modify.
If you go into the AB config and toggle the ad blocking off and on do you see your download speeds directly affected?

What test are you using to measure d/l?

 

[DonnyJohnny]

I have a 250 MB download speed with cable company. Once enabling AB the download speed goes from about 270MB down to less that a hundred WTH. Anybody having similar issues? Turn off AB and my speeds go back to normal. Does this somehow disable the NAT HW acceleration feature?

Suggest you call your cable company and ask them what the hell is going on..
I am on 1 gbps fiber , my download speed test is 700-900mb on avg.

Ad block improve browsing speed by removing ad. Before they even set queries out to internet, it it blocked via hosts file.
Even if you have a lot of ad block list. It shouldn’t affect your download speed, at most it may affect browsing experience depending on the performance of router to run check whether the domain is blocked. But we talking about less than 1-2 sec kind of lag.

 

[thelonelycoder]

I have a 250 MB download speed with cable company. Once enabling AB the download speed goes from about 270MB down to less that a hundred WTH. Anybody having similar issues? Turn off AB and my speeds go back to normal.

WTH? Not possible. AB cannot affect the throughput of your router.

Does this somehow disable the NAT HW acceleration feature?

No.

 

[thelonelycoder]

Would you consider putting an option like this into the logging?

I will, but not for now. The next major release of AB-Solution is based on a completely new code base. We will have to take it from there once it's out.

Also, I have a question about the experimental option (4). How exactly is supposed to work? When I use it entering the domain "appsflyer.com" this is what I see:

Once the domain is seen in the logfile, it will show only results from the IP that requested that domain.

 

[thelonelycoder]

I'm just thinking out loud here, but have you ever considered trying to implement the "server=" and/or "address=" features in dnsmasq instead of (or in addition to) the host file? With its ability to support entire domains and the ability to return NXDOMAIN where required it might be able to provide some additional power to the application. Note I'm not at all aware of what many server/address entries might do performance wise compared to using them in a hosts file, but performance aside I think it is an interesting though exercise.

The next major release of AB-Solution has wildcard-blacklist support built in. It uses the address entries in dnsmasq.

 

[bengalih]

The next major release of AB-Solution has wildcard-blacklist support built in. It uses the address entries in dnsmasq.

Sweet. Glad to see my brain brought me to a similar conclusion your did regarding how to solve the problem

I've got a couple other questions - more like growing pains adjusting to the application and was interested in your thoughts - or any AB user who might have gone through this:

1) What level blacklists to people typically use? I started off with the AdsBeGone!, but on my rebuild I just did a day or two later I opted for shooter40sw's instead because I felt that too much was being blocked. Even with this one I'm finding sites in the blacklist that I'm very surprised are there. I can't remember all of them, there were at least 2-3, but a good example was "passport.lenovo.com." This is the sign-in page for Lenovo in order to participate on their forums/support site/etc. Now I suppose it is possible that ads was served from here at some point and that is why it is on the list - but Lenovo is a solid site and to block this page seems pretty strict. I'm not sure if there is another forum for whomever maintains these lists that might be a better place to ask - but what I wanted to know here is what lists to people normally use? I feel like if I am adding 2-3 "respected" sites to my whitelist weekly that is almost too much. OTOH I don't want to water down the effectiveness of AB by choosing the low lists. Just curious if my experience is typical.

2) I've been noticing some weird issues with a couple of sites, coincidentally enough the issue so far has been most prevalant with snbforums.com. I notice that AB has auto-whitelisted some ad domains from snbforums.com - ok, cool, just mentioning that, because... What happens is that when I have AB enabled the snbforums site seems to lag painfully. It looks like the page loads, but as I try to scroll through the page and or type in this box here I get huge lags/hangs that can last well over 10 seconds. I'm using Chrome and I can constantly see in the bottom left of Chrome that it looks like it keeps trying to load other resources. If I switch AB off this doesn't happen. I realize that AB can't account for how any site is coded, but I would think that based on its affinity for snbforums that the default configuration of allowed ads would be including anything that might make site performance slow. I'm not sure how much more info I can give other than AB On = horrible lag on snbforums, AB Off = seemingly normal usage. I can basically recreate this at will. Any ideas?

thanks!