AB-Solution - The Ad Blocking Solution

[Makaveli]

is that bug still present so that I cannot unable pixelserv inside of AB?

Am I going to have to manual upgrade it?

 

[thelonelycoder]

is that bug still present so that I cannot unable pixelserv inside of AB?

Am I going to have to manual upgrade it?

Until the issue is fixed by @kvic the Entware package manager opkg does not see v2. a newer version.

I have opened an issue to help bump this up in his to-do list
https://github.com/kvic-z/pixelserv-tls/issues/11

 

[DonnyJohnny]

is that bug still present so that I cannot unable pixelserv inside of AB?

Am I going to have to manual upgrade it?

suggest uninstall and install them separately, then go back ab-solution to enable it.

 

[thelonelycoder]

Can I has AB 4 with WebUI only? (without ssh ui and without pixelserv-tls)

Your remark got me thinking. Why shoo out the ones that only want basic http blocking by denying them the new shiny WebUI?

I will either add an option to install the WebUI or make it the default for all Editions, along of course with the traditional SSH UI.
I just have to figure out a way to do that without having to rewrite too much code I already wrote for that part.

 

[XIII]

On iOS it does not seem to fail gracefully as their appears a pop-up for every HTTPS request captured by pixselserv?

Unable to reproduce so far... (not sure whether the update to iOS 11.2.5 changed anything)

 

[JJQuin]

I think I found a possible solution, well more appropriate a work around for the DNS leaks that occur with AB-Solution and OpenVPN Clients on Asus Merlin. Was chatting with @Xentrk and he recommended TorGuard. I looked at their knowledge base and they have a script to route specific port/ip combinations through the Asus Merlin OpenVPN Client. I was thinking maybe I could set the script up to forward port 53 (DNS) to my VPN service's DNS server using the script. This is more of a work-around since this would cause ALL dns traffic to go through the VPN instead of just the DNS resolves from the devices using the OpenVPN Client on the router. In most cases that shouldn't cause an issue with clients bypassing the VPN.

TorGuard script and instructions:
https://torguard.net/knowledgebase.php?action=displayarticle&id=165

I was thinking this might be an even better solution since so many ISP's are either spying on their customers' DNS traffic or even setting up a transparent DNS proxy to force their customers to use their DNS servers. This way all DNS traffic would be encrypted and go to a DNS server that you could be confident is not keep or selling its logs!

Transparent DNS Proxy:
https://www.dnsleaktest.com/what-is-transparent-dns-proxy.html

The only other solution I could think of would be to setup an unbound server either on the router or another device on the local network and force DNS traffic to use DNS over TLS. But that is a lot more work. I know some people would recommend DNScrypt but it doesn't offer any real privacy since only a small part of each packet is encryted (just the DNS resolve).

DNS over TLS vs DNScrypt:
https://tenta.com/blog/post/2017/12/dns-over-tls-vs-dnscrypt

Unbound:
https://calomel.org/unbound_dns.html

It's going to be a few days before I can try this since my children are home for the weekend and they HATE when I take down the router for testing!

Anyway I was curious if anyone else had tried this successfully or was willing to try. Thanks!

 

[thelonelycoder]

I think I found a possible solution, well more appropriate a work around for the DNS leaks that occur with AB-Solution and OpenVPN Clients on Asus Merlin. Was chatting with @Xentrk and he recommended TorGuard. I looked at their knowledge base and they have a script to route specific port/ip combinations through the Asus Merlin OpenVPN Client. I was thinking maybe I could set the script up to forward port 53 (DNS) to my VPN service's DNS server using the script. This is more of a work-around since this would cause ALL dns traffic to go through the VPN instead of just the DNS resolves from the devices using the OpenVPN Client on the router. In most cases that shouldn't cause an issue with clients bypassing the VPN.

TorGuard script and instructions:
https://torguard.net/knowledgebase.php?action=displayarticle&id=165

I was thinking this might be an even better solution since so many ISP's are either spying on their customers' DNS traffic or even setting up a transparent DNS proxy to force their customers to use their DNS servers. This way all DNS traffic would be encrypted and go to a DNS server that you could be confident is not keep or selling its logs!

Transparent DNS Proxy:
https://www.dnsleaktest.com/what-is-transparent-dns-proxy.html

The only other solution I could think of would be to setup an unbound server either on the router or another device on the local network and force DNS traffic to use DNS over TLS. But that is a lot more work. I know some people would recommend DNScrypt but it doesn't offer any real privacy since only a small part of each packet is encryted (just the DNS resolve).

DNS over TLS vs DNScrypt:
https://tenta.com/blog/post/2017/12/dns-over-tls-vs-dnscrypt

Unbound:
https://calomel.org/unbound_dns.html

It's going to be a few days before I can try this since my children are home for the weekend and they HATE when I take down the router for testing!

Anyway I was curious if anyone else had tried this successfully or was willing to try. Thanks!

I'm totally tied up with AB4 coding but I look forward to your conclusions. If you get this working with not too many steps such a post might be a good addition to the AB-Solution FAQ.

 

[Martineau]

I looked at their knowledge base and they have a script to route specific port/ip combinations through the Asus Merlin OpenVPN Client. I was thinking maybe I could set the script up to forward port 53 (DNS) to my VPN service's DNS server using the script.

Apologies to @thelonelycoder for a slight hijack of the thread...

@JJQuin FYI, the script you refer to is intended to open up your LAN server(s) to INBOUND requests via your VPN ISP's Tunnel.

i.e. usually, VPN ISPs ONLY provide an OUTBOUND tunnel and (most) will consequently block ALL unsolicited INBOUND traffic, so you couldn't host say an obfuscated eBAY style shopping Web server etc.

However, there are ('non-logging') public DNS servers that you could define for use by your nominated LAN VPN clients rather than your VPN ISP's or even Google's DNS servers.
e.g.
https://freedns.zone/en/

I even created scripts

openvpnclientX.postconf

to allow which type of DNS I want from the VPN Client GUI

 case $USEDNS in
  opendns)
     #Add_Custom_DNS "isp" "OpenDNS" "208.67.222.222" "208.67.220.220"
     Add_Custom_DNS "isp" "OpenDNS" "$(grep -i opendns /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  dyn)
     #Add_Custom_DNS "isp" "Dyn"
     Add_Custom_DNS "isp" "Dyn" "$(grep -i dyn /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  anycast|opennic)
     #Add_Custom_DNS "isp" "OpenNIC (Anycast Tier 2)"
     Add_Custom_DNS "isp" "OpenNIC" "$(grep -i opennic /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  comodo)
     #Add_Custom_DNS "isp" "Comodo Secure"
     Add_Custom_DNS "isp" "Comodo Secure" "$(grep -i comodo /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  torguard)
     Add_Custom_DNS "isp" "TorGuard" "104.223.91.194" "104.223.91.210"
     ;;
  quad9)
     Add_Custom_DNS "isp" "Quad9" "9.9.9.9" "149.112.112.11"
     ;;
  freedns)
     #Add_Custom_DNS "isp" "FreeDNS"
     Add_Custom_DNS "isp" "FreeDNS" "$(grep -i freedns /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  usrandom)
     Add_Custom_DNS "public" "us" "USA"
     ;;
  *)
     # logger -st "($(basename $0))" $$ "HMA DNS left as pushed from server!"
     #
     ;;
  esac

NOTE: Since v380.69, @RMerlin has modified the search order for the DNSFILTER GUI rules to force individual LAN device to use a specified third-party DNS server except if 'Accept DNS Configuration=EXCLUSIVE'

Good luck with your quest.

 

[JJQuin]

Apologies to @thelonelycoder for a slight hijack of the thread...

@JJQuin FYI, the script you refer to is intended to open up your LAN server(s) to INBOUND requests via your VPN ISP's Tunnel.

i.e. usually, VPN ISPs ONLY provide an OUTBOUND tunnel and (most) will consequently block ALL unsolicited INBOUND traffic, so you couldn't host say an obfuscated eBAY style shopping Web server etc.

However, there are ('non-logging') public DNS servers that you could define for use by your nominated LAN VPN clients rather than your VPN ISP's or even Google's DNS servers.
e.g.
https://freedns.zone/en/

I even created scripts

openvpnclientX.postconf

to allow which type of DNS I want from the VPN Client GUI

 case $USEDNS in
  opendns)
     #Add_Custom_DNS "isp" "OpenDNS" "208.67.222.222" "208.67.220.220"
     Add_Custom_DNS "isp" "OpenDNS" "$(grep -i opendns /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  dyn)
     #Add_Custom_DNS "isp" "Dyn"
     Add_Custom_DNS "isp" "Dyn" "$(grep -i dyn /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  anycast|opennic)
     #Add_Custom_DNS "isp" "OpenNIC (Anycast Tier 2)"
     Add_Custom_DNS "isp" "OpenNIC" "$(grep -i opennic /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  comodo)
     #Add_Custom_DNS "isp" "Comodo Secure"
     Add_Custom_DNS "isp" "Comodo Secure" "$(grep -i comodo /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  torguard)
     Add_Custom_DNS "isp" "TorGuard" "104.223.91.194" "104.223.91.210"
     ;;
  quad9)
     Add_Custom_DNS "isp" "Quad9" "9.9.9.9" "149.112.112.11"
     ;;
  freedns)
     #Add_Custom_DNS "isp" "FreeDNS"
     Add_Custom_DNS "isp" "FreeDNS" "$(grep -i freedns /jffs/configs/publicDNS | cut -d" " -f2)"
     ;;
  usrandom)
     Add_Custom_DNS "public" "us" "USA"
     ;;
  *)
     # logger -st "($(basename $0))" $$ "HMA DNS left as pushed from server!"
     #
     ;;
  esac

NOTE: Since v380.69, @RMerlin has modified the search order for the DNSFILTER GUI rules to force individual LAN device to use a specified third-party DNS server except if 'Accept DNS Configuration=EXCLUSIVE'

Good luck with your quest.

Thanks for the information @Martineau. The issue I have with using public DNS servers, especially for VPN traffic, is that your ISP can still track every DNS request. Your ISP can simply listen for DNS packets on port 53 or worse setup a Transparent DNS Proxy that forces all DNS resolves to go through their server even if you have a different DNS server configured. You can bypass this for your VPN clients by setting Accept DNS Configuration to "Exclusive." Unfortunately, AB-Solution requires the setting to be set for "Strict" which allows DNS traffic to bypass the VPN. I've tried using the DNS Filter feature to force DNS traffic over the VPN iface, but when I do no ads are blocked. I'm going to try testing with that script on Monday, but I'm not sure I'll have much luck. I don't think it will be much trouble to change the script for outbound DNS traffic, but I have a feeling that even if I get it working, AB-Solution will stop blocking ads again. It just seems that no matter what method you use to force your DNS through the VPN it breaks AB-Solutions ad-blocking. Again not much thelonelycoder can do about this right now.

 

[thelonelycoder]

It just seems that no matter what method you use to force your DNS through the VPN it breaks AB-Solutions ad-blocking. Again not much thelonelycoder can do about this right now.

Maybe unbound really is the solution to that. Use Dnsmasq to do the local DHCP and DNS and forward every upstream request to unbound to be resolved.

 

[Jack Yaz]

Maybe unbound really is the solution to that. Use Dnsmasq to do the local DHCP and DNS and forward every upstream request to unbound to be resolved.

Would that then help exclude devices from AB-Solution and still maintain secure DNS resolving (which is the main thing that's kept me from using AB-S with dnscrypt-proxy)?

 

[thelonelycoder]

Would that then help exclude devices from AB-Solution and still maintain secure DNS resolving (which is the main thing that's kept me from using AB-S with dnscrypt-proxy)?

That would have to be tested. But not by me, at least not now. I have my mind set on other coding problems.
I hear a lot of good things about unbound, it would mean to partially truncate Dnsmasqs job on the router but in a good way.

 

[Jack Yaz]

That would have to be tested. But not by me, at least not now. I have my mind set on other coding problems.
I hear a lot of good things about unbound, it would mean to partially truncate Dnsmasqs job on the router but in a good way.

I'll see if I can get my head round Unbound, lots of configuration options. In my head it should work, allowing dnsmasq to resolve in the first instance (therefore allowing AB-S to block with the relevant files?), and set upstream to unbound, which resolves using DNS over TLS.

 

[thelonelycoder]

I'll see if I can get my head round Unbound, lots of configuration options. In my head it should work, allowing dnsmasq to resolve in the first instance (therefore allowing AB-S to block with the relevant files?), and set upstream to unbound, which resolves using DNS over TLS.

This is how it will work if you put all the pieces together.
AB, or better: Dnsmasq will direct all non-local and not-blocked domain queries upstream. It sounds simple but a steep learning curve might hinder instant success.

 

[Xentrk]

For the PBS app to work, I have to whitelist

b.scorecardresearch.com
b.scorecardresearch.com.edgesuite.net
pubads.g.doubleclick.net
partnerad.l.doubleclick.net

The second and forth host names are the CNAME for the host above it. It did a test and found that PBS worked without white listing the CNAME. But I added the CNAMEs on my pfSense per a recommendation from the pfBlockerNG author. I like to be consistent on my white and black lists for theses two routers. So I included the CNAMEs. What is your recommendation?

If I add these entries to dnsmasq.conf.add, I no longer get served the ads and it gives PBS the response it needs. The IP Address listed is the one assigned to pixelserv-tls.

address=/b.scorecardresearch.com/192.168.22.1
address=/b.scorecardresearch.com.edgesuite.net/192.168.22.1
address=/pubads.g.doubleclick.net/192.168.22.1
address=/partnerad.l.doubleclick.net/192.168.22.1

For next steps, I need to debug what in Skynet is causing the graphics for the shows not to display.

Update: These addresses blocked by Skynet that prevented the show graphics from being displayed are.

205.185.216.10
205.185.216.42

Once white listed, I had to power cycle the Roku. The graphics now appear and the problem is resolved.

 

[M@rco]

@thelonelycoder: not sure whether @kvic or @ryzhov_al have already corrected something to bypass the version number scheme, but to my pleasant surprise, when I just installed pixelsrv-tls from within AB, the new version got installed rightaway .

I did a fresh install of the latest @RMerlin 's 384 alpha2 yesterday (installed pixelserv-tls v2.0.1 previously by using 'the dirty solution' as @ryzhov_al suggested on Github prior to performing a clean install and was actually preparing to use the same trick again), but when I just installed it from within AB to my suprise it installed v2.0.1 rightaway, without any modification of /opt/lib/opkg/status needed.

Remark: for anyone reading this, this might only be applicable to new pixelserv-tls installations, I can't say for sure if this will work for upgrading current installs of pixelserv-tls as well, as I don't what has changed.

 

[thelonelycoder]

If I add these entries to dnsmasq.conf.add, I no longer get served the ads and it gives PBS the response it needs. The IP Address listed is the one assigned to pixelserv-tls.

Interesting that the address options solved it for you for those sub-domains.
Generally, these two ways do the same in Dnsmasq:
As a host entry (blacklist, blocking file, as addn-hosts):
192.168.22.1 b.scorecardresearch.com
As a conf setting (trough dnsmasq.conf.add or dnsmasq.postconf):
address=/b.scorecardresearch.com/192.168.22.1

Both direct Dnsmasq to look no further when found and resolve the (sub)domain as directed: To the given IP.
However, and this is important here, the host entry does not allow wildcard entries.
192.168.22.1 *b.scorecardresearch.com will be ignored when the file is read in as the wildcard * is an invalid character.
On the other hand, the address option is by default a wildcard entry.
address=/b.scorecardresearch.com/192.168.22.1 resolves everything that it encounters to the given IP.
foo.b.scorecardresearch.com or bar.b.scorecardresearch.com will end up at the host behind 192.168.22.1

And that may be the reason why adding these to the regular blacklist may not fully work for you but adding it as address will.
AB4 will have that wildcard feature built in, as a separate wildcard-blacklist.

 

[thelonelycoder]

Remark: for anyone reading this, this might only be applicable to new pixelserv-tls installations, I can't say for sure if this will work for upgrading current installs of pixelserv-tls as well, as I don't what has changed.

It's not might, it will work for new installs. opkg keeps a list of locally installed packages. When updating or upgrading it looks into that list and compares the version number locally to the one upstream. With pixelserv-tls Kk installed, v35... looks newer than the v2.0... upstream. Hence the many headaches.
With a new install that local packages file does not contain any previous version of pixelserv. Ergo you get the one upstream which is v2.0.1

 

[DonnyJohnny]

Interesting that the address options solved it for you for those sub-domains.
Generally, these two ways do the same in Dnsmasq:
As a host entry (blacklist, blocking file, as addn-hosts):
192.168.22.1 b.scorecardresearch.com
As a conf setting (trough dnsmasq.conf.add or dnsmasq.postconf):
address=/b.scorecardresearch.com/192.168.22.1

Both direct Dnsmasq to look no further when found and resolve the (sub)domain as directed: To the given IP.
However, and this is important here, the host entry does not allow wildcard entries.
192.168.22.1 *b.scorecardresearch.com will be ignored when the file is read in as the wildcard * is an invalid character.
On the other hand, the address option is by default a wildcard entry.
address=/b.scorecardresearch.com/192.168.22.1 resolves everything that it encounters to the given IP.
foo.b.scorecardresearch.com or bar.b.scorecardresearch.com will end up at the host behind 192.168.22.1

And that may be the reason why adding these to the regular blacklist may not fully work for you but adding it as address will.
AB4 will have that wildcard feature built in, as a separate wildcard-blacklist.

Come on baby.. give it to me baby... I am getting excited....

 

[thelonelycoder]

Come on baby.. give it to me baby... I am getting excited....

Alright, the sun just came up from behind the mountains to my left, looks like a beautiful day and I'm in a good mood for some reason.

I was able to pry this screenshot off of my development router. Note the subtle differences to AB3.
The m option is the former sub menu, it expands the menu to include all options in one screen.
Everything you see works of course, but still no ETA. The WebUI still refuses to be captured.

Note: This is a preview of AB-Solution 4.0 and NOT available to users outside the four walls I live in.