Diversion Diversion - the Router Ad-Blocker

[loady]

now im getting ads again, i installed the crt into the trusted root ca store, as per the wiki.

EDIT: its working fine on my android and thats without reinstalling the regenerated .crt, i rebooted the laptop and its the same, getting ads, both devices showing as connected in the vpn server.

 

[thelonelycoder]

Hi @thelonelycoder,

1. Trying to look into Diversion a bit further, it seems like the description on the following pages is a bit misleading (at least for new Diversion users):
   https://diversion.ch/diversion/installation.html
   
   https://diversion.ch/faq-reader/what-does-pixelserv-tls-do.html
   
   
   From what I can tell, The Lite and The Standard Edition both can block http AND https domains,
   but The Standard Edition with pixelserv-tls can additionally fill the space left by blocked content (only if the pixelserv-tls' certificate is manually imported into clients),
   and, at least according to @kvic's manually performed tests
   https://kazoo.ga/pixelserv-tls-more-is-less/
   https://kazoo.ga/new-features-in-pixelserv-tls-2-1-0/
   the pages with blocked content could load faster with pixelserv-tls.
   
   
2. 
   Do you use the original (MIPS-based) RT-AC66U or RT-AC66U_B1? What edition of Diversion do you use/recommend for such MIPS-based routers?
   
   According to your comment
   https://github.com/kvic-z/pixelserv-tls/issues/18#issuecomment-456516745
   the last pixelserv-tls version for MIPS-based routers is v2.0.1, and the URL http://<pixelserv ip>/ca.crt is not working,
   so have you found the way to import this pixelserv-tls cert?
   Do you use Diversion Standard without this cert or Diversion Lite?
   
   
3. Looking through the logs by selecting f follow dnsmasq.log
   and then 2. Unfiltered log extra highlighted (or 3. Filtered by blocked domains)
   I've noticed a small side-effect:
   the log line containing "blockinglist" or "blacklist" will be highlighted as blocked (in red),
   even if the corresponing URL wasn't actually in the blocking list:
   try visiting, for instance, blacklist.reddit.com
   
   Looking through
   

   /opt/share/diversion/file/functions.div

   it seems to happen because the functions highlighted() and blocked_domains() contain the same line
   

   if echo "$line" | grep -q "blockinglist\|blacklist\| config .* is $blockingIP"; then

   that just checks if the log file line contains specific keywords.

Noted, thanks.

 

[thelonelycoder]

Why incoming CNAME filtering is now, and will become more of a necessity over time:

uBlock Origin for Firefox addresses new first-party tracking method
gHacks Technology News / Martin Brinkmann

The latest version of the content blocker uBlock Origin for the Mozilla Firefox web browser includes a new feature to detect a new first-party tracking method that some sites have started to use recently.

The issue was first reported ten days ago by user Aeris on the project's official GitHub page. Some sites started to use canonical name records (CNAMEs) to bypass filters used in content blockers. First-party resources, e.g. a subdomain, are not blocked usuall unless they are known to only serve advertisement.

The main issue from a content blocking perspective is that identification and detection is difficult. The extensions would have to uncloak alias hostnames in order to provide the user with information and the ability to do something about it.

Raymond Hill, the developer of uBlock Origin, found a way to address the new first-party tracking method in Mozilla Firefox.

Side-note: Why only Firefox? Because Mozilla has created DNS APIs that may be used to expose the CNAME while Google has not. For now, it is not possible to protect against this form of tracking in Google Chrome. Hill writes "Best to assume it can't be fixed on Chromium if it does not support the proper API".

Firefox users who upgrade to the latest version of uBlock Origin, may notice a new permission request (Access IP address and hostname information). This is required to unlock access to the DNS API in the browser extension.

Firefox users who run the extension need to do the following to set things up properly on their end:

1. Open the Settings of the extension, e.g. from about:addons or by clicking on the dashboard icon in the uBlock Origin interface.
2. Check the "I am an advanced user" box on the first page that opens.
3. Activate the settings icon next to the option to open the advanced settings.
4. Change the value of the parameter cnameAliasList to *.

The change runs the actual hostnames through the filtering that uBlock Origin applies again. The log highlights these in blue.

Network requests for which the actual hostname differs from the original hostname will be replayed through uBO's filtering engine using the actual hostname. [..] Regardless, uBO is now equipped to deal with 3rd-party disguised as 1st-party as far as Firefox's browser.dns allows it.

The setting of the wildcard means that the process is done for any hostname that differs; this works but it means that a certain number of network requests are processed twice by uBlock Origin.

The next step is for me to pick a cogent way for filter list maintainers to be able to tell uBO to uncloak specific hostnames, as doing this by default for all hostnames is not a good idea -- as this could cause a huge amount of network requests to be evaluated twice with no benefit for basic users (default settings/lists) while having to incur a pointless overhead -- for example when it concerned CDNs which are often aliased to the site using them.

Hill wants to switch to using a maintained list of known offenders that uBlock Origin (UMatrix will support this as well) will process while leaving any other hostname untouched.

Closing Words
Firefox users may change the configuration to make sure that they are protected against this new form of tracking. Chromium users cannot because the browser's APIs for extensions does not have the capabilities at the time of writing.

Thank you for being a Ghacks reader. The post uBlock Origin for Firefox addresses new first-party tracking method appeared first on gHacks Technology News.



Original Article: https://www.ghacks.net/2019/11/20/u...ox-addresses-new-first-party-tracking-method/

I read a similar discussion too, I hope the Dnsmasq developer Simon Kelley will add a switch or something to disable/restrict this. In the meantime, add the hosts file posted by @Twiglets below your original post to the blocking list(s) in Diversion.

 

[thelonelycoder]

This may be of interest to others.

It is, I'm using it. Thanks.

 

[thelonelycoder]

I think i have noticed a speed bump in diversion's DNSMASQ setting section....
View attachment 20036
it mainly concerns bogus-priv and domain-needed functions
When it gets enabled it allows for it to be enabled twice in DNSMASQ.conf, note mine is automatically on in DNSMASQ, but when I use diversion to enable it, it allows for it to make two entries inside DNSMASQ.conf
View attachment 20037
the first place they appear listed is under the expanded-hosts option inside dnsmasq.
View attachment 20038
the second place it is listed is inside the diversion directives under add-hosts list....

Would this cause a problem for DNSMASQ if it is listed twice?
I can see that if you select No inside Diversion, that it should remove those entries from dnsmasq, but I don't think selected Yes should allow the entries to appear twice in the .conf file.

As mentioned by other users, these double entries have no impact on Dnsmasq. It simply sets the last read directive.
I'm not sure this requires changes in Diversion.

 

[thelonelycoder]

Could Diversion possibly knock out the WAN? I get intermittent WAN dropouts every few days or so which rebooting the router fixes. I notice a WiFi symbol on my phone (Samsung S10) with an exclamation mark when it happens.

So I looked in my traffic graph and noticed a big download died around 2am, and looked in to the system logs and could only see Diversion updating stuff.. Is there any way this could be linked? Seems odd but never know.

I have DNS set to Cloudflare too, so could be something funky anywhere. But at least I now know the dcp related kernel errors in the logs are hardware related so that's an issue

Nov 29 02:00:00 Central uiDivStats: Starting Diversion statistic generation...
Nov 29 02:00:01 Central kernel: echo (26625): drop_caches: 3
Nov 29 02:01:50 Central uiDivStats: Diversion statistic generation completed successfully!
Nov 29 02:03:52 Central Diversion: pgl.yoyo.org is not hosts file, keeping blocking list, from /opt/share/diversion/file/update-bl.div
Nov 29 02:17:08 Central Diversion: https://hosts-file.net/emd.txt not hosts file, using backup file, from /opt/share/diversion/file/update-bl.div
Nov 29 02:21:05 Central Diversion: https://hosts-file.net/exp.txt not hosts file, using backup file, from /opt/share/diversion/file/update-bl.div
Nov 29 02:24:56 Central Diversion: https://hosts-file.net/hjk.txt not hosts file, using backup file, from /opt/share/diversion/file/update-bl.div
Nov 29 02:28:47 Central Diversion: https://hosts-file.net/mmt.txt not hosts file, using backup file, from /opt/share/diversion/file/update-bl.div
Nov 29 02:32:38 Central Diversion: https://hosts-file.net/psh.txt not hosts file, using backup file, from /opt/share/diversion/file/update-bl.div
Nov 29 02:36:33 Central Diversion: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts not hosts file, using backup file, from /opt/share/diversion/file/update-bl.div
ov 29 02:36:54 Central Diversion: updated Standard+ blocking list from 6 hosts files, 1071238 domains are now blocked, from /opt/share/diversion/file/update-bl.div
Nov 29 02:36:59 Central Diversion: updated and reset weekly ads counter: 92,989 total, 21,558 this week, 59 new since last count, from /opt/bin/diversion
Nov 29 02:36:59 Central Diversion: reset dnsmasq log files (weekly cron job), from /opt/share/diversion/file/update-bl.div

Nothing in your posted log shows why the WAN is down. The Diversion error messages simply state that the files it tries to download contain not hosts entries. Which happens when the server returns an error message or is not available at the time of downloading.

 

[thelonelycoder]

Hi Lonely coder
When I try and install Diversion from amtm 3.0 I get
Diversion installation failed.

I updated firmware on AC5300 to latest beta and ended up with diversion option missing in amtm. When I try and reinstall i get Diversion installation failed message above. Any ideas?

It'll say much more than that in the terminal as Diversion tries to install, post these errors by scrolling up the terminal after.

 

[thelonelycoder]

i tried to install the newer pixel crt.....

 This updates installed Entware packages.

 Entware version: Entware (armv7sf-k3.2)
 Installed from: bin.entware.net

 1. show pixelserv-tls info
 2. show installed packages
 3. update or upgrade pixelserv-tls
 4. update list of available packages
 5. update and upgrade installed packages

 Enter selection [1-5 e=Exit] 3
____________________________________________________

 This updates or upgrades pixelserv-tls v2.2.1

 1. Update pixelserv-tls, regular Entware version
 2. Upgrade pixelserv-tls, regular Entware version
 3. Upgrade pixelserv-tls to v2.3.0, Jack Yaz version

 Enter your selection [1-3 e=Exit] 3
____________________________________________________

 This upgrades pixelserv-tls to v2.3.0, Jack Yaz version.
 This version is compliant with the new required
 security settings enforced by Apple and other Companies.
 See https://github.com/jackyaz/pixelserv-tls/releases/tag/2.3.0

 It will install the  appropriate version for your
 Entware (armv7sf-k3.2) installation.

 After successful upgrade, purge and re-generate the
 CA certificate in  ep , 3, 2.

 Continue? [1=Yes e=Exit] 1
____________________________________________________

  i  Downloading pixelserv-tls

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   629    0   629    0     0   2428      0 --:--:-- --:--:-- --:--:--  2526
100 26936  100 26936    0     0  34139      0 --:--:-- --:--:-- --:--:-- 34139

  i  Download successful, installing...

Package pixelserv-tls (2.2.1-1) installed in root is up to date.

 Diversion 4.1.6                  by thelonelycoder

 RT-AC86U (aarch64) FW-384.13 @ 192.168.11.1

 1.213M  blocked domains by  1  hosts file(s)
 649 t  649 w  649 n ads since Nov 29 17:20
____________________________________________________

 d   Diversion Standard   enabled
 c   communication        DivUn stats backup FWun

 a   ad-blocking          to IP 192.168.11.2
 l   logging              dnsmasq.log 32.0K

 ep  pixelserv-tls        192.168.11.2 v2.2.1

 b   blocking list        Large Thu @ 2:00
 el  edit lists            0 w  0 b  0 wb

 f   follow dnsmasq.log

 e   exit Diversion                 more options  o
____________________________________________________

 Done  Failed to update pixelserv-tls

 What do you want to do?

Try updating the Entware packages in ep first.

 

[Davidncali001]

I have done that, works fine on my android but not on the laptop, well it works, ads are blocked but nod 32 keeps hammering me with those warnings

I use the Diversion Script, AMTM, UIStats and Skynet with a 1GB Swap File.

I also use Eset Nod32. Inside Nod32 settings did you remember to not use the certificate that comes with the Eset Nod32 Security?

https://www.snbforums.com/threads/ab-solution-webpage-antivirus-problems.46817/

I disabled it in the settings of Nod32.

EDIT: also make sure the pixelserv certificate you are importing is placed in the Local Machine Computer Trust Root Certificate Authoritiers Store .

 

[loady]

I use the Diversion Script, AMTM, UIStats and Skynet with a 1GB Swap File.

I also use Eset Nod32. Inside Nod32 settings did you remember to not use the certificate that comes with the Eset Nod32 Security?

No, I have had nod32 installed as is from way before I had this router

https://www.snbforums.com/threads/ab-solution-webpage-antivirus-problems.46817/

I disabled it in the settings of Nod32.

EDIT: also make sure the pixelserv certificate you are importing is placed in the Local Machine Computer Trust Root Certificate Authoritiers Store .

 

[Twiglets]

I read a similar discussion too, I hope the Dnsmasq developer Simon Kelley will add a switch or something to disable/restrict this. In the meantime, add the hosts file posted by @Twiglets below your original post to the blocking list(s) in Diversion.

DNSCrypt-proxy Beta release 2.0.34-beta.1 supports:
'Blacklisted names are now also blocked if they appear in CNAME pointers.'
'DNSCrypt-proxy can now act as a local DoH server. Firefox can be configured to use it, so that ESNI can be enabled without bypassing your DNS proxy.'

As advised by @zastoff in https://www.snbforums.com/threads/release-dnscrypt-installer-for-asuswrt.36071/page-70#post-528987

Follow instructions in latest example-dnscrypt-proxy.toml to configure 'Local DoH server' then assuming you are using 'https://127.0.0.1:3000/dns-query'* as your server url do the following:

• Use Firefox to browse the following URL: https://127.0.0.1:3000/dns-query *
• Then click Advanced and I accept the risk (there are no risks, you are only connecting to your own machine)
• Then, open about:config
  
  
• Set network.trr.custom_uri and network.trr.uri to https://127.0.0.1:3000/dns-query*
  
  
• Set network.trr.mode to 2
  
  
• Set network.security.esni.enabled to true
  
  
• Restart Firefox

* Default in example file.
Substitute the correct URL for your configuration ..... don't forget the '/dns-query' on the end !!!!

(Instructions 'borrowed' from https://www.reddit.com/r/dnscrypt/comments/dy405m/enabling_esni_with_dnscryptproxy/

 

[Adamm]

Dnsmasq-proxy Beta release 2.0.34-beta.1 supports:

You mean DNSCrypt-proxy

 

[Twiglets]

You mean DNSCrypt-proxy

Correct .... cannot type & think at the same time !!!!

Fixed: Thank you.
Muscle-memory is a wonderful thing .... I had just typed 'service restart_dnsmasq' and even though I thought DNSCrypt typed dnsmasq !!!???

 

[Treadler]

It is, I'm using it. Thanks.

As indeed am I.

 

[loady]

Right, sorted the ESET notifications, disabled TLS as advised, hope i have not turned off anything important !.

Still i cant get this win10 laptop running with diversion, ive import pixel.crt so many times now, works fine on my win10 desktop and android device, any idea ?

Also, my router running diversion sits behind my ISP router, anything connected to its wifi or ethernet (isp router) doesnt get use of diversion, how do i sort this ?, the subnets are different, the isp router is 192.168.1.1 and my router is 192.168.11.1, isp router has issued my router with a WAN ip of 192.168.1.22

 

[Adamm]

Right, sorted the ESET notifications, disabled TLS as advised, hope i have not turned off anything important !.

Still i cant get this win10 laptop running with diversion, ive import pixel.crt so many times now, works fine on my win10 desktop and android device, any idea ?

Also, my router running diversion sits behind my ISP router, anything connected to its wifi or ethernet (isp router) doesnt get use of diversion, how do i sort this ?, the subnets are different, the isp router is 192.168.1.1 and my router is 192.168.11.1, isp router has issued my router with a WAN ip of 192.168.1.22

You are running a double NAT situation, you need to put your ISP modem into bridge mode.

 

[loady]

You are running a double NAT situation, you need to put your ISP modem into bridge mode.

Dang, not the bridge mode conundrum again , dont think this can be put in bridge mode, also my setup requires that my isp modem is downstairs and my router is upstairs connected by power lines to its wan port

 

[Adamm]

Dang, not the bridge mode conundrum again , dont think this can be put in bridge mode, also my setup requires that my isp modem is downstairs and my router is upstairs connected by power lines to its wan port

Your situation can work, but its not ideal by a long shot. Whenever you connect directly to your ISP modem you won't get access to any custom scripts like Diversion, only clients serviced by the Asus router will.

 

[Jumpstarter]

Nothing in your posted log shows why the WAN is down. The Diversion error messages simply state that the files it tries to download contain not hosts entries. Which happens when the server returns an error message or is not available at the time of downloading.

I have run into a problem, dnsmasq log-async has taken over my CPU processing power, it only happens when diversion is running. do you have any tips on how to troubleshoot this problem to pinpoint the cause?

 

[loady]

Your situation can work, but its not ideal by a long shot. Whenever you connect directly to your ISP modem you won't get access to any custom scripts like Diversion, only clients serviced by the Asus router will.

Im just looking into m,

Your situation can work, but its not ideal by a long shot. Whenever you connect directly to your ISP modem you won't get access to any custom scripts like Diversion, only clients serviced by the Asus router will.

I am looking into my isp forum, there may be an un-official-ish way to get it into a bridge mode, if so, can i still have the asus upstairs connected by power lines?..i also have a GB switch, could i hang the asus off the isp router and leave the asus downstairs next to it and then use the power lines to put the switch upstairs, i only need the switch for the lan ports