Diversion Diversion - the Router Ad-Blocker

[pattiri]

4 of 13 hostsfiles? How big is your blocking file?

lol. it's 32 MB; 990,466 blocked domains 13 hosts file(s) in use

 

[JemTheWire]

Can you help please?

Running Diversion v4.0.2 on my AC88U, Merlin 384.7

I have one app that fails to work with Diversion enabled. If I disable Diversion, the app works.

The app in question is the BBC Sport app in the UK.

Diversion appears to be blocking something, but how can I find out what so I can add it to the White list?

Thanks in advance.

 

[truglodite]

SSH in and...

diversion
f
1
[Enter]

...open the app and make note of related urls in red. Add those to Diversion's whitelist, process whitelist, enjoy. A tip... not necessarily all of the red lines that pop up need to be whitelisted. Try to minimize white listing to only the urls needed for the app to function properly. I accomplish this by adding urls one at a time, and testing functionality. If nothing changes when I whitelist a url, I take it off and try another. Sometimes it is a game like whack a mole... getting only the needed sites whitelisted without allowing unneeded ads through.

 

[JemTheWire]

Thank you. After some trial and error, I found it was:

ping.chartbeat.net

That was causing the issue. Added to the Whitelist and now all Ok.

Thanks again.

 

[Protik]

I nuked my router today and reinstalling everything. For some reason I cannot turn on stats for diversion. I tried to get into the logging feature by pressing l but it says the following,

Any help on how can I enable the stats?

Thanks.

 

[thelonelycoder]

I nuked my router today and reinstalling everything. For some reason I cannot turn on stats for diversion. I tried to get into the logging feature by pressing l but it says the following,

Any help on how can I enable the stats?

Thanks.

Can you post the content of the config file? Use sf.

 

[Protik]

Can you post the content of the config file? Use sf.

 /opt/share/diversion/.conf/diversion.conf has this content:

 START FILE, --- lines are not part of file
 ---------------------------------------------------
 ## DO NOT EDIT THIS FILE ##
 
 # General settings #
 NAME=Diversion
 thisVERSION=4.0
 thisM_VERSION=2
 EDITION=Standard
 THEME=blue_on_white
 INSTALL_URL=https://diversion.ch/diversion
 DIVERSION_STATUS=enabled
 adblocking=on
 logging=
 editorColor=
 editorPaginate=
 shHistory=on
 
 # Communication settings #
 divUpdate=on
 weeklyStats=on
 wsTo="email"
 wsAs="body"
 wsDo="delete"
 wsTopHosts=10
 wsTopClients=10
 wsFilterLN=off
 backup=on
 buFrequency=weekly
 buType="Key-files"
 buAs=compressed
 buCompression=tar.gz
 
 # Blocking file settings #
 bfType=Standard
 bfPlusHosts=on
 bfUpdateDay=Sunday
 bfUpdateDOW=Sun
 bfUpdateHour=2
 bfUpdateLastRun="Oct 07 21:31:36"
 bfUpdatePrevRun=""
 blockingIP="192.168.2.2"
 
 # Entware settings #
 entPath="/mnt/pdas001/entware"
 entVersion="Entware (armv7sf-k2.6)"
 psState=on
 prevPsState=on
 psIP="192.168.2.2"
 
 # Ad-blocking counters #
 blockedDomains=840680
 adsBlocked=0
 adsWeek=0
 adsNew=0
 adsPrevCount=""
 adsLastCount=""
 
 # Dnsmasq settings #
 dsSetting=on
 logAsync=
 cacheSize=
 bogusPriv=
 domainNeeded=
 loqQueriesExtra=
 
 # Messaging #
 
 ---------------------------------------------------
 END FILE

 

[thelonelycoder]

 /opt/share/diversion/.conf/diversion.conf has this content:

 START FILE, --- lines are not part of file
 ---------------------------------------------------
 ## DO NOT EDIT THIS FILE ##
 
 # General settings #
 NAME=Diversion
 thisVERSION=4.0
 thisM_VERSION=2
 EDITION=Standard
 THEME=blue_on_white
 INSTALL_URL=https://diversion.ch/diversion
 DIVERSION_STATUS=enabled
 adblocking=on
 logging=
 editorColor=
 editorPaginate=
 shHistory=on
 
 # Communication settings #
 divUpdate=on
 weeklyStats=on
 wsTo="email"
 wsAs="body"
 wsDo="delete"
 wsTopHosts=10
 wsTopClients=10
 wsFilterLN=off
 backup=on
 buFrequency=weekly
 buType="Key-files"
 buAs=compressed
 buCompression=tar.gz
 
 # Blocking file settings #
 bfType=Standard
 bfPlusHosts=on
 bfUpdateDay=Sunday
 bfUpdateDOW=Sun
 bfUpdateHour=2
 bfUpdateLastRun="Oct 07 21:31:36"
 bfUpdatePrevRun=""
 blockingIP="192.168.2.2"
 
 # Entware settings #
 entPath="/mnt/pdas001/entware"
 entVersion="Entware (armv7sf-k2.6)"
 psState=on
 prevPsState=on
 psIP="192.168.2.2"
 
 # Ad-blocking counters #
 blockedDomains=840680
 adsBlocked=0
 adsWeek=0
 adsNew=0
 adsPrevCount=""
 adsLastCount=""
 
 # Dnsmasq settings #
 dsSetting=on
 logAsync=
 cacheSize=
 bogusPriv=
 domainNeeded=
 loqQueriesExtra=
 
 # Messaging #
 
 ---------------------------------------------------
 END FILE

I have uploaded a hotfix for your case. Use 33 to download the main script and then run l again.

 

[Protik]

I have uploaded a hotfix for your case. Use 33 to download the main script and then run l again.

Fixed. Thanks!!!

 

[thelonelycoder]

Fixed. Thanks!!!

I wonder how logging got that state logging=
I made changes to the logic when switching state of adblocking or logging.
Will be part of the next Diversion update.

 

[Protik]

I wonder how logging got that state logging=
I made changes to the logic when switching state of adblocking or logging.
Will be part of the next Diversion update.

The router was wiped out. So I set all the settings from scratch.

As the flash drive was still attached, tried running diversion install script. The install script tried to merge with earlier file, I guess. Somewhere the settings got mixed up.

 

[thelonelycoder]

The router was wiped out. So I set all the settings from scratch.

As the flash drive was still attached, tried running diversion install script. The install script tried to merge with earlier file, I guess. Somewhere the settings got mixed up.

If you wiped the router and with it /jffs/* and did not uninstall Diversion first and the drive is still attached, then:
The function restore should have kicked in:

 [ ! ] Found a previous Diversion Standard installation
       at /tmp/mnt/sda1/entware

 1. Restore previous installation, keeping settings
    (recommended)
 2. New installation, deleting previous installation

 Enter your selection [1=Restore 2=New]

Did you see that prompt?

 

[Protik]

If you wiped the router and with it /jffs/* and did not uninstall Diversion first and the drive is still attached, then:
The function restore should have kicked in:

 [ ! ] Found a previous Diversion Standard installation
       at /tmp/mnt/sda1/entware

 1. Restore previous installation, keeping settings
    (recommended)
 2. New installation, deleting previous installation

 Enter your selection [1=Restore 2=New]

Did you see that prompt?

Yes, I did.

I don't know if it's related. I enabled AiCloud and that messed up with my pixelserv IP. So diversion could not start pixelserv at all as that particular IP was being used by AiCloud. Later on after I disabled AiCloud which fixed the pixelserv issue. But the logging issue remained.

 

[thelonelycoder]

Yes, I did.

I don't know if it's related. I enabled AiCloud and that messed up with my pixelserv IP. So diversion could not start pixelserv at all as that particular IP was being used by AiCloud. Later on after I disabled AiCloud which fixed the pixelserv issue. But the logging issue remained.

OK, could you post the Diversion install and update logs, also found in sf.

 

[Protik]

OK, could you post the Diversion install and update logs, also found in sf.

Install log [part 1]

 /opt/share/diversion/log/Diversion-install.log has this content:

 START FILE, --- lines are not part of file
 ---------------------------------------------------
 Diversion install log Sun, 26 Aug 2018 04:53:04 -0700
 ----------------------------------------------------
  [ i ] Checking router
  [ ? ] Asuswrt-Merlin
  [ ? ] dos2unix
  [ ? ] Netstat
  [ ? ] Compatible device(s)
  [ ? ] Internet
  [ i ] Router check complete
 
  [ ? ] All clear to install Diversion 4.0
 
 
  [ i ] But first, let's add some color!
  [ ? ] theme.div           integrated
 ____________________________________________________
 
  [ ? ] Select a theme that works best in your
        SSH client. All colors in use are shown.
 
  1. standard       standard       ?   ?  Done
     standard       standard       i   ?  Error
     standard       standard      standard
  _______________________________________________
 
  2. green          green          ?   ?  Done
     green          green          i   ?  Error
     green          green         green
  _______________________________________________
 
  3. blue           blue           ?   ?  Done
     blue           blue           i   ?  Error
     blue           blue          blue
  _______________________________________________
 
  4. blue_on_white  blue_on_white  ?   ?  Done
     blue_on_white  blue_on_white  i   ?  Error
     blue_on_white  blue_on_white blue_on_white
  _______________________________________________
 
  5. high_contrast  high_contrast  ?   ?  Done
     high_contrast  high_contrast  i   ?  Error
     high_contrast  high_contrast high_contrast
  _______________________________________________
 
  6. basic          basic   [ ? ] [ ? ] [ Done ]
     basic          basic   [ i ] [ ? ] [ Error ]
  This basic theme may not show actual services
  states due to the lack of colors.
  _______________________________________________
 
  Select Diversion theme: [1-6]
 
 ____________________________________________________
 
   ?  Select Diversion Edition to install
 
  1. Diversion Lite        SSH UI, Entware,
                           no pixelserv-tls
 
  2. Diversion Standard    SSH UI, Entware,
                           with pixelserv-tls
 
  Select Edition [1-2 e=Exit]
   i  Checking port 443 availability
   ?  Port 443 is available
 
 ____________________________________________________
 
   i  Reserving one IP address for pixelserv-tls
 
   !  DO THIS NOW: Log into this routers WebUI
 
  - Go to LAN > DHCP Server
  - Adjust "IP Pool Starting Address" from the
    standard xxx.xxx.xxx.2 (typically 192.168.1.2)
    to xxx.xxx.xxx.3 (e.g 192.168.1.3)
    (xxx.2 would be the reserved address)
  - Make sure the reserved address is not used
    by your other devices
  - Then click "Apply"
 
  You will need to enter the IP address next
 
   !  Press [Enter] to continue when ready
 ____________________________________________________
 
  Enter pixelserv-tls IP Address:
   ?  192.168.2.2 is a valid IP address
   i  checking if 192.168.2.2 is available to use
 
   ?  pixelserv-tls IP IP address set to 192.168.2.2
 ____________________________________________________
 
   ?  Select Logging state
 
  Dnsmasq can create a log file of all DNS queries.
  The log is helpful for the curious, will show
  stats in the header and allow the weekly stats
  report to be run.
  Logging can be enabled or disabled anytime.
 
  Enable logging now? [1=Yes 2=No] ____________________________________________________
 
   i  Supporting SmallNetBuilder.com
 
  To support SmallNetBuilder.com, some domains are
  added to the whitelist to allow ads on their
  websites.
 
  Please do not remove them. Thanks
 
   !  Press [Enter] to acknowledge
   i  Probing for Entware
   i  Entware not installed, running checks
   !  Corrected invalid Entware symlink
   ?  Entware checks passed
 ____________________________________________________
 
   ?  Select device to install Entware to
 
  1. /tmp/mnt/pdas001
  2. /tmp/mnt/usb2
 
  Select device [1-2]
   i  Running device checks
   ?  Device checks passed
 ____________________________________________________
 
   ?  Diversion is now ready to install,
      this process runs fully automatic.
 
   !  Press [Enter] to continue
   i  Creating install directory at /tmp/mnt/pdas001/entware
   i  Installing Entware (armv7), using external script
 
 Info: Checking for prerequisites and creating folders...
 Warning: Folder /opt exists!
 Info: Opkg package manager deployment...
 Info: Basic packages installation...
 Downloading http://bin.entware.net/armv7sf-k2.6/Packages.gz
 Updated list of available packages in /opt/var/opkg-lists/entware
 Installing entware-opt (227000-3) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/entware-opt_227000-3_all.ipk
 Installing libc (2.23-8) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/libc_2.23-8_armv7-2.6.ipk
 Installing libgcc (7.3.0-8) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/libgcc_7.3.0-8_armv7-2.6.ipk
 Installing libssp (7.3.0-8) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/libssp_7.3.0-8_armv7-2.6.ipk
 Installing librt (2.23-8) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/librt_2.23-8_armv7-2.6.ipk
 Installing libpthread (2.23-8) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/libpthread_2.23-8_armv7-2.6.ipk
 Installing libstdcpp (7.3.0-8) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/libstdcpp_7.3.0-8_armv7-2.6.ipk
 Installing entware-release (1.0-2) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/entware-release_1.0-2_all.ipk
 Installing zoneinfo-asia (2018e-1) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/zoneinfo-asia_2018e-1_armv7-2.6.ipk
 Installing zoneinfo-europe (2018e-1) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/zoneinfo-europe_2018e-1_armv7-2.6.ipk
 Installing findutils (4.6.0-1) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/findutils_4.6.0-1_armv7-2.6.ipk
 Installing terminfo (6.1-1) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/terminfo_6.1-1_armv7-2.6.ipk
 Installing locales (2.23-8) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/locales_2.23-8_armv7-2.6.ipk
 Installing grep (2.26-1) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/grep_2.26-1_armv7-2.6.ipk
 Installing libpcre (8.41-2) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/libpcre_8.41-2_armv7-2.6.ipk
 Installing opkg (2011-04-08-9c97d5ec-17b) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/opkg_2011-04-08-9c97d5ec-17b_armv7-2.6.ipk
 Installing entware-upgrade (1.0-1) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/entware-upgrade_1.0-1_all.ipk
 Entware uses separate locale-archive file independent from main system
 Creating locale archive - /opt/usr/lib/locale/locale-archive
 Adding en_EN.UTF-8
 Adding ru_RU.UTF-8
 You can download locale sources from http://pkg.entware.net/sources/i18n_glib223.tar.gz
 You can add new locales to Entware using /opt/bin/localedef.new
 Upgrade operations are not required
 Configuring libgcc.
 Configuring libc.
 Configuring libssp.
 Configuring libpthread.
 Configuring librt.
 Configuring terminfo.
 Configuring libpcre.
 Configuring grep.
 Configuring locales.
 Configuring entware-upgrade.
 Configuring opkg.
 Configuring zoneinfo-europe.
 Configuring zoneinfo-asia.
 Configuring libstdcpp.
 Configuring entware-release.
 Configuring findutils.
 Configuring entware-opt.
 Info: Congratulations!
 Info: If there are no errors above then Entware was successfully initialized.
 Info: Add /opt/bin & /opt/sbin to your PATH variable
 Info: Add '/opt/etc/init.d/rc.unslung start' to startup script for Entware services to start
 Info: Found a Bug? Please report at https://github.com/Entware/Entware/issues
 
   ?  Entware (armv7sf-k2.6) install complete
 
   ?  Diversion directory structure created
      at  /opt/share/diversion
 
   i  Getting Diversion Standard files from
      https://diversion.ch/diversion/4.0
   ?  functions.div       integrated
   ?  install.div         integrated
   ?  post-conf.div       integrated
   ?  post-mount.div      integrated
   ?  rc.func.div         integrated
   ?  rotate-logs.div     integrated
   ?  S80pixelserv-tls    integrated
   ?  theme.div           integrated
   ?  update-bf.div       integrated
   ?  write-config.div    integrated

 

[Protik]

OK, could you post the Diversion install and update logs, also found in sf.

install log [part 2],

   i  Checking white and blacklists
   ?  Whitelist created
   ?  Blacklist created
   ?  Wildcard blacklist created
 
   i  Probing for pixelserv-tls
   i  Installing pixelserv-tls
 
 Downloading http://bin.entware.net/armv7sf-k2.6/Packages.gz
 Updated list of available packages in /opt/var/opkg-lists/entware
 Installing pixelserv-tls (2.1.1-1) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/pixelserv-tls_2.1.1-1_armv7-2.6.ipk
 Installing libopenssl (1.0.2o-1) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/libopenssl_1.0.2o-1_armv7-2.6.ipk
 Installing zlib (1.2.11-2) to root...
 Downloading http://bin.entware.net/armv7sf-k2.6/zlib_1.2.11-2_armv7-2.6.ipk
 Place ca.crt and ca.key to /opt/var/cache/pixelserv before starting HTTPS mode.
 Start options can be adjusted via /opt/etc/init.d/S80pixelserv-tls.
 
 See 'pixelserv-tls -h' for details
 Configuring zlib.
 Configuring libopenssl.
 Configuring pixelserv-tls.
 
   ?  pixelserv-tls installed
 
   i  Creating pixelserv-tls certificates
 
 
   ?  pixelserv-tls certificates created
 
   i  Checking /jffs/scripts entries
   ?  dnsmasq.postconf entry added
   ?  post-mount entry added
   ?  services-stop entry added
 
   i  Initializing Diversion
 
   i  Running update-bf.div to create the blocking file
 
 
  preparing temporary whitelist
  updated /jffs/shared-Diversion-whitelist
  refreshing Skynet to whitelist domains in shared-Diversion-whitelist
  including /jffs/shared-Skynet-whitelist in Diversion
  including /jffs/shared-Skynet2-whitelist in Diversion
 
  downloading Standard blocking file, 1 hosts files
  using 192.168.2.2 as blocking IP, adding "::" as IPv6 blocking IP
 
  getting hosts file 1 of 1, preprocessing file
  https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  processing file 1 done
 
  removing whitelisted content, sorting file
  this may take a while...
  adding IPv6 blocking entries
 
  blocking file Standard update complete
 
   i  (Re)starting pixelserv-tls
 
  Starting pixelserv-tls (Diversion)... done
 
   ?  pixelserv-tls is running
 
   i  Checking cron jobs
   ?  Blocking file update
   ?  Rotate logs
   ?  Ads counter
 
   i  Checking dnsmasq.conf entries
   ?  Additional hosts
   ?  Log facility
 
   ?  Diversion Standard v4.0 install complete
 
  Diversion install exited or aborted
 ---------------------------------------------------
 END FILE

 

[Protik]

Update log,

 /opt/share/diversion/log/Diversion-update.log has this content:

 START FILE, --- lines are not part of file
 ---------------------------------------------------
 Diversion update log Sun, 07 Oct 2018 21:27:48 -0700
 ____________________________________________________
 
 
   i  Probing for Entware
   ✔  Entware is already installed
 
   i  Getting Diversion Standard files from
      https://diversion.ch/diversion/4.0
   ✔  backup.div          integrated
   ✔  functions.div       integrated
   ✔  post-conf.div       integrated
   ✔  post-mount.div      integrated
   ✔  rc.func.div         integrated
   ✔  rotate-logs.div     integrated
   ✔  stats.div           integrated
   ✔  S80pixelserv-tls    integrated
   ✔  theme.div           integrated
   ✔  update.div          integrated
   ✔  update-bf.div       integrated
   ✔  write-config.div    integrated
 
   i  Checking /jffs/scripts entries
   ✔  dnsmasq.postconf
   ✔  post-mount
   ✔  services-stop
 
   i  Initializing Diversion
 
   ✔  Blocking file
   i  (Re)starting pixelserv-tls
 
  Starting pixelserv-tls (Diversion)... done
 
 
   ✖  pixelserv-tls not running, restarting...
 
  Starting pixelserv-tls (Diversion)... done
 
 
   ✖  pixelserv-tls not running, check syslog for errors
 
   !  Press [Enter] to acknowledge
 
   i  Checking cron jobs
   ✔  Blocking file update
   ✔  Backup
   ✔  Diversion update check
 
   i  Checking dnsmasq.conf entries
   ✔  Additional hosts
 
   ✔  Diversion Standard v4.0.2 update complete
 ---------------------------------------------------
 END FILE

 

[thelonelycoder]

Diversion install exited or aborted

I believe this is where it all happened, you pressed CTRL-C at some point during the install.

 

[Protik]

I believe this is where it all happened, you pressed CTRL-C at some point during the install.

I don't remember doing that, but that may be the reason. I vaguely remember doing this once yesterday.

Thanks again for looking into it!

 

[Spydawg]

after I install Diversion, i get this in logs every 10 minutes.

Oct 8 16:47:32 lldpd[438]: removal request for address of 142.113.36.57%7, but no knowledge of it

Running later firmware 384.7 on a RT-AC3100