Diversion Diversion - the Router Ad-Blocker

[kernol]

Apologies in advance ... I'm sure you are kept busy enough without me chiming in ... but here comes another "non-coder" feature request: -

Relates to my post under fw 384.12-alpha1 here ... https://www.snbforums.com/threads/384-12_alpha-builds-testing-all-variants.56639/page-6#post-492419 ... but not sure that firmware change is feasible.

I then discovered this post [dealing with another issue - but with ntp sync at the heart of the problem] https://www.snbforums.com/threads/t...ugged-on-rt-ax88u-11.56524/page-2#post-492172

So ... I thought perhaps the solution for your amtm / disk check ... and its many add-on scripts - was to build in a short delay - [10 to 20 seconds?] -before any of the amtm provided scripts execute - so that the WAN is up and ntp is in sync first?

Not only would this give an accurate time stamp for Disk Check log - but also possibly resolve some of the other issues which arise for lack of ntp sync?

 

[thelonelycoder]

@thelonelycoder, my two previous questions remain, if you don't mind answering them:

1. Now that I'm using a hosted whitelist, if I want to add additional domains, I have to edit the hosted file and add the domains to it? No way to manually add individual domains though Diversion alone, like before?
2. Auto-add www/non-www domain is currently set to "off". Should I turn it on?

1. Yes that's the idea. Managing your <lists> however you want. On the PC, through a Web interface, whatever suits you best. I have multiple routers, using hosted lists makes the black and whitelisting much simpler.
2. You do the www/non-www stuff. This option is only active for the non-hosted whitelist. The menu entries will be greyed out in next update.

Also, have you seen this?

This is ... NOT what I recommend using on the router.
The list is 33 MB in raw format, 24.4 MB after Diversion processed it. It then contains the insane number of 1.190.510 domains.

I'm taking a wild guess here, but 85% of those domains will very likely never become active and block something. For the simple reason that one does not visit it as a first party or it gets loaded as a third party domain from a site you visit.

I am very conservative with the selection of hosts lists I ship Diversion with. This one is close to the event horizon, or already beyond reach just as Cooper is in Interstellar.

 

[thelonelycoder]

This should work as a hosted blacklist in Diversion, no?

No, NO. This is not a blacklist in the sense Diversion uses the term. It's not even remotely a good idea to add it as a hosts file in the blocking list options (this is where hosts lists go), see my other post about that hosts list.
The blocking list is the main blocking file. Your personal adjustments to the blocking list are done through the blacklist and the whitelist. I'll let you guess what these do.

 

[thelonelycoder]

I'm not sure I'd want a filterlist which uses a spam email host filter as a source. Completely different applications.

https://diversion.ch/diversion/use/blocking-file-hosts-files.html

 

[thelonelycoder]

So - perhaps a feature add? When uninstalling - provide option to retain configs / whitelists etc [effectively disabling Diversion] - then on reinstall give option [if prior install found] to start with "clean install" [in which case all prior configs etc wiped] or "reinstall to existing" [and thereby retaining prior configs etc]?

It's on my (long) to do list. Option will likely be to look for a specific file in /jffs/ or one of the mounted USB devices. No promises though.

BTW - really enjoy the fact that after ejecting USB - then plugging it back in again ... it automatically fires up Diversion etc etc [Thumbs-Up +1]

This "side effect" is no different from booting the router. The processes it triggers are part of the amazing firmware that Asuswrt-Merlin offers on our routers.
Thank @RMerlin for his forethought and excellent implementation.

 

[thelonelycoder]

So ... I thought perhaps the solution for your amtm / disk check ... and its many add-on scripts - was to build in a short delay - [10 to 20 seconds?] -before any of the amtm provided scripts execute - so that the WAN is up and ntp is in sync first?

Not only would this give an accurate time stamp for Disk Check log - but also possibly resolve some of the other issues which arise for lack of ntp sync?

No, the disk check already delays the mounting and every script that uses a post-conf file such as diversion add more time to fully start up. The firmware-set default timestamp is a minor issue to me. The amtm-disk-check.log timestamp is correct though.

 

[Dabombber]

https://diversion.ch/diversion/use/blocking-file-hosts-files.html

Ah yea, I guess my terminology is a bit off. I meant filterlist as a generic umbrella term for any kind of blocking file.

FilterList : A text file containing a list of rules for blocking or manipulating internet traffic

 

[thelonelycoder]

Ah yea, I guess my terminology is a bit off. I meant filterlist as a generic umbrella term for any kind of blocking file.

Blocking list or blocking file, it's all the same. I have largely renamed it in Diversion to blocking list as it is just a list, same with whitelist, blacklist. Fragments still remain like the name of the file that runs the update: update-bf.div.

Pi-Hole has it's own terminology around their black hole. If you look at the code there's gravity, matterandlight, preEventHorizon and so on.
I should rename functions, files and wordings to something 'funny' too. Like rerouting, redirection, turning aside, deflection, digression, deviation, divergence.
Wait, that's not funny. Better would be mergingLane, RoadClosed, ReducedSpeedLimitAhead, MenAtWork. Or Umleitung, Ablenkung, Zeitvertreib, Umweg, Zerstreuung, Ablenkungsmanöver, Abzweigung (German synonyms for Diversion). Or in my native Swiss German which is only a spoken language, rarely written: Omfahrig, Verchehrsomleitig, Spuurversetzig, Ziit verplämple.

 

[Grisu]

Spuurversetzig, Ziit verplämple.

These are your best in english latter would be something like "time wasting"

 

[thelonelycoder]

These are your best in english latter would be something like "time wasting"

Something that breaks the routine

 

[thelonelycoder]

Not only would this give an accurate time stamp for Disk Check log - but also possibly resolve some of the other issues which arise for lack of ntp sync?

OK: https://www.snbforums.com/threads/a...erlin-terminal-menu.42415/page-49#post-492518

 

[kernol]

No, the disk check already delays the mounting and every script that uses a post-conf file such as diversion add more time to fully start up. The firmware-set default timestamp is a minor issue to me. The amtm-disk-check.log timestamp is correct though.

Bold type above - not the case ... at least not on every single reboot I have done in the recent past [and there have been many]. Not sure if it is because the AC5300 is faster than many other routers ... but here is an extract from my syslog ...

May  5 07:05:20 kernel: usbcore: registered new interface driver usblp
May  5 07:05:20 custom_script: Running /jffs/scripts/service-event (args: start firewall)
May  5 07:05:21 kernel: scsi 0:0:0:0: Direct-Access     SanDisk  Cruzer Blade     1.27 PQ: 0 ANSI: 6
May  5 07:05:21 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
May  5 07:05:21 kernel: sd 0:0:0:0: [sda] 15330304 512-byte logical blocks: (7.84 GB/7.30 GiB)
May  5 07:05:21 kernel: sd 0:0:0:0: [sda] Write Protect is off
May  5 07:05:21 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
May  5 07:05:21 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
May  5 07:05:21 kernel:  sda: sda1
May  5 07:05:21 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
May  5 07:05:21 kernel: sd 0:0:0:0: [sda] Attached SCSI removable disk
May  5 07:05:22 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
May  5 07:05:22 kernel: nf_conntrack_rtsp v0.6.21 loading
May  5 07:05:22 kernel: nf_nat_rtsp v0.6.21 loading
May  5 07:05:22 custom_script: Running /jffs/scripts/firewall-start (args: ppp0)
May  5 07:05:23 rc_service: ip-up 573:notify_rc start_ntpd
May  5 07:05:23 rc_service: waitting "stop_ntpd" via ip-up ...
May  5 07:05:23 custom_script: Running /jffs/scripts/service-event (args: stop ntpd)
May  5 07:05:24 rc_service: ip-up 573:notify_rc stop_upnp
May  5 07:05:24 rc_service: waitting "start_ntpd" via ip-up ...
May  5 07:05:24 custom_script: Running /jffs/scripts/service-event (args: start ntpd)
May  5 07:05:24 custom_script: Running /jffs/scripts/pre-mount (args: /dev/sda1 ext4)
May  5 07:05:24 ntpd: Started ntpd
May  5 07:05:25 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
May  5 07:05:25 amtm disk-check: Probing ext4 on device /dev/sda1
May  5 07:05:25 amtm disk-check: Running disk check (v2.2) 'e2fsck -p' on /dev/sda1
May  5 07:05:25 rc_service: ip-up 573:notify_rc start_upnp
May  5 07:05:25 rc_service: waitting "stop_upnp" via ip-up ...
May  5 07:05:25 custom_script: Running /jffs/scripts/service-event (args: stop upnp)
May  5 07:05:26 amtm disk-check: Disk check done on /dev/sda1
May  5 07:05:26 custom_script: Running /jffs/scripts/service-event (args: start upnp)
May  5 07:05:27 start_ddns: update WWW.ASUS.COM update@asus.com, wan_unit 0
May  5 07:05:27 syslog: USB ext4 fs at /dev/sda1 mounted on /tmp/mnt/usb2
May  5 07:05:27 usb: USB ext4 fs at /dev/sda1 mounted on /tmp/mnt/usb2.
May  5 07:05:27 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: user_xattr
May  5 07:05:27 inadyn[988]: In-a-dyn version 2.5 -- Dynamic DNS update client.
May  5 07:05:28 custom_script: Running /jffs/scripts/post-mount (args: /tmp/mnt/usb2)
May 22 07:43:38 kernel: Adding 524284k swap on /tmp/mnt/usb2/myswap.swp.  Priority:-1 extents:3 across:540668k
May 22 07:43:38 qos: Applying codel patch
May 22 07:43:38 inadyn[988]: Update forced for alias kernolac5300.asuscomm.com, new IP# xxxxxxxxx
May 22 07:43:38 ntp: Initial clock set

As you can see from the above - several /jffs/scripts are being called to execute well before time has been synced - and the amtm disk check is done and dusted before the clock is set to correct time from firmware date stamp. I don't believe that ANY of these /jffs/scripts should execute before time is in sync. Not sure if this is something for @RMerlin to fix - or whether you can control?

The 384.12-alpha thread I referred to you above is informative - as is the suggestion in a reply to it.

 

[thelonelycoder]

Not sure if this is something for @RMerlin to fix - or whether you can control?

Please read my comment in the amtm thread and continue this topic there.
And no, I cannot control everyone else's code.

 

[RMerlin]

Scripts that require an accurate clock should check that ntp_ready is set to 1.

 

[joe68000]

Diversion is great, thanks!

I have noticed that if I am watching the log file with "tail -f /tmp/syslog.log" after "Starting Diversion statistic generation..." the tail stops. I have to kill it and restart to pick up the logging again.

Also, both Diversion and Skynet both update exactly on the hour. Any easy way to offset them? Not sure if I can just edit the crontab. I use the equally awesome amtm, so I don't want to make alterations that may break anything.

 

[thelonelycoder]

I have noticed that if I am watching the log file with "tail -f /tmp/syslog.log" after "Starting Diversion statistic generation..." the tail stops. I have to kill it and restart to pick up the logging again.

I'm not sure why Diversion would have any influence on the tailing of the syslog file. But try with the upper case f switch, that retries if file is not found:

tail -F /tmp/syslog.log

Also, both Diversion and Skynet both update exactly on the hour.

Is that a problem for you or the router? I'm not aware it is for the latter and I doubt so if you think it is.

Also, both Diversion and Skynet both update exactly on the hour. Any easy way to offset them? Not sure if I can just edit the crontab. I use the equally awesome amtm, so I don't want to make alterations that may break anything.

Don't edit the Diversion cron jobs, Diversion will overwrite it. The hour and minute Diversion runs the blocking list update is tied to other things it does. You may break things if you do. You can change the hour it runs on in b "Change update day and time".

 

[Mircica]

Dear friend

After last minor update this appeared:

" blocking list options:

1. Change composition Large fs
2. Disable + Plus hosts
3. Change update day and time
4. Update blocking list now
5. Find domain in hosts file(s)
6. Alternate blocking list for specified clients

Enter selection [1-6 e=Exit] 4

i Running update-bf.div

using temporary pgl.yoyo.org hosts file to lower memory usage while updating
remote file same: using local hosts file
https://pgl.yoyo.org/adservers/serv...&showintro=0&mimetype=plaintext&useip=0.0.0.0
writing temporary pgl.yoyo.org blocking list

preparing temporary whitelist
updated /jffs/shared-Diversion-whitelist

downloading Large+ blocking list, 10 hosts file(s)
using 192.168.1.2 as blocking IP

processing file 1 of 10
no local file: getting new hosts file
http://support.it-mate.co.uk/downloads/hosts.txt
######################################################################## 100.0%
/opt/bin/grep: invalid option -- P
BusyBox v1.27.2 () multi-call binary.

Usage: grep [-HhnlLoqvsriwFE] [-m N] [-A/B/C N] PATTERN/-e PATTERN.../-f FILE [FILE]...

Search for PATTERN in FILEs (or stdin)

-H Add 'filename:' prefix
-h Do not add 'filename:' prefix
-n Add 'line_no:' prefix
-l Show only names of files that match
-L Show only names of files that don't match
-c Show only count of matching lines
-o Show only the matching part of line
-q Quiet. Return 0 if PATTERN is found, 1 otherwise
-v Select non-matching lines
-s Suppress open and read errors
-r Recurse
-i Ignore case
-w Match whole words only
-x Match whole lines only
-F PATTERN is a literal (not regexp)
-E PATTERN is an extended regexp
-m N Match up to N times per file
-A N Print N lines of trailing context
-B N Print N lines of leading context
-C N Same as '-A N -B N'
-e PTRN Pattern to match
-f FILE Read pattern from file
http://support.it-mate.co.uk/downloads/hosts.txt
not hosts file, no backup found

......................................
repeats 9 times and then:

mv: can't rename '/opt/share/diversion/backup/inuse*': No such file or directory"

Hardware: Asus RT-AC66U
Firmware: ASUSWRT-Merlin RT-AC66U 380.68-4
What can I do?
How can I go back to previous minor update?

 

[EeK]

1. Yes that's the idea. Managing your <lists> however you want. On the PC, through a Web interface, whatever suits you best. I have multiple routers, using hosted lists makes the black and whitelisting much simpler.
2. You do the www/non-www stuff. This option is only active for the non-hosted whitelist. The menu entries will be greyed out in next update.

Thanks for the answers, as always.

1. Using Pastebin does make it easier. For Diversion, I'm supposed to use the link with "dl" in it, right (pastebin.com/dl/<ID>)?
2. Oh, so if I'm using a hosted whitelist, that option doesn't matter? My current hosted whitelist only has two domains starting with "www." (www.msftncsi.com and www.xboxlive.com), all the others are in the "domain.com; otherdomain.com" format. Should I remove the "www." part from those two?

This is ... NOT what I recommend using on the router.
The list is 33 MB in raw format, 24.4 MB after Diversion processed it. It then contains the insane number of 1.190.510 domains.

I'm taking a wild guess here, but 85% of those domains will very likely never become active and block something. For the simple reason that one does not visit it as a first party or it gets loaded as a third party domain from a site you visit.

I am very conservative with the selection of hosts lists I ship Diversion with. This one is close to the event horizon, or already beyond reach just as Cooper is in Interstellar.

No, NO. This is not a blacklist in the sense Diversion uses the term. It's not even remotely a good idea to add it as a hosts file in the blocking list options (this is where hosts lists go), see my other post about that hosts list.
The blocking list is the main blocking file. Your personal adjustments to the blocking list are done through the blacklist and the whitelist. I'll let you guess what these do.

Does a hosted list substitute whatever Diversion ships with?

When adding either a hosted whitelist or a hosted blacklist, Diversion mentions that the pre-configured lists will be "replaced" by the hosted ones. From your comment, it seems that whitelists/blacklists work on top of blocking lists already included in Diversion.

As an experiment, I tried adding that linked blacklist as a hosted one in Diversion, to see if anything would change with my Xbox - it didn't. Should I remove it, then? The issue is the sheer size of it? I'm always eager to learn more about how stuff works.

 

[thelonelycoder]

Dear friend

After last minor update this appeared:

" blocking list options:

1. Change composition Large fs
2. Disable + Plus hosts
3. Change update day and time
4. Update blocking list now
5. Find domain in hosts file(s)
6. Alternate blocking list for specified clients

Enter selection [1-6 e=Exit] 4

i Running update-bf.div

using temporary pgl.yoyo.org hosts file to lower memory usage while updating
remote file same: using local hosts file
https://pgl.yoyo.org/adservers/serv...&showintro=0&mimetype=plaintext&useip=0.0.0.0
writing temporary pgl.yoyo.org blocking list

preparing temporary whitelist
updated /jffs/shared-Diversion-whitelist

downloading Large+ blocking list, 10 hosts file(s)
using 192.168.1.2 as blocking IP

processing file 1 of 10
no local file: getting new hosts file
http://support.it-mate.co.uk/downloads/hosts.txt
######################################################################## 100.0%
/opt/bin/grep: invalid option -- P
BusyBox v1.27.2 () multi-call binary.

Usage: grep [-HhnlLoqvsriwFE] [-m N] [-A/B/C N] PATTERN/-e PATTERN.../-f FILE [FILE]...

Search for PATTERN in FILEs (or stdin)

-H Add 'filename:' prefix
-h Do not add 'filename:' prefix
-n Add 'line_no:' prefix
-l Show only names of files that match
-L Show only names of files that don't match
-c Show only count of matching lines
-o Show only the matching part of line
-q Quiet. Return 0 if PATTERN is found, 1 otherwise
-v Select non-matching lines
-s Suppress open and read errors
-r Recurse
-i Ignore case
-w Match whole words only
-x Match whole lines only
-F PATTERN is a literal (not regexp)
-E PATTERN is an extended regexp
-m N Match up to N times per file
-A N Print N lines of trailing context
-B N Print N lines of leading context
-C N Same as '-A N -B N'
-e PTRN Pattern to match
-f FILE Read pattern from file
http://support.it-mate.co.uk/downloads/hosts.txt
not hosts file, no backup found

......................................
repeats 9 times and then:

mv: can't rename '/opt/share/diversion/backup/inuse*': No such file or directory"

Hardware: Asus RT-AC66U
Firmware: ASUSWRT-Merlin RT-AC66U 380.68-4
What can I do?
How can I go back to previous minor update?

You are missing the Entware grep binary /opt/bin/grep. For some reason there's a symlink there to presumably Entware's (or maybe your firmwares) BusyBox grep which does not support the switch I use in that file.

I think this will work, enter it into the terminal, it should install regular Entware grep:

opkg install grep

If you encounter an error, try removing the symlink:

rm /opt/bin/grep

then try to install it again.

Let me know if that works.

 

[thelonelycoder]

1. Using Pastebin does make it easier. For Diversion, I'm supposed to use the link with "dl" in it, right (pastebin.com/dl/<ID>)?

The raw link to the file, click on the icon and copy/paste the URL in the browser: