Diversion Diversion - the Router Ad-Blocker

[Xentrk]

@thelonelycoder How is Diversion handling what used to be the shared white list file you set up for sharing with Skynet and other apps in AB-Solution? My selective routing project needs to use a shared whitelist containing an entry for ipinfo.io as Skynet, and ABS, may block it depending on the blocking files selected by the user.

 

[thelonelycoder]

@thelonelycoder How is Diversion handling what used to be the shared white list file you set up for sharing with Skynet and other apps in AB-Solution? My selective routing project needs to use a shared whitelist containing an entry for ipinfo.io as Skynet, and ABS, may block it depending on the list.

I have this code in Diversion in update-bf.div. Skynet, Diversion and AB include all /jffs/shared-*-whitelist found except its own whenever the code runs.

for wlists in /jffs/shared-*-whitelist; do
    if [ -e "$wlists" ] && [ "$wlists" != "/jffs/shared-Diversion-whitelist" ]; then
        echo " including $wlists in $NAME"
        trim_file "$wlists"
        grep "^[^#]" "$wlists" | sed 's#http[s]*://##;s#/.*##' | sed -e 's/^/ /;s/$/$/' >>"${whitelist}.tmp"
        sort -u "${whitelist}.tmp" -o "${whitelist}.tmp"
    fi
done

Note that I do additional stuff with the .tmp file after.
Just name yours shared-<unique specific name>-whitelist and it'll be included.

 

[Xentrk]

I have this code in Diversion in update-bf.div. Skynet, Diversion and AB include all /jffs/shared-*-whitelist found except its own whenever the code runs.

for wlists in /jffs/shared-*-whitelist; do
    if [ -e "$wlists" ] && [ "$wlists" != "/jffs/shared-Diversion-whitelist" ]; then
        echo " including $wlists in $NAME"
        trim_file "$wlists"
        grep "^[^#]" "$wlists" | sed 's#http[s]*://##;s#/.*##' | sed -e 's/^/ /;s/$/$/' >>"${whitelist}.tmp"
        sort -u "${whitelist}.tmp" -o "${whitelist}.tmp"
    fi
done

Note that I do additional stuff with the .tmp file after.
Just name yours shared-<unique specific name>-whitelist and it'll be included.

Ah. I was looking in the /jffs/configs directory. Forgot the shared white list file is located in the /jffs directory. I have updated three routers so far and have one more to do. They all converted from ABS without any issues. Great job.

 

[Xentrk]

I like how you placed Diversion in the /opt/share directory. I was placing my IPSET backups in /opt/tmp. But now you have me thinking about creating a backup directory under /opt/share.

 

[thelonelycoder]

I like how you placed Diversion in the /opt/share directory. I was placing my IPSET backups in /opt/tmp. But now you have me thinking about creating a backup directory under /opt/share.

The main file diversion is in /opt/bin while the rest of the Diversion files are in /opt/share/diversion. Dnsmasq logfiles are written to /opt/var/log

 

[Makaveli]

When it happened? Straight after reboot and loading diversion? How Long it last? What is the frequency of it having spike?
Mine is 384.7 alpha 1 and the cpu quite normal.

I only noticed it after randomly going into the GUI to check something else. I was sitting at about 37 days uptime so it wasn't from a reboot. And it was sitting high load like that for hours if not days I just didn't notice it.

I had also tried turning Diversion off which didn't change the usage.
Resetting the shell history was the same result.
Reboot also same result.
Only me ending the process brought things back to normal.

 

[bengalih]

It sorts the .ash_history file, a function that is not built into the command line options of BusyBox. It also keeps the .ash_history file in jffs so it survives a reboot. Running it at boot ensures you have a sorted file when next logging in with ssh.

Adding the last diversion commands to the top is a feature that Diversion users might find handy.
Keeping the ash history and sorting it has been added way back in earlier AB-Solution releases.

If your history is too long, just runs sh in Diversion, this resets the file. It can be found in the expanded menu o.

I missed this line:

ln -sf /jffs/.ash_history /home/root/

That is indeed a nice feature to have, I just have 3 comments/concerns/suggestions:

1) It is possible - especially since we are preserving history that this file could grow very large. Therefore the sort feature could indeed peg CPU while it operates on a large file. How large this file has to be and how many CPU cycles it could take up are unknown to me, but this is what could be happening to some?

2) It would be nice if the preservation of the file wasn't linked to the sorting. Some people might like to have the file retained, but don't want it sorted for a couple of reasons:
- they have a large file and don't want the CPU cycles associated with it. (if indeed that is the cause of the observed issues some have).
- they want their history preserved as-is and don't want recent commands to just be put in randomly (based on sort order).
- possible concerns about excess write activity to jffs/flash memory. (I don't really know the true impact of this, but I know some users thing it is a big deal).

I know you state you can just reset/delete your history, but a user shouldn't have to choose between deleting their history or retaining a large one and dealing with large CPU cycles from the Diversion sort process.

3) Continuing on #2...
Please don't take this in the wrong way since all your efforts are appreciated and I'm just stating a personal preference/design choice. I've been seeing a lot of "feature creep" in ab-solution/Diversion's history. Lots of things that really have nothing (or very little) to do with the function of as-blocking seem to be integrated into the script.
Normally, I wouldn't have a problem with this. In fact, I have been guilty myself in previous projects I coded of adding in all sorts of little features as "nice to have" either because they were requested or because they were just something I wanted/used so I added them in even if they were not really related. In the end though I learned this leads to bloat, and more maintenance, and ultimately more work in documentation, support, and confusion.

Additionally, while I have no issue with a feature like this being in Diversion, I am a little concerned that unless you are actually reviewing the source code (or at least reviewing the finished contents of all the jffs scripts) you would never know that this feature is happening. If a user didn't want their history retained (or sorted), or had something similar setup this feature could disturb their setup. It would be great that this feature, and any like it (in terms of system/user impact not directly related to Diversion functionality) was at the *very least* documented and presented to the user in a post-install "readme" that displays to the screen. This way the user is aware of changes to their system.
Ideally, the user would be able to opt in/out of these features without having to just go do post-install cleanup.
Obviously this would mean some more choices/lines in the installation code.

To be honest, a feature like this I think has much better place in amtm then in Diversion.
I would almost rather see you ship amtm with Diversion and install it automatically.
You could then remove/move all non-diversion specific (i.e. ad blocking) into amtm.
amtm could then be added as a sub menu item in Diversion, so the end effect is all these things could be maintained within diversion, but are part of the amtm code base. Additionally/preferably all these things in amtm are opt-in (which is currently the case with the available amtm options afaik).

Anyway, again these are just some observations from an interested outsider (with some coding/project management experience) new to the solution.
I might question some of the choices made, but I'll still remain grateful to whatever work is done!
Thanks.

 

[jtp10181]

Just updated from 3.x and seems like it went pretty well. It did replace my update-notification script with its own without making a backup, luckily I made backups ahead of time. Not really thrilled it just replaced that file without warning and no way to shut that feature off like previously.

Also noticed that the update notify and any other emails from the script no longer work for me. To fix it, in the email config file I had to change the protocol to plain "smtp" and also removed the "--ssl-reqd" switch from the curl command in the script. Specifying the protocol for this is not even necessary in my experience so IMO that could be totally removed. And if the --ssl-reqd could be removed from the script itself and included in the config file with the $SSL_FLAG var then it could be working for me without editing the script (which will break with updates).

Now before everyone replies and says its not secure without the SSL... my ISP (and I suspect many still do this) allows a local connection to the smtp server on port 25 with NO user/pass (in the config I just pass none/none and it works). I actually like this better because I am not saving my password in a plain-text file on the router. I will have to test and maybe see if they now allow a connection on a secure port still without the user/pass combo.

 

[cvx01]

@thelonelycoder. Okay, what I am doing wrong here? I noticed weekly stats haven't been compiling locally (I don't use the email option) since I upgraded to D4.

I ran C, 2, 2, 1 (log below in the code window) which should give me a file in /opt/share/diversion/stats

No file is output and an error is shown after a "Done" message

Done The current stats are being compiled now.
This will take some time. If compiling fails,
check the stats.div.log in sf

What do you want to do? /opt/share/diversion/file/stats.div: .: line 54: can't open '/opt/share/diversion/.conf/email.conf'

I then "follow the prompts" above and check stats.div.log under /mnt/AB/entware/share/diversion/log

stats.div.log indicates what you see below. Nothing useful.

Sun, 02 Sep 2018 09:25:59 -0700
diversion.conf found, compiling stats

The error seems to indicate an attempt to email is being performed. I do not and have not ever set that up. I am simply trying to export a log to the usual stats folder (/mnt/AB/entware/share/diversion/stats)

What am I doing wrong here? Thank you!

 Diversion 4.0                    by thelonelycoder

 RT-AC68U (armv7l) FW-374.43 @ 192.168.1.1

 261,345  blocked domains  9  hosts file(s) in use
 10.468M t  1,154 w  931 n ads since Sep 02 05:20
____________________________________________________

 d   Diversion Standard   enabled
 c   communication        DivUn stats backup FWun

 a   ad-blocking          to IP 192.168.1.2
 l   logging              /opt/var/log/dnsmasq.log

 ep  pixelserv-tls        192.168.1.2 v2.1.1

 b   blocking file        Custom Sun @ 2:00
 el  edit lists

 f   follow dnsmasq.log

 e   exit Diversion                 more options  o
____________________________________________________

 What do you want to do?  c
____________________________________________________

 Communication settings:

 1. Diversion update notification DivUn
 2. Diversion stats stats:file,body,save
 3. Diversion backup off
 4. Firmware update notification FWun
 5. Edit email settings, send testmail

 Enter selection [1-5 e=Exit] 2
____________________________________________________

 Router stats info:

 Set weekly stats to on or off.
 Edit settings or compile current stats.

 1. Weekly stats settings
 2. Compile current stats report
 3. Top 'x' setting  Top 30 hosts for 30 clients
 4. Filter local client names  off
 5. Show local clients list

 Select what to do: [1-5 e=Exit] 2
____________________________________________________

 This compiles a current stats report.

 Select how to output the current stats report:

 1. Save to file in /stats/
 2. As email, stats in body
 3. As email, stats in body & safe to file
 4. As email, stats as attachment
 5. As email, stats as attachment & safe to file

 Select output [1-5 e=Exit] 1
 Diversion 4.0                    by thelonelycoder

 RT-AC68U (armv7l) FW-374.43 @ 192.168.1.1

 261,345  blocked domains  9  hosts file(s) in use
 10.468M t  1,154 w  931 n ads since Sep 02 05:20
____________________________________________________

 d   Diversion Standard   enabled
 c   communication        DivUn stats backup FWun

 a   ad-blocking          to IP 192.168.1.2
 l   logging              /opt/var/log/dnsmasq.log

 ep  pixelserv-tls        192.168.1.2 v2.1.1

 b   blocking file        Custom Sun @ 2:00
 el  edit lists

 f   follow dnsmasq.log

 e   exit Diversion                 more options  o
____________________________________________________

 Done  The current stats are being compiled now.
 This will take some time. If compiling fails,
 check the stats.div.log in  sf

 What do you want to do?  /opt/share/diversion/file/stats.div: .: line 54: can't  open '/opt/share/diversion/.conf/email.conf'

 

[thelonelycoder]

Just updated from 3.x and seems like it went pretty well. It did replace my update-notification script with its own without making a backup, luckily I made backups ahead of time. Not really thrilled it just replaced that file without warning and no way to shut that feature off like previously.

You have modified an AB-Solution generated script, hence it was replaced with the version that works with Diversion.
The line you left in is "generated by AB-Solution".

Also noticed that the update notify and any other emails from the script no longer work for me. To fix it, in the email config file I had to change the protocol to plain "smtp" and also removed the "--ssl-reqd" switch from the curl command in the script. Specifying the protocol for this is not even necessary in my experience so IMO that could be totally removed. And if the --ssl-reqd could be removed from the script itself and included in the config file with the $SSL_FLAG var then it could be working for me without editing the script (which will break with updates).

Noted, but some complain about the SSL_FLAG (--insecure) not being necessary, others have specific problems with their ISP's.
Can't do it right for everyone.

 

[jtp10181]

You have modified an AB-Solution generated script, hence it was replaced with the version that works with Diversion.
The line you left in is "generated by AB-Solution".

I had the update notification script disabled in ABS and below was the full contents of my previous update-notification script (and the file that runs is also totally custom does not say anything about AB-Solution):

#!/bin/sh

#Run My Update Notify
/jffs/custom/update-notify-jtp

 

[thelonelycoder]

The error seems to indicate an attempt to email is being performed. I do not and have not ever set that up. I am simply trying to export a log to the usual stats folder (/mnt/AB/entware/share/diversion/stats)

My oversight. Just create the file in c, with fake stuff like asdf or whatever. The stats file also looks for the friendly name of your router in there.

 

[thelonelycoder]

I had the update notification script disabled in ABS and below was the full contents of my previous update-notification script (and the file that runs is also totally custom does not say anything about AB-Solution):

#!/bin/sh

#Run My Update Notify
/jffs/custom/update-notify-jtp

I don't see a way how Diversion did not honor your custom script then.
The code it runs is:

if [ -f "/jffs/scripts/update-notification" ] && grep -q "generated by AB-Solution" /jffs/scripts/update-notification; then
    rm -f /jffs/scripts/update-notification
    fwUpdate=on
fi

If it finds the file /jffs/scripts/update-notification and the wording of "generated by AB-Solution" then it is replaced. If not, nothing is done to it.

 

[jtp10181]

I don't see a way how Diversion did not honor your custom script then.
The code it runs is:

if [ -f "/jffs/scripts/update-notification" ] && grep -q "generated by AB-Solution" /jffs/scripts/update-notification; then
    rm -f /jffs/scripts/update-notification
    fwUpdate=on
fi

If it finds the file /jffs/scripts/update-notification and the wording of "generated by AB-Solution" then it is replaced. If not, nothing is done to it.

Dont know how but I am telling you with 100% certainty that it did in fact replace it and what I posted above is 100% for sure what was there before (I just made a jffs backup 5 minutes before I updated and used the port option). I guess if no one else reports the same issue maybe its a fluke.

Also, for the email issue above, I went out and made a new dummy email account with my ISP with a new password to use just for connecting to the SMTP server that supports SSL. So I did get that to work as-is now and at least the password saved on the router would not give anyone access to any settings in my ISP account.

 

[jtp10181]

Ok I messed with it a little more, I had the fwUpdate=off in my config file but the menu had the option to disable and would not let me enabled it. I manually set it to on, then the menu let me disable it and I found when disabled it was removed from the config file totally. Ok So then I made sure no update script was there anymore and enabled / disabled it checking it each time. Seems to work fine. Then I disabled it again, moved my custom script back and enabled it to see what would happen. Well I got an error which seems to indicate it would not be doing anything since my custom file was there, but then it proceeded to replace it anyway. Not sure if this is your intended behavior, or possibly why my custom script got replaced in the first place?

 1. Enable firmware notification

 Select what to do: [1-1 e=Exit] 1

 Found a custom notification script at:
 /jffs/scripts/update-notification
 This needs to be removed first.
  ✔  update-notification integrated
____________________________________________________

 Firmware update notification info:

 Receive email notification for newer
 Asuswrt-Merlin firmware availability for this
 router within about 48 hours of release.

 1. Disable firmware notification
 2. Send a test notification email

 Select what to do: [1-2 e=Exit] e

 

[thelonelycoder]

Dont know how but I am telling you with 100% certainty that it did in fact replace it and what I posted above is 100% for sure what was there before (I just made a jffs backup 5 minutes before I updated and used the port option). I guess if no one else reports the same issue maybe its a fluke.

Also, for the email issue above, I went out and made a new dummy email account with my ISP with a new password to use just for connecting to the SMTP server that supports SSL. So I did get that to work as-is now and at least the password saved on the router would not give anyone access to any settings in my ISP account.

I have to correct myself. You say it was turned to "off" in AB-Solution. In that case the file was indeed replaced during the install.
Will be fixed in a future update.

 

[thelonelycoder]

Not sure if this is your intended behavior, or possibly why my custom script got replaced in the first place?

It is indeed. Don't run Re-install or Update in d until the update is posted or else it will be replaced again.

 

[thelonelycoder]

Thanks for reporting btw @jtp10181

 

[jtp10181]

It is indeed. Don't run Re-install or Update in d until the update is posted or else it will be replaced again.

Sounds good, and just to clarify to prevent future issues once its fixed. To use a custom notify script should I leave Diversion set to disabled for the fwUpdate (and that will tell it to just leave it alone and not mess with my custom one)? Or should I enabled it and then copy over my custom script (and then it would detect it is custom and leave it alone)? I suspect leaving it disabled would give me the desired results, and if I leave it enabled any subsequent update or reinstall would revert it back to your script.

 

[thelonelycoder]

To use a custom notify script should I leave Diversion set to disabled for the fwUpdate (and that will tell it to just leave it alone and not mess with my custom one)?

No need to. The updater first gets the main script which contains the flawed test for the firmware notification.