Diversion Diversion - the Router Ad-Blocker

[thelonelycoder]

1) It is possible - especially since we are preserving history that this file could grow very large. Therefore the sort feature could indeed peg CPU while it operates on a large file. How large this file has to be and how many CPU cycles it could take up are unknown to me, but this is what could be happening to some?

I just tested it on a 39 MB large file. It took 55 seconds to sort it on my RT-AC1900P. It would have to be a VERY large history file.

2) It would be nice if the preservation of the file wasn't linked to the sorting. Some people might like to have the file retained, but don't want it sorted for a couple of reasons:
- they have a large file and don't want the CPU cycles associated with it. (if indeed that is the cause of the observed issues some have).
- they want their history preserved as-is and don't want recent commands to just be put in randomly (based on sort order).
- possible concerns about excess write activity to jffs/flash memory. (I don't really know the true impact of this, but I know some users thing it is a big deal).

It's a minor issue, don't blow it up as if it nuked a router. Diversion is installed and it keeps the history file. I just keep the shell history commands, sort and remove duplicates. As a bonus to Diversion users.

I've been seeing a lot of "feature creep" in ab-solution/Diversion's history. Lots of things that really have nothing (or very little) to do with the function of as-blocking seem to be integrated into the script.

What feature, except for keeping the shell history do you think is unnecessary in Diversion?

 

[bengalih]

I just tested it on a 39 MB large file. It took 55 seconds to sort it on my RT-AC1900P. It would have to be a VERY large history file.
...
It's a minor issue, don't blow it up as if it nuked a router. Diversion is installed and it keeps the history file. I just keep the shell history commands, sort and remove duplicates. As a bonus to Diversion users.
...

What feature, except for keeping the shell history do you think is unnecessary in Diversion?

Yeah...as I said I'd expect it to have to be exceptionally large, which is unlikely. However if some users are seeing exceptionally high CPU utilization on the "diversion ash_history", what else could it be attributed to? I would think it would mean either diversion is acting as intended and is just taking a long time due to the file size, or perhaps there is an issue with one of the underlying file systems that is causing the wheels to spin.

It isn't a major issue...and I'm not trying to make it a big deal - it isn't a nuked router. But, if it is causing issues with some configurations, then I would err on the side of caution. TBH, my biggest issue as I stated is simply that it is configuring this option without user knowledge. It isn't like this is an "evil" thing or you are installing spyware, but it may disrupt a user's configuration. I think many users (myself included) may install amtm/diversion as the first step after a clean router and as such these tools are great menu driven ways to automate a bunch of system setup. That being said, I wouldn't take for granted that a user is going to do this and disrupt/interfere with other things they might have going on. Case in point with the release of Diversion it seems (based on thread posts I'm reading..I'm not sure because I just rebuilt from scratch) that many of the jffs scripts are getting removed. While it is true that diversion/ab-solution might have used and/or even created this scripts, they aren't *solely* used by these. It appears that part of the upgrade wipes these whole hog (though it does backup). I realize you did speak to that point in terms of it not being worth the effort to try and upgrade the scripts, and that's fine, with ample warnings/documentation things are ok. I used to tell my development team that I considered a ton of stuff "bugs", but if they document them I'm willing to call them "features"

As far as other "unnecessary" stuff...first I don't really consider it "unnecessary" but I do think it adds more work for you to maintain the base diversion code. Especially when you have another such neat idea in amtm that can be used as a plugin for script and tweaks. IMHO amtm is underused and I think if you made it a bit extensible such that other devs could "drop-in" a tweak pack, etc that it could have much more utility.

Just doing a brief look, what comes to mind is:
1) Swap file - yes, I know it is recommended, but for instance I know of no windows or linux apps that I have used that walk you through or automate configuring the swap file. It is a nice feature for people who might not know how, and I realize you probably did this to prevent having to explain the process. But, one issue this has is that it seems to only recognizes a swap file on the system that has been added by diversion/amtm (or at least has a comment line pointing to it in the /jffs/scripts). This could be misleading for someone maintaining a swap file off the system and could cause them to create another unnecessary one. Relatively minor? Yes...but you asked
2) The "sf" option - problematic? No...not in the least, but not really necessary to have a built in file viewer (not editor) just for the config files. This isn't as problematic to me as the .ash_history because it doesn't actually modify anything, but it was a bunch of extra coding that provides the same function as a simple cat/echo/vi.

Look, I'm not trying to bust your balls or act like you've done something shady and/or dumb. I'm just offering an opinion which you are free to take or leave. Obviously I appreciate your work and want to see it succeed or else I wouldn't be wasting my time

 

[Adamm]

1) Swap file - yes, I know it is recommended, but for instance I know of no windows or linux apps that I have used that walk you through or automate configuring the swap file. It is a nice feature for people who might not know how, and I realize you probably did this to prevent having to explain the process. But, one issue this has is that it seems to only recognizes a swap file on the system that has been added by diversion/amtm (or at least has a comment line pointing to it in the /jffs/scripts). This could be misleading for someone maintaining a swap file off the system and could cause them to create another unnecessary one. Relatively minor? Yes...but you asked

The swap file code like many other things use somewhat of a "coding standard" a few of us developed and try to adhere to.

 

[thelonelycoder]

2) The "sf" option - problematic? No...not in the least, but not really necessary to have a built in file viewer (not editor) just for the config files. This isn't as problematic to me as the .ash_history because it doesn't actually modify anything, but it was a bunch of extra coding that provides the same function as a simple cat/echo/vi.

Remember this: The reason why I automated "Adblocking with a combined hostsfile" is to make it accesable to a larger audience. Many of the users of Diversion/AB-Solution don't want to or can't remember all the commands sprinkled around this board. You use a GUI on your PC, not the commandline for trivial tasks.

A file or cron jobs viewer comes naturally with an application such as the ones I have coded.
Even the simple amtm is a godsent to some users. It makes complex procedures so much simpler for them.

 

[M@rco]

@thelonelycoder: not sure whether this is a bug or whether something else went wrong, but my weekly backup got sent without the requested backup archive. Mail settings have been tested and confirmed to work.

My settings:

Tried to run it manually, which did work as supposed:

What I received (00.50 being the automated weekly backup yet without attachment, 07.01 being the manually sent Backup mail):

Any idea what has caused this? If you need anything to troubleshoot, just let me know.

 

[Xentrk]

I was trying to trouble shoot what first appeared to be a pixelserv-tls cert message error for @kvic when I tried to access reddit.com. In the process, I set up syslog-ng to help get more information. It appears Diversion is not white listing domains I have in the white list file.

I have processed the whitelist and all files, bounced Diversion and pixelserv.tls. 192.168.22.2 is the pixelserv-tls IP address

Sep  3 12:50:59 dnsmasq[32317]: query[A] www.reddit.com from 192.168.22.152
Sep  3 12:50:59 dnsmasq[32317]: blocked by blockinglist www.reddit.com is 192.168.22.2

I did a reinstall of Diversion.

 

[kvic]

@Xentrk

"-u" is not for enabling logging. It shall be "-l 4" that will enable log LEVEL 4 in pixelserv-tls. Alternatively you can also dynamically change to log LEVEL 4 with command like this:

curl -ks http://<pixelserv ip>/log=4

To see a full list of pixelserv-tls command line options. try "pixelserv-tls -h" or visit the manpage here: https://github.com/kvic-z/pixelserv-tls/wiki/Command-Line-Options

 

[M@rco]

@thelonelycoder I found a type in the wc-whitelist menu after trying to whitelist a domain which is blacklisted:

 reddit.com
 is in the blacklist, removit there first.

  !  Press [Enter] to return

and another remark, shouldn't 'starting with' be 'ending with' or 'matching with' ?

 The wildcard-blacklist blocks everything
 starting with *example.com.

 

[Xentrk]

@thelonelycoder I found a type in the wc-whitelist menu after trying to whitelist a domain which is blacklisted:

 reddit.com
 is in the blacklist, removit there first.

  !  Press [Enter] to return

and another remark, shouldn't 'starting with' be 'ending with' or 'matching with' ?

 The wildcard-blacklist blocks everything
 starting with *example.com.

What do you see when you go to a blocked site? I tested with some gambling sites and see this type of error message:

Or, should I be seeing a blank screen?

 

[M@rco]

@Xentrk I had the 'Secure Connection Failed' error first, after that it became a blank screen. Reddit.com was explicitely blacklisted in my case, but after removing it from the blacklist, it still wouldn't load. So I checked Skynet debug to see if I could find a cause, and all of a sudden it loads fine. I haven't explicitely whitelisted it anywhere, so it's apparently not in any other blocking file. To make things (at least to me) more confusing: at first it showed Pixelserv CA when hovering over the padlock, now it shows DigiCert Inc. Not sure I'm following this. Actually, I'm sure I'm not following this...

 

[Xentrk]

@Xentrk

"-u" is not for enabling logging. It shall be "-l 4" that will enable log LEVEL 4 in pixelserv-tls. Alternatively you can also dynamically change to log LEVEL 4 with command like this:

curl -ks http://<pixelserv ip>/log=4

To see a full list of pixelserv-tls command line options. try "pixelserv-tls -h" or visit the manpage here: https://github.com/kvic-z/pixelserv-tls/wiki/Command-Line-Options

I accidentally deleted my original reply. But thank you again for pointing out my error. So the issue now appears to be with Diversion somehow not using my whitelist file.

 

[Xentrk]

@Xentrk I had the 'Secure Connection Failed' error first, after that it became a blank screen. Reddit.com was explicitely blacklisted in my case, but after removing it from the blacklist, it still wouldn't load. So I checked Skynet debug to see if I could find a cause, and all of a sudden it loads fine. I haven't explicitely whitelisted it anywhere, so it's apparently not in any other blocking file. To make things (at least to me) more confusing: at first it showed Pixelserv CA when hovering over the padlock, now it shows DigiCert Inc. Not sure I'm following this. Actually, I'm sure I'm not following this...

I can see in dnsmasq.log that Diversion is blocking reddit.com even though it is in the whitelist. I also made sure reddit.com was whitelisted in Skynet using the Unban->Domains option.

Will wait for others to chime in if their whitelist is working or not working.

I wonder if @kvic can add a new category to the servstats page called wtf to capture the Secure Connection Failed error message

tmo   0   # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls   689   # of dropped requests (client disconnect without sending any request)
cly   0   # of dropped requests (client disconnect before response sent)
clt   0   # of dropped requests (reached maximum service threads)
err   0   # of dropped requests (unknown reason)
wtf  2   # of secure connection failed errors (unknown reason)

 

[Xentrk]

I did a reboot and pixelserv-tls refused to start-up on Port 80 and 443, complaining they were in use. I ended up uninstalling and reinstalling Diversion. The whitelist is working now! Perhaps something did not convert from AB-Solution. I will check out the other two sites and report back.

UPDATE
On one site that I converted from AB-Soluton to Diversion, it looks like pixelserv-tls was not starting up

Sep  3 20:27:26 RT-AC88U-5248 pixelserv[13517]: 194659 uts, 0 log, 134961 req, 2853 avg, 46888 rmx, 423 tav, 18380 tmx, 65249 slh, 24 slm, 0 sle, 7883 slu, 11225 nfe, 46 gif, 2 ico, 388 txt, 3 jpg, 0 png, 0 swf, 0 sta, 0 stt, 151 ufe, 75 rdr, 0 nou, 0 pth, 2 204, 48300 pst, 0 hed, 78 bad, 0 err, 58 tmo, 74511 cls
Sep  3 20:27:26 RT-AC88U-5248 pixelserv[13517]: exit on SIGTERM

I fixed doing an uninstall/install of Diversion

Sep  3 20:33:09 RT-AC88U-5248 pixelserv-tls[29838]: pixelserv-tls 2.1.1 (compiled: May 12 2018 06:33:51) options: 192.168.2.2
Sep  3 20:33:09 RT-AC88U-5248 pixelserv-tls[29838]: Listening on :192.168.2.2:443
Sep  3 20:33:09 RT-AC88U-5248 pixelserv-tls[29838]: Listening on :192.168.2.2:80

NO issues with pixelserv-tls on the third site.

 

[skeal]

I can see in dnsmasq.log that Diversion is blocking reddit.com even though it is in the whitelist. I also made sure reddit.com was whitelisted in Skynet using the Unban->Domains option.

Will wait for others to chime in if their whitelist is working or not working.

I wonder if @kvic can add a new category to the servstats page called wtf to capture the Secure Connection Failed error message

tmo   0   # of timeout requests (client connect w/o sending a request in 'select_timeout' secs)
cls   689   # of dropped requests (client disconnect without sending any request)
cly   0   # of dropped requests (client disconnect before response sent)
clt   0   # of dropped requests (reached maximum service threads)
err   0   # of dropped requests (unknown reason)
wtf  2   # of secure connection failed errors (unknown reason)

When trying to unblock something in Skynet you should use the whitelist not unban. IMHO.

 

[Xentrk]

When trying to unblock something in Skynet you should use the whitelist not unban. IMHO.

Excellent point!

 

[thelonelycoder]

@thelonelycoder: not sure whether this is a bug or whether something else went wrong, but my weekly backup got sent without the requested backup archive. Mail settings have been tested and confirmed to work.

That's a confirmed bug for the automated backup. Manual backup works.
Will be fixed with the next update.

Thanks for reporting.
Stats look OK?

 

[M@rco]

Thanks for reporting.

Most welcome

Stats look OK?

Stats look OK as far as I can tell, let's see after a full week if there's anything out of the ordinary.

 

[thelonelycoder]

I was trying to trouble shoot what first appeared to be a pixelserv-tls cert message error for @kvic when I tried to access reddit.com. In the process, I set up syslog-ng to help get more information. It appears Diversion is not white listing domains I have in the white list file.

View attachment 14297
I have processed the whitelist and all files, bounced Diversion and pixelserv.tls. 192.168.22.2 is the pixelserv-tls IP address

Sep  3 12:50:59 dnsmasq[32317]: query[A] www.reddit.com from 192.168.22.152
Sep  3 12:50:59 dnsmasq[32317]: blocked by blockinglist www.reddit.com is 192.168.22.2

I did a reinstall of Diversion.

FYI: The Medium blocking file uses github.com/StevenBlack/fakenews-gambling-porn-social, which blocks social media, including reddit:

# Reddit

0.0.0.0 i.reddit.com
0.0.0.0 redd.it
0.0.0.0 reddit.com
0.0.0.0 www.reddit.com
0.0.0.0 amp-reddit-com.cdn.ampproject.org
0.0.0.0 old.reddit.com
0.0.0.0 new.reddit.com

In that case, if you whitelist reddit.com or www.reddit.com only the exact match will be removed.

 

[SamHNB]

I am running Diversion 4.0 after fresh install and configuration of Merlin 384.6 running on RT-AC68U.

So far, things are working great except I am having issues with pixelserv-tls, it works until it crashes. The virtual interface for pixel server keep crashing periodically. I am not sure when does it actually die, when I run ifconfig, I DON'T see the interface br0: pixelserv-tls in list of interface. My next step is restart the service, so i run "S80pixelserv-tls stop" and "S80pixelserv-tls start" from /tmp/mnt/entware/entware/etc/init.d directory. After this it operates as usual for couple hours and I can see the servstats page. After some hours the interface mysteriously disappears from ifconfig and query to http://VirtualIP/servstats fails but the process itself keeps running.

What could be a possible reason for failure? Is there a way to see logs of when the interface crashes?

Also, I noticed that the netmask for virtual interface is different than for my router, I use /24 whereas the virtual IP uses /8.

 

[thelonelycoder]

I am running Diversion 4.0 after fresh install and configuration of Merlin 384.6 running on RT-AC68U.

So far, things are working great except I am having issues with pixelserv-tls, it works until it crashes. The virtual interface for pixel server keep crashing periodically. I am not sure when does it actually die, when I run ifconfig, I DON'T see the interface br0: pixelserv-tls in list of interface. My next step is restart the service, so i run "S80pixelserv-tls stop" and "S80pixelserv-tls start" from /tmp/mnt/entware/entware/etc/init.d directory. After this it operates as usual for couple hours and I can see the servstats page. After some hours the interface mysteriously disappears from ifconfig and query to http://VirtualIP/servstats fails but the process itself keeps running.

What could be a possible reason for failure? Is there a way to see logs of when the interface crashes?

Also, I noticed that the netmask for virtual interface is different than for my router, I use /24 whereas the virtual IP uses /8.

Option 4 in ep restarts pixelserv-tls...
And since Entware is installed, you could use the /opt path:

/opt/etc/init.d/S80pixelserv-tls stop

And lastly: Any Syslog entries of interest?