uiDivStats uiDivStats - WebUI for Diversion statistics

[HorseCalledHorse]

Odd behaviour on an upgrade install here. Did it through AMTM, took less than a second, which seemed odd, but I left it for a half-hour or so and came back to find no stat generation and the router (86U) CPUs spiking. Opened AMTM, hit j5 to go to uiDivStats (which AMTM said was v2.0.0), which appeared to trigger the update process. Left it alone again but still no updating of stats and the CPUs were still spiking so I uninstalled and reinstalled and nothing has changed: no stat generation, CPUs spiking. I'm going to be away for the rest of the weekend so I've just uninstalled for now. Sorry I can't be more helpful.

 

[HorseCalledHorse]

I've uninstalled Diversion and uiDivDtats, rebooted the router (twice), but their two tabs remain under LAN. Is there something else I need to do, or do they just stay there?

From my reading, that's where they're both meant to be now, Anton.

 

[QuikSilver]

From my reading, that's where they're both meant to be now, Anton.

Yes they are supposed to be in the Lan section and will not be changed.

@RMerlin specifically added this to the API to add a page anywhere, only @john9527 interpreted it differently.
No matter what the opinions, I stay firm along with @Adamm that the tab goes where it belongs.
End of discussion for me about this topic, it is moot. I am the coder and made the decision based on logic and my preferences.

 

[AntonK]

Yes they are supposed to be in the Lan section and will not be changed.

Thanks. Just double checking. You mean they stay there after they are uninstalled?

 

[heslo]

Odd behaviour on an upgrade install here. Did it through AMTM, took less than a second, which seemed odd, but I left it for a half-hour or so and came back to find no stat generation and the router (86U) CPUs spiking. Opened AMTM, hit j5 to go to uiDivStats (which AMTM said was v2.0.0), which appeared to trigger the update process. Left it alone again but still no updating of stats and the CPUs were still spiking so I uninstalled and reinstalled and nothing has changed: no stat generation, CPUs spiking. I'm going to be away for the rest of the weekend so I've just uninstalled for now. Sorry I can't be more helpful.

I had similar issues... try this

uiDivStats uninstall
rm /opt/share/uiDivStats.d/dnsqueries.db

Then re-install and see how it goes for you

 

[thelonelycoder]

I've uninstalled Diversion and uiDivDtats, rebooted the router (twice), but their two tabs remain under LAN. Is there something else I need to do, or do they just stay there?

Thanks. Just double checking. You mean they stay there after they are uninstalled?

For a custom page to be mounted, three things MUST happen while booting the router:
1. The app must be installed (uiDivstats or Diversion, ...)
2. The page Name gets added to the menu list
3. The page file(s) are copied over to the volatile /www/user/ directory

None of that can happen if you had Diversion uninstalled through its uninstall function.

 

[LimJK]

v2.0.0 is now available!
Changelog:

Complete re-write! uiDivStats now pushes dnsmasq log entries to SQLite in realtime via the "taildns" script. All graphical data is then generated against this database. The database will keep the last 30 days worth of requests and is typically a minimum of 200-300MB with 30 days' worth of data. The database is stored on an attached USB, in /opt/share/uiDivStats.d

The tab has been relocated from Addons to LAN - it may stay in Addons until you reboot, however.

Data is now displayed for:
- Last 24 hours
- Last 7 days
- Last 30 days
and is available per-client.

Data for "Last 24 hours" is refreshed hourly. Data for "Last 7 days" and "Last 30 days" is refreshed daily, at midnight.

A query log is now available in the WebUI, for ease of checking if a domain is blocked (refreshes every 60 seconds)

A big thanks to @thelonelycoder @dave14305 @QuikSilver @SomeWhereOverTheRainBow @visortgw @heslo @brianward317 for the massive help in the testing effort! Apologies to anyone I've missed!

Hi Jack,
Thanks. I noticed that in the graph, when there is zero blocked DNS Query, it is not shown as zero

 

[underdose]

Thanks. Just double checking. You mean they stay there after they are uninstalled?

I had a similar issue on my N66U, and solved it by removing user files belonging to that already uninstalled script manually by

rm -rf /tmp/var/wwwext/user4*

where 4 at the end of user4 comes from the uninstalled script.

 

[joe scian]

If you have installed Unbound and are using Unbound as Primary DNS rather than Dnsmasq ( ie Dnsmasq is disabled) this is what you get when installing UIDivStats. Make sure you make Dnsmasq your DNS resolver in Unbound ( ie enable Dnsmasq) before attempting UIDivStats install.

Author: Jack Yaz
 https://www.snbforums.com/threads/uidivstats-webui-for-diversion-statistics.56393/
_____________________________________________

 Continue? [1=Yes e=Exit] 1

uiDivStats: Welcome to uiDivStats v2.0.0, a script by JackYaz

uiDivStats: Checking your router meets the requirements for uiDivStats

Downloading http://bin.entware.net/armv7sf-k2.6/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Package grep (3.4-1) installed in root is up to date.
Package sqlite3-cli (3310100-1) installed in root is up to date.
Package procps-ng-pkill (3.3.15-4) installed in root is up to date.
uiDivStats: New version of uidivstats_www.asp downloaded

uiDivStats: Mounting uiDivStats WebUI page as user1.asp

uiDivStats: New version of taildns.tar.gz downloaded

Downloading http://bin.entware.net/armv7sf-k2.6/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Package grep (3.4-1) installed in root is up to date.
Package sqlite3-cli (3310100-1) installed in root is up to date.
Package procps-ng-pkill (3.3.15-4) installed in root is up to date.
uiDivStats: Creating database table and enabling write-ahead logging...

uiDivStats: Creating database table indexes...

uiDivStats: Database ready, starting services...

uiDivStats: Starting first run of stat generation...

awk: cmd. line:1: Division by zero
awk: cmd. line:1: Division by zero
awk: cmd. line:1: Division by zero
uiDivStats: Stats updated successfully

 

[Jack Yaz]

Hi Jack,
Thanks. I noticed that in the graph, when there is zero blocked DNS Query, it is not shown as zero

View attachment 23955

Looks like group by won't return a "group" if there are no matches for blocked. Doesn't look like there is a particular easy way around this in SQLite.

 

[Jack Yaz]

Looks like group by won't return a "group" if there are no matches for blocked. Doesn't look like there is a particular easy way around this in SQLite.

@LimJK How much of an issue is this? The workaround I have so far is to revert to previous behaviour for Last 24 hours, and generate a query for each 15 minute block rather than rely on group by. This has an inherent performance hit, so I am proposing to make this change to Last 24 only. Weekly and Monthly I would find it hard to believe that you get 0 blocked domains over the aggregate window of group by!

 

[Dan Ljungberg]

I've just pushed an update that may well resolve the issue. While fixing another issue I (and @dave14305) spotted I'd goofed a positional argument in the "extra" version of the log parsing script

Yes, it solved my problems.
Big thanks for your help!

 

[Jack Yaz]

Yes, it solved my problems.
Big thanks for your help!

Sorry it took so long. I assumed that the awk script for "extra" query logging was working fine

 

[Jack Yaz]

Love your work @Jack Yaz

Wondering if it would be possible for the "Client To Display" selection for Top Blocked and Top Requested to show friendly/host names rather than IP addresses, rather like what happens in Network Tools | Wake On LAN when you go to add an entry, shows active and even inactive clients. Don't know if you can access that data somehow to use in uiDivStats?

Likewise, for the Query Log filtering, could the Client field of the filter similarly be a dropdown list of hostnames rather than having to enter an IP perhaps? Maybe keep a separate IP field as well so it goes Domain | IP | Client | Type | Result ?

What do others think?

Depends on the most trustworthy source of the "name" - something entered in NetworkMap, the hostname returned by the device itself, a name set in a DHCP reservation?

 

[Dan Ljungberg]

Sorry it took so long. I assumed that the awk script for "extra" query logging was working fine

I didn't think it took so long.
You had solved my problem after about 8 hours and I think that is very fast

 

[LimJK]

@LimJK How much of an issue is this? The workaround I have so far is to revert to previous behaviour for Last 24 hours, and generate a query for each 15 minute block rather than rely on group by. This has an inherent performance hit, so I am proposing to make this change to Last 24 only. Weekly and Monthly I would find it hard to believe that you get 0 blocked domains over the aggregate window of group by!

Jack,
I am actually very happy with informative DNS Queries Chart as is

• If I am able to choose, I would prioritise "performance" the reason of your re-write, over the little display issue.
• I can make visual adjustment when looking at the graph

Thank you so much for all your work to make Version 2.0.0 available to us.

 

[dave14305]

If you have installed Unbound and are using Unbound as Primary DNS rather than Dnsmasq ( ie Dnsmasq is disabled) this is what you get when installing UIDivStats. Make sure you make Dnsmasq your DNS resolver in Unbound ( ie enable Dnsmasq) before attempting UIDivStats install.

Why would someone who deliberately disabled dnsmasq (and therefore Diversion) need to install uiDivStats?

 

[JemTheWire]

Odd behaviour on an upgrade install here. Did it through AMTM, took less than a second, which seemed odd, but I left it for a half-hour or so and came back to find no stat generation and the router (86U) CPUs spiking. Opened AMTM, hit j5 to go to uiDivStats (which AMTM said was v2.0.0), which appeared to trigger the update process. Left it alone again but still no updating of stats and the CPUs were still spiking so I uninstalled and reinstalled and nothing has changed: no stat generation, CPUs spiking. I'm going to be away for the rest of the weekend so I've just uninstalled for now. Sorry I can't be more helpful.

Yeah. Similar to me with the CPU cores spiking, my stats are working fine though.
I’ve uninstalled/reinstalled but cores still spike.

When I run scMerlin and see memory/CPU usage, it doesn’t show anything out of the ordinary. Most weirded.

 

[dave14305]

Just an idea to throw out there for future consideration/complications: the log generation runs every minute of every day in cron, for obvious reasons. Maybe update the invoked function to exit immediately if no one is logged into the router GUI (nvram login_timestamp is empty). You could still let it run it every nth time to make sure the data isn’t too stale when a user finally logs in.

 

[Jack Yaz]

Just an idea to throw out there for future consideration/complications: the log generation runs every minute of every day in cron, for obvious reasons. Maybe update the invoked function to exit immediately if no one is logged into the router GUI (nvram login_timestamp is empty). You could still let it run it every nth time to make sure the data isn’t too stale when a user finally logs in.

Would leaving the cron intact work, but only pull out the data if there is a login? It'd still mean a task every minute, but with no logins it would be a tiny logic/nvram evaluation