uiDivStats uiDivStats - WebUI for Diversion statistics

[joe scian]

Would leaving the cron intact work, but only pull out the data if there is a login? It'd still mean a task every minute, but with no logins it would be a tiny logic/nvram evaluation

yes it was done by me - at the time I had thumb in bum and mind in neutral - mea culpa !!

 

[Jack Yaz]

Just an idea to throw out there for future consideration/complications: the log generation runs every minute of every day in cron, for obvious reasons. Maybe update the invoked function to exit immediately if no one is logged into the router GUI (nvram login_timestamp is empty). You could still let it run it every nth time to make sure the data isn’t too stale when a user finally logs in.

login_timestamp isn't cleared if the user simply closes the tab (tested by waiting 10+mins). Returning to GUI shows login prompt, but timestamp is still set.

 

[dave14305]

Would leaving the cron intact work, but only pull out the data if there is a login? It'd still mean a task every minute, but with no logins it would be a tiny logic/nvram evaluation

Yes, exactly. Still run every minute, but do nothing if no one is logged in. If no one is logged in but the minute is 00,15,30,45 run it anyway so the log data is relatively recent at all times. 15 minutes is arbitrary. Or make it part of the hourly stats generation.

 

[dave14305]

login_timestamp isn't cleared if the user simply closes the tab (tested by waiting 10+mins). Returning to GUI shows login prompt, but timestamp is still set.

Ok, maybe I’m remembering different httpd behavior specific to John’s fork where it would clear those after a timeout.

 

[JT Strickland]

Mine appears to be working as expected after the last fix and a reboot. Everything else seems in harmony too. In my router, at least, but that's about it. The rest of the US ain't, it don't look like.
Thanks again,
jts

RT-AC86U w/ 384.17, RT-AC68U Aimesh node w/ same, 250/10 cable w/ Netgear CM-1000, Diversion, UiDivstats, Skynet, AiProtection, DoT, Scribe, UiScribe, Connmon, SpdMerlin, ScMerlin, Nsrum, NtpMerlin

 

[Jack Yaz]

Ok, maybe I’m remembering different httpd behavior specific to John’s fork where it would clear those after a timeout.

I'm testing out a few options

 

[andresmorago]

@Jack Yaz thanks so much for the update on the script. It looks fantastic!

I wanted to ask you if you could consider the possibility of showing the hostname along with the ip that is already shown, when expanding the client list on the charts. Sometimes just seeing the IP address makes difficult to identify which device to filter by.

 

[Jack Yaz]

For anyone interested, I have some changes on the develop branch I'd like feedback on please!

Changelog:

NEW: Hostnames are shown along with the IP Address in the client dropdown. These are determined by using dig for reverse DNS lookups
CHANGED: "Last 24 hours" now explicitly collects data for all 15 minute periods. The previous Group By would exclude any points where no queried/blocked domains were found in a 15 minute period, which would leave gaps in data plots.
CHANGED: Lengthy domain names on the Y-Axis are now split to multiple lines

On my AC86U, points 1 and 2 have extended the hourly stats generation from <5s to <30s.

If you would like to test, run

uiDivStats develop

and use option 1 to generate the updated files.

 

[LimJK]

For anyone interested, I have some changes on the develop branch I'd like feedback on please!

Changelog:

NEW: Hostnames are shown along with the IP Address in the client dropdown. These are determined by using dig for reverse DNS lookups
CHANGED: "Last 24 hours" now explicitly collects data for all 15 minute periods. The previous Group By would exclude any points where no queried/blocked domains were found in a 15 minute period, which would leave gaps in data plots.
CHANGED: Lengthy domain names on the Y-Axis are now split to multiple lines

On my AC86U, points 1 and 2 have extended the hourly stats generation from <5s to <30s.

If you would like to test, run

uiDivStats develop

and use option 1 to generate the updated files.

Jack,
The DNS Queries Chart is displaying correctly for 2.0.1. Thank you.

PS: Confirm the following is working as advertised

• Hostnames are shown along with the IP Address in the client dropdown
• Lengthy domain names on the Y-Axis are now split to multiple lines

 

[LimJK]

Jack,

A small point / suggestion, Can you consider to make the "Type" column, drop down selection a little wider. Thanks.
PS: I am using Safari on macOS Catalina.

 

[HorseCalledHorse]

Yes, it solved my problems.
Big thanks for your help!

Still having issues here, CPUs spiking, no stats being generated after multiple installs, router restarts, etc. I'm still seeing version 2.0.0 when I install. Is there a later version with a fix?

 

[AntonK]

Still having issues here, CPUs spiking, no stats being generated after multiple installs, router restarts, etc. I'm still seeing version 2.0.0 when I install. Is there a later version with a fix?

This was my problem. I ended up with both the Division tab, and the uiDivstats tab. But both were actually just the Division tab. No stats generated, high cpu usage, and multiple reboots did not fix it. I'd been using Division and uiDivstats for months prior to this without issue. I'm not linux-savvy enough to start running command line operations to troubleshoot the issue, so no need request I do so. Not complaining, just reporting what happened here. I've uninstalled for now, and will be happy to try again when a stable release comes out that can be installed with confidence.

Thanks,
Anton

 

[andresmorago]

For anyone interested, I have some changes on the develop branch I'd like feedback on please!

Changelog:

NEW: Hostnames are shown along with the IP Address in the client dropdown. These are determined by using dig for reverse DNS lookups
CHANGED: "Last 24 hours" now explicitly collects data for all 15 minute periods. The previous Group By would exclude any points where no queried/blocked domains were found in a 15 minute period, which would leave gaps in data plots.
CHANGED: Lengthy domain names on the Y-Axis are now split to multiple lines

On my AC86U, points 1 and 2 have extended the hourly stats generation from <5s to <30s.

If you would like to test, run

uiDivStats develop

and use option 1 to generate the updated files.

Working great for me! Thanks

 

[JT Strickland]

This was my problem. I ended up with both the Division tab, and the uiDivstats tab. But both were actually just the Division tab. No stats generated, high cpu usage, and multiple reboots did not fix it. I'd been using Division and uiDivstats for months prior to this without issue. I'm not linux-savvy enough to start running command line operations to troubleshoot the issue, so no need request I do so. Not complaining, just reporting what happened here. I've uninstalled for now, and will be happy to try again when a stable release comes out that can be installed with confidence.

Thanks,
Anton

I tried upgrading, and amtm said mine was 2.0.0 too. I forced an update, and amtm still said it was 2.0.0, but when I rebooted, the tab was there, with no data. Later the data shows up in a couple hours. I'm not linux savvy, either, barely can carry on a conversation with you guys, but keep trying is all the advice I can give. It worked for me.
BTW, it would be neat to see the host names along side the IP's in Skynet, but I'm not complaining.

Edit: I may have spoken out of turn there, it isn't going to update from the develop channel through amtm, I don't suppose. Mine is working fine with 2.0.1 so far, and I am grateful. I haven't seen the spikes with the cpu's like some 0f you have either on my RT-AC86U. But it seems like it's downloading a lot and that's spiked out. My ram is 93-94% but it has been there a while now.

 

[Jack Yaz]

Still having issues here, CPUs spiking, no stats being generated after multiple installs, router restarts, etc. I'm still seeing version 2.0.0 when I install. Is there a later version with a fix?

Did you try deleting the database and reinstalling as suggested earlier? Re cpu spikes, what does top/htop show?

 

[Jack Yaz]

This was my problem. I ended up with both the Division tab, and the uiDivstats tab. But both were actually just the Division tab. No stats generated, high cpu usage, and multiple reboots did not fix it. I'd been using Division and uiDivstats for months prior to this without issue. I'm not linux-savvy enough to start running command line operations to troubleshoot the issue, so no need request I do so. Not complaining, just reporting what happened here. I've uninstalled for now, and will be happy to try again when a stable release comes out that can be installed with confidence.

Thanks,
Anton

Please try https://www.snbforums.com/threads/u...iversion-statistics.56393/page-37#post-591561

 

[HorseCalledHorse]

Did you try deleting the database and reinstalling as suggested earlier? Re cpu spikes, what does top/htop show?

Yes, I ran the uninstall command and deleted the database each time, following these commands:

uiDivStats uninstall
rm /opt/share/uiDivStats.d/dnsqueries.db

Made no difference.
I did run top but didn't capture the output, primarily because it didn't appear to show anything untoward. If I try it again and hit the same problem I'll run top. Any particular switches you'd like me to run?

 

[JemTheWire]

I know I sound like I am copying you, but I am still having the CPU spike issue.

Each one of the four cores will max out randomly but constantly. If I uninstall uiDivStats v2, it stops straight away. When I reinstall uiDivStats, it starts again straight away.

I have uninstalled from the command like you and also use ran the rm command as noted in an earlier post to remove the db.

Interestingly, after removing the db and then immediately installing uiDivStats again, the tabs obviously state ‘no data to display’. While it does this the CPU cores are NOT spiking.

However, once the stats are populated, CPU cores spike again.

I have run TOP while the spiking is occurring but the CPU usage is normal and does not follow the graph in the UI. Could this be because TOP takes an average of all four cores and as it’s only one core at a time that spikes, the average load is much lower?

 

[Jack Yaz]

Yes, I ran the uninstall command and deleted the database each time, following these commands:

uiDivStats uninstall
rm /opt/share/uiDivStats.d/dnsqueries.db

Made no difference.
I did run top but didn't capture the output, primarily because it didn't appear to show anything untoward. If I try it again and hit the same problem I'll run top. Any particular switches you'd like me to run?

If you run top and press P to sort by CPU usage it should capture whatever generates the spikes

 

[Jack Yaz]

Yes, I ran the uninstall command and deleted the database each time, following these commands:

uiDivStats uninstall
rm /opt/share/uiDivStats.d/dnsqueries.db

Made no difference.
I did run top but didn't capture the output, primarily because it didn't appear to show anything untoward. If I try it again and hit the same problem I'll run top. Any particular switches you'd like me to run?

Also, does the reinstall complete successfully? Do you get any errors if you run option 1?

Some things to also check:
https://www.snbforums.com/threads/u...iversion-statistics.56393/page-34#post-591352