uiDivStats uiDivstats 4.0.3 - WebUI Stats for Diversion, November 19, 2024

[Daveo]

Updated as follows...

• amtm to 4.5
• Diversion to 5.1.2
• j5u to switch to thelonelycoder's version of uiDivStats
• uiDivStats to v4.0.0

...and now I see...

The current release of uiDivStats (4.0.0) is
no longer compatible with Diversion 5.1.2.

Watch out for an update with u in amtm,
it will show when a new version is available.

Looks like uiDivStats v4.0.0 is NOT compatible with Diversion v5.1.2!

 

[Ripshod]

They're working together just fine. I'm thinking something is triggering this warning, possibly a false positive on some test.
My problem is different, is not causing any real problems, and appears to just be a gui glitch.
Thinking about what I've just written it does seem both problems may just be glitches in amtm.

 

[thecheapseats]

if/when changing uiDivStats from jackyaz's 'curl' (3.1.0)
to decoderman's 'curl' (4.0.0) - would best practices be to remove v3 (via the GUI) before running the new v4 command line 'curl'?...

asking for a friend...

 

[L&LD]

Always. Along with a router reboot in between.

 

[thecheapseats]

if/when changing uiDivStats from jackyaz's 'curl' (3.1.0)
to decoderman's 'curl' (4.0.0) - would best practices be to remove v3 (via the GUI) before running the new v4 command line 'curl'?...

additionally - is there any CLI filesystem housekeeping that might be necessary?...

 

[myStar]

Had troubles at the installation too.

amtm is 4.5 and diversion is 5.2.1.

Finally i used the command from uiDivStats repo.

/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/decoderman/uiDivStats/master/uiDivStats.sh" -o "/jffs/scripts/uiDivStats" && chmod 0755 /jffs/scripts/uiDivStats && /jffs/scripts/uiDivStats install

That worked flawlessly, and the "old" (not anymore supported) version is gone.

Thanks for the work!

 

[Quoc Huynh]

@thelonelycoder I think there is a minor typo in the WebUI, regarding the Diversion Statistics Report section. The weekly stats function is in menu option c 1, rather than c 2.

 

[heslo]

Uninstalled the old version, updated AMTM, Diversion, restarted the router then installed UIDivStats again and worked a treat; thanks mate!

 

[Aziron5]

Hi

Seems like UiDivStats and taildns are spamming:

waiting for NTP to sync ...

messages in the System Log, multiple times per minute, if WAN is not connected.

I think that's a bit too often, otherwise not that of a big deal.

 

[bjark]

There seems to a problem with uidivstats trimdb (set to run at 00:01 every night by default) severely disrupting the router for a long time (consistently around one hour for me) for some users. In this other thread are reports by me and one more. Is there anything I can do to help see if this is a bug with the code or a problem with my setup? Or should I just take the easy way out and delete the db (now and then?)?

(tested on uiDivStats v4.0.1 and v4.0.2)

 

[Martinski]

Release Notes for uiDivStats 4.0.2 version (now available):

1) Redirected version update requests from the 'develop' branch to the 'master' branch.

The 'develop' branch is *not* supported/available on the repository.​

2) Added a new menu option to allow users to set a preferred time for the daily cron job that trims the database.

3) Various code improvements & fine-tuning.

The new option 6) is available in the SSH CLI menu:

This allows users to set a preferred time when the trim database process would not affect internet activity (e.g. scheduled at 3:00 AM rather than the current default 12:01 AM).

Modified time:

 

[bjark]

Release Notes for uiDivStats 4.0.2 version (now available):

1) Redirected version update requests from the 'develop' branch to the 'master' branch.

The 'develop' branch is *not* supported/available on the repository.​

2) Added a new menu option to allow users to set a preferred time for the daily cron job that trims the database.

3) Various code improvements & fine-tuning.

The new option 6) is available in the SSH CLI menu:

View attachment 62034

This allows users to set a preferred time when the trim database process would not affect internet activity (e.g. scheduled at 3:00 AM rather than the current default 12:01 AM).

View attachment 62035

Modified time:

View attachment 62036

Are there some other recommended practices to avoid issues with trimdb holding up the router for a long time? I see that the oldest records in my db does not seem to match the setting for how many days to be kept (30 days vs Sep 3), what could be the reason for this?

 

[Martinski]

Are there some other recommended practices to avoid issues with trimdb holding up the router for a long time? I see that the oldest records in my db does not seem to match the setting for how many days to be kept (30 days vs Sep 3), what could be the reason for this?

I haven't reviewed the code that trims the database, but I suppose it's possible that there's a bug somewhere there that causes the trimming task to fail. I don't use Diversion/uiDivStats so I can't really tell you if there are best practices WRT keeping the database at a reasonable size. Your current database at 2.8GB is likely contributing to the total time it takes to process it.

If I get some free time next weekend I can take a look at the code.

 

[Ripshod]

@Martinski I've made a note of a similar observation in amongst one of the threads re UiDivStats. Don't ask me where, I didn't bookmark it. At that time it seemed to me the database wasn't being trimmed and I made the assumption that it had gotten too large to be loaded/handled correctly. Could this still be possible? Can a debug log be generated by this script?

 

[Martinski]

@Martinski I've made a note of a similar observation in amongst one of the threads re UiDivStats. Don't ask me where, I didn't bookmark it. At that time it seemed to me the database wasn't being trimmed and I made the assumption that it had gotten too large to be loaded/handled correctly. Could this still be possible? Can a debug log be generated by this script?

Apologies for the delayed response. I was busy all Sunday with family/personal errands and then visiting relatives for a birthday celebration.

Anyway, WRT your question; yeah, it's possible that the SQLite3 version installed via Entware does not handle very large databases (e.g. over 1GB) as well as the standard version does since it's a "light" version that's meant to run on embedded systems which are assumed to have limited resources.

I briefly looked at the current code, and nothing appeared suspicious. So I decided to write a small shell script to gather some diagnostics data while attempting to trim the database using essentially the same code.

Are there some other recommended practices to avoid issues with trimdb holding up the router for a long time? I see that the oldest records in my db does not seem to match the setting for how many days to be kept (30 days vs Sep 3), what could be the reason for this?

Would you both mind running this script on your current uiDivStats database and providing the results?

If OK, I'll provide details via a DM.

 

[aru]

It is recommended to refer to the Skype addon package and add database or log file size information in uiDivStats to facilitate easier monitoring of whether the database becomes abnormally large after pruning.

 

[Martinski]

Release Notes for uiDivStats 4.0.3 version (now available):

1) Modified SQLite3 configuration parameters to improve the trimming of records from the database and then perform "garbage collection" of deleted entries to reclaim unused space & avoid excessive fragmentation.

This is an effort to prevent cases in which, after some weeks of operation, the database file size becomes increasingly & unnecessarily very large (>> 2GB) causing long processing times that interfere with & disrupt normal internet activity.

2) To help with future debugging & monitoring, a log file is now created during the record trimming process to keep ~60 days of processing results (assuming once per day):

/opt/share/uiDivStats.d/uiDivStats_Trim.LOG

3) Modified SQLite3 configuration parameters to improve processing database records.

4) The previously added CLI menu option 6) to set a preferred time for the daily cron job that trims the database was modified to set ONLY the HOUR parameter. This was done because the scheduled minutes must be coordinated with 3 other cron jobs to avoid possible conflicts while accessing & generating database records.

5) Show the current database file size information on the CLI menu and the webGUI page (see screenshots below).

6) Miscellaneous code improvements & fine-tuning.

I'd like to give special thanks to @Ripshod & @bjark for their generous donation of their time & patience when running a custom diagnostics shell script on their routers to gather SQLite3 configuration data while trimming records from their very large databases to troubleshoot the problem & narrow down a solution. Their collaboration is highly appreciated.

BTW, both @Ripshod & @bjark have been running the Beta 4.0.3 version for about 3 weeks now and have not reported any issues so far.

Screenshots:

 

[aru]

When using uiDivStats for an extended period, once the number of records in the database reaches a certain threshold, the statistics charts either appear blank or display "no data to display." This issue occurs regardless of whether I use Chrome or Edge browsers.

However, if I clear the database via the command line and restart the recording, then click the "Update stats" button, the charts display correctly again. I’m unsure what is causing this issue.

#!/bin/bash

echo "Before size..."
ls -hl /opt/share/uiDivStats.d/dnsqueries.db

echo "Trimming dnsqueries.db..."

# Database file path
database_path="/opt/share/uiDivStats.d/dnsqueries.db"

# Table name column
table_name="dnsqueries"

# SQLite3 commands
sqlite3 "$database_path" <<EOF

-- Get the row count before deletion
.headers on
.mode column
SELECT 'Rows before deletion:', COUNT(*) FROM $table_name;

-- Delete data
DELETE FROM $table_name;

-- Perform VACUUM (optional)
VACUUM;

-- Get the row count after deletion
SELECT 'Rows after deletion:', COUNT(*) FROM $table_name;

-- Exit SQLite3
.exit
EOF

echo "Done!"

echo "After size..."
ls -hl /opt/share/uiDivStats.d/dnsqueries.db

 

[Martinski]

When using uiDivStats for an extended period, once the number of records in the database reaches a certain threshold, the statistics charts either appear blank or display "no data to display." This issue occurs regardless of whether I use Chrome or Edge browsers.
...

For the love of God, couldn’t you have made the screenshots even smaller and, therefore, much less readable & useful?

You provided very little context (e.g. "… for an extended period, …") and without actual details (e.g. "… reaches a certain threshold, …") when describing, in vague terms, the problem you’re seeing.

You will need to post a better description that includes as much relevant context & details as possible, with clear & readable screenshots. Also, check if there are any errors reported in the "Browser Console" window when the "No data to display" issue is occurring. Without more details, context & readable screenshots, there are not enough clues to know where to begin looking.