uiDivStats uiDivStats 3.x

[Jack Yaz]

v3.0.2
Updated 2021-12-20

A graphical representation of domain blocking performed by Diversion.

uiDivStats is free to use under the GNU General Public License version 3 (GPL 3.0).

This project is hosted on GitHub

Love the script and want to support future development? Any and all donations gratefully received!
PayPal donation
Buy me a coffee

Supported firmware versions
You must be running firmware Merlin 384.15/384.13_4 or Fork 43E5 (or later) Asuswrt-Merlin

Installation
Using your preferred SSH client/terminal, copy and paste the following command, then press Enter:

/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/jackyaz/uiDivStats/master/uiDivStats.sh" -o "/jffs/scripts/uiDivStats" && chmod 0755 /jffs/scripts/uiDivStats && /jffs/scripts/uiDivStats install

Usage
To launch the uiDivStats menu after installation, use:

uiDivStats

If this does not work, you will need to use the full path:

/jffs/scripts/uiDivStats

Updating
Launch uiDivStats and select option u

 

[Jack Yaz]

Screenshots:

Spoiler: WebUI

Spoiler: CLI

 

[Jack Yaz]

Please note that at midnight (1 past) uiDivStats will perform maintenance on the database. The first time this runs after upgrading from uiDivStats 2.x involves creating new database indexes to increase performance, but can take a long time. My database is roughly 1GB in size and my RT-AC86u took an hour to complete index creation. During this time uiDivStats will stop logging DNS queries so you will have a gap in your data, but this ensures the maintenance can complete as quickly as possible.

Future maintenance jobs will run at the same time every night but are much quicker (<5 minutes, as conditions as above).

 

[SomeWhereOverTheRainBow]

uiDivStats 3.0.0 will be dropping tomorrow (Monday 7 June)

Please note that at midnight (1 past) uiDivStats will perform maintenance on the database. The first time this runs after upgrading from uiDivStats 2.x involves creating new database indexes to increase performance, but can take a long time. My database is roughly 1GB in size and my RT-AC86u took an hour to complete index creation. During this time uiDivStats will stop logging DNS queries so you will have a gap in your data, but this ensures the maintenance can complete as quickly as possible.

Future maintenance jobs will run at the same time every night but are much quicker (<5 minutes, as conditions as above).

Great job, Jack!. keep on chuggin' along!

 

[QuikSilver]

awesome! can't wait!

 

[Jack Yaz]

v3.0.0 is now available

PLEASE NOTE THE INFORMATION HERE: http://www.snbforums.com/threads/uidivstats-3-x.72966/post-693164

Changelog:

• NEW: Overnight database maintenance to ensure query performance is fast
• NEW: Configure how long data is kept in the database
• NEW: Configure how many recent results are displayed in the WebUI
• NEW: Option to refresh query log in WebUI on demand
• NEW: CLI menu shows URL for WebUI page
• NEW: CLI commands for "about" and "help"
• NEW: Add option to reset database
• IMPROVED: Statistic generation is now much faster due to improved use of database indexes and optimised queries. This will take effect after first maintenance run at midnight
• IMPROVED: CPU intensive tasks are now run with a lower priority to minimise hogging the CPU
• IMPROVED: Overnight database maintenance will remove invalid entries such as those with a timestamp in the future
• IMPROVED: Hostname matching for client IP addresses
• IMPROVED: Update function now includes a prompt rather than applying update
• IMPROVED: Use colours in CLI menu to highlight settings
• IMPROVED: Use of keyboard keys r,f for chart functions (reset zoom, toggle fill)
• IMPROVED: Use ajax to load dependent files in WebUI to avoid complete page load failures if a file was unavailable
• IMPROVED: Stale processes will be cleared as necessary
• CHANGED: Use of cache/tmpfs for query logging is now the default in place of direct write to disk
• CHANGED: taildns will wait for NTP to sync before starting, with a maximum wait of 10 minutes
• CHANGED: Increase NTP timeout from 5 minutes to 10 minutes for core script

 

[Jack Yaz]

Has anyone run into problems following this upgrade?

 

[QuikSilver]

Has anyone run into problems following this upgrade?

Unless I'm overlooking something mine seems to be perfectly fine. If it is then it must be a minor issue.

 

[L&LD]

None that I've seen.

 

[visortgw]

No issues here, either.

 

[Kingp1n]

I updated recently and I haven't seen any issues.

 

[Jack Yaz]

My main concerns were if anything hung during the first nightly maintenance as that will take ages to re-do the indexes, but then after that you should see 10-20s for normal generation and sub 2 minutes for full refreshes

 

[jsbeddow]

My main concerns were if anything hung during the first nightly maintenance as that will take ages to re-do the indexes, but then after that you should see 10-20s for normal generation and sub 2 minutes for full refreshes

True, the initial evening when this version was installed (through the GUI this time too!), I lost all the graphs/stats, but everything was back in place by the next morning, so I didn't bother mentioning it before and figured it was expected, as you had told us to expect one long maintenance session initially.

 

[Jack Yaz]

True, the initial evening when this version was installed (through the GUI this time too!), I lost all the graphs/stats, but everything was back in place by the next morning, so I didn't bother mentioning it before and figured it was expected, as you had told us to expect one long maintenance session initially.

it should all be worth it. i spent a significant chunk of time learning about SQL performance improvements and analysing the queries uiDivStats uses to ensure they were all covered by an index to make them quick, as well as remove multiple commands when 1 would do, just needed some advanced syntax

 

[SomeWhereOverTheRainBow]

v3.0.0 is now available

PLEASE NOTE THE INFORMATION HERE: http://www.snbforums.com/threads/uidivstats-3-x.72966/post-693164

Changelog:

• NEW: Overnight database maintenance to ensure query performance is fast
• NEW: Configure how long data is kept in the database
• NEW: Configure how many recent results are displayed in the WebUI
• NEW: Option to refresh query log in WebUI on demand
• NEW: CLI menu shows URL for WebUI page
• NEW: CLI commands for "about" and "help"
• NEW: Add option to reset database
• IMPROVED: Statistic generation is now much faster due to improved use of database indexes and optimised queries. This will take effect after first maintenance run at midnight
• IMPROVED: CPU intensive tasks are now run with a lower priority to minimise hogging the CPU
• IMPROVED: Overnight database maintenance will remove invalid entries such as those with a timestamp in the future
• IMPROVED: Hostname matching for client IP addresses
• IMPROVED: Update function now includes a prompt rather than applying update
• IMPROVED: Use colours in CLI menu to highlight settings
• IMPROVED: Use of keyboard keys r,f for chart functions (reset zoom, toggle fill)
• IMPROVED: Use ajax to load dependent files in WebUI to avoid complete page load failures if a file was unavailable
• IMPROVED: Stale processes will be cleared as necessary
• CHANGED: Use of cache/tmpfs for query logging is now the default in place of direct write to disk
• CHANGED: taildns will wait for NTP to sync before starting, with a maximum wait of 10 minutes
• CHANGED: Increase NTP timeout from 5 minutes to 10 minutes for core script

HE IS BACK!<3 who is back? JACK IS BACK!

 

[umrico]

My main concerns were if anything hung during the first nightly maintenance as that will take ages to re-do the indexes, but then after that you should see 10-20s for normal generation and sub 2 minutes for full refreshes

Good morning! Thanks so much for this! Just wanted to ask/provide feedback. Was the tmpfs supposed to be enabled automatically upon update/first maintenance mode? Mine was still off a couple days later.

I just also saw there was a hotfix this morning. I updated to that but tmpfs was still off (however it hasn't got to tonight's maintenance mode yet).

In any case i just enabled it. Is that for the better? I have an AX-86U.

Thank you again!

 

[jrichard326]

Is there a way to change the time of daily maintenance?

 

[heslo]

Just updated. Haven't gone through the maintenance part yet but that will be tonight. So far so good though

 

[Jack Yaz]

Is there a way to change the time of daily maintenance?

if you're comfortable editing the script and remembering any time you update it will be overwritten, yes

 

[Jack Yaz]

Good morning! Thanks so much for this! Just wanted to ask/provide feedback. Was the tmpfs supposed to be enabled automatically upon update/first maintenance mode? Mine was still off a couple days later.

I just also saw there was a hotfix this morning. I updated to that but tmpfs was still off (however it hasn't got to tonight's maintenance mode yet).

In any case i just enabled it. Is that for the better? I have an AX-86U.

Thank you again!

tmpfs is a new default but won't be imposed on existing installs. if you have a super fast USB/SSD connected then by all means use no cache, but from testing I've found tmpfs provides less hangups on the router