uiDivStats uiDivstats 4.0.6 - WebUI Stats for Diversion, December 23, 2024

[agbommarito]

When I said "tag" I was referring to the log level, i.e. the third parameter in the "Print_Output()" function that is used in the script.

Here's a screen shot of the router's system log:
View attachment 62905

Here's a screenshot of the log captured on a remote log server:

View attachment 62906

Note that the log entries are at a "warning" log level.

The router is set to only log entries more urgent than "notice". Since the log entries for stats updates are "tagged" (sorry for the term) with a level of "warning" they appear in the log. That is what prompted my suggestion that the log level be changed to "notice" or "info".

Maybe this behavior is somehow related to the use of a remote log server. I'll try turning that off and seeing if anything changes.

Here's a screenshot of the router log after turning off the remote log server at 07:56:14. The stats update messages still appear.

 

[agbommarito]

Here's a screenshot of the router log after turning off the remote log server at 07:56:14. The stats update messages still appear.

View attachment 62909

I did some more digging into the script. The logger command in the Print_Output() function doesn't use the third parameter that specifies the log level. The logger command must default the log level to "warning", though I couldn't find any documentation that corroborates that assumption. I modified the script to set the log level based on the third parameter. If there isn't a third parameter, the log level is set to "notice".

Here's a screenshot from the log server after script modification:

A third parameter of $PASS now translates to "info". Ignore the two lines that say "notice". They are from when I was debugging the script change.

When the router is set to "Log only messages more urgent than: notice", the "info" lines do not appear.

Here is the script change:

 

[Martinski]

I did some more digging into the script. The logger command in the Print_Output() function doesn't use the third parameter that specifies the log level. The logger command must default the log level to "warning", though I couldn't find any documentation that corroborates that assumption. I modified the script to set the log level based on the third parameter. If there isn't a third parameter, the log level is set to "notice".

Here's a screenshot from the log server after script modification:
...
A third parameter of $PASS now translates to "info". Ignore the two lines that say "notice". They are from when I was debugging the script change.

When the router is set to "Log only messages more urgent than: notice", the "info" lines do not appear.

Here is the script change:
...

Ah, OK. Now I fully understand what you mean. Very good posts with supporting screenshots. Thank you for the clear explanation & clarification. I’ll make the changes in the script function for the next release.

UPDATE: It's done.

 

[Martinski]

Release Notes for uiDivStats 4.0.4 version (now available):

1) Modified SQLite3 calls to capture and log errors in the system log.

2) Added new CLI options to disable & later enable the background processing of Diversion statistics.

/jffs/scripts/uiDivStats disableprocs
/jffs/scripts/uiDivStats enableprocs

This allows users to disable the background cron jobs & daemon process that handle statistical data & database operations. While in this DISABLED state, the script can be manually executed and runs in a "constrained mode." It's recommended to re-enable the background processes as soon as it's conveniently possible.

3) Added the "TMPFS free space" information to the "Cache Mode" option.
This allows users to see if/when TMPFS free space is getting too low to support database operations when the "Cache Mode" option is set to Enabled.

4) Modified code to set the corresponding priority level of log entries when calling the built-in logger utility.
Thanks to @agbommarito for reporting and providing an explanation of the issue.

5) Various code improvements & fine-tuning.

Screenshots:

 

[ColinTaylor]

Hi @Martinski.

I saw your post mentioning tmpfs and it reminded me of this "gotcha". So I thought I'd bring it to your attention in case it's relevant here.

[Diversion] failed to write /var/lib/misc/dnsmasq.leases: No space left on device

I suspect OP has a very busy router and therefore large dnsmasq log files. I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.

www.snbforums.com

 

[Rhialto]

Was worried for a second, I updated from GUI and after the update it was displaying
Database Size: 927.1MB [949336.0KB]
TMPFS free space: 0MB
I clicked the uiDivStats tab to refresh and now I see TMPFS free space: 494.9MB [99.5%]

 

[aru]

uiDivStats 4.0.4 adds TMPFS, background cron job, and daemon process monitoring. Both automatic and manual updates for graphic display work perfectly. Thanks to the author for taking time out of their busy schedule to optimize the script and bring these improvements!

 

[Martinski]

Hi @Martinski.

I saw your post mentioning tmpfs and it reminded me of this "gotcha". So I thought I'd bring it to your attention in case it's relevant here.

[Diversion] failed to write /var/lib/misc/dnsmasq.leases: No space left on device

I suspect OP has a very busy router and therefore large dnsmasq log files. I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.

www.snbforums.com

Yes, very good point, and you’re absolutely right. The TMPFS total "Size" reported by the df command is not the actual total amount currently allocated for storage but the maximum that could ever be allocated, provided that there's indeed enough available RAM for it, which means that the amount of TMPFS "Available" space is subject to the same restriction: only if there's enough RAM available that can be allocated for it at the time that is needed.

Something to keep in mind if/when creating large files (or accumulating thousands of small files) within the "/tmp" filesystem and not deleting them when they’re no longer needed.

WRT uiDivStats usage of TMPFS when "Cache Mode" is enabled, AFAICT, it would normally be only a handful of MBytes at a time since the cached data gets flushed to the USB-attached disk drive every 5 minutes by a cron job.

Personally, I’d highly recommend disabling "Cache Mode" (i.e. set to "none") so the statistical data is written directly to the USB-attached drive, especially if using an SSD - which is the current recommendation for increased performance & reliability.

I think I'm going to modify the current "TMP free space" message to include the amount of RAM currently available so users can see both next to each other. IMO, the more info the better.

Thanks for the heads-up.

 

[Martinski]

Release Notes for uiDivStats 4.0.5 version (now available):

1) Modified the recently added "TMPFS free space" information associated with the "Cache Mode" option to include the current "RAM Available" (as provided by /proc/meminfo).
This allows users to see both TMPFS and RAM available space and check if the memory is becoming too low to support database operations when the Cache Mode is set to Enabled.

Thanks to @ColinTaylor for pointing out that the TMPFS info from df cmd does not really provide meaningful information WRT the storage space available, and one has to look at the actual RAM info to have a better idea of memory availability for storage.

See the following post for further details about the results of his investigation.

[Diversion] failed to write /var/lib/misc/dnsmasq.leases: No space left on device

I suspect OP has a very busy router and therefore large dnsmasq log files. I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.

www.snbforums.com

Screenshots:

 

[Ripshod]

Release Notes for uiDivStats 4.0.5 version (now available):

1) Modified the recently added "TMPFS free space" information associated with the "Cache Mode" option to include the current "RAM Available" (as provided by /proc/meminfo).
This allows users to see both TMPFS and RAM available space and check if the memory is becoming too low to support database operations when the Cache Mode is set to Enabled.

Thanks to @ColinTaylor for pointing out that the TMPFS info from df cmd does not really provide meaningful information WRT the storage space available, and one has to look at the actual RAM info to have a better idea of memory availability for storage.

See the following post for further details about the results of his investigation.

[Diversion] failed to write /var/lib/misc/dnsmasq.leases: No space left on device

I suspect OP has a very busy router and therefore large dnsmasq log files. I think everybody agrees that he needs more space for the logs. I'm stuck on why dnsmasq thought it couldn't get space to update it's leases file when all the data suggests there was plenty of space on tmpfs.

www.snbforums.com

Screenshots:
View attachment 62954

View attachment 62955

Sorry to spoil the party, but since updating to 4.0.5 I seem to have 0 Bytes of RAM left. even after several refreshes????

Scrub that, everything's fine after CTRL+F5.

 

[aru]

General users typically do not pay constant attention to the relationship between the database (DB), tmpfs, and RAM. It is suggested that the developer consider adding an automatic detection and switching feature to dynamically select the most suitable resource allocation, ensuring stable and proper system functionality.

The diagram is as follows:

 

[Martinski]

Release Notes for uiDivStats 4.0.6 version (now available):

1) Fixed a bug where after making a "force update" shell script call, the cron jobs were created even when the "Background processing of Diversion statistics" option was set to DISABLED.

2) Some minor code improvements.

 

[Martinski]

General users typically do not pay constant attention to the relationship between the database (DB), tmpfs, and RAM. It is suggested that the developer consider adding an automatic detection and switching feature to dynamically select the most suitable resource allocation, ensuring stable and proper system functionality.
...

Nowadays, with the availability of large-capacity SSDs and their fairly low prices (compared to just 5-6 years ago), I'd say the default setting of the "Cache Mode" should be "DISABLED" and never worry about it anymore. Also, IMO, if you have a >> 100MB database file and your total number of DNS queries per day is quite large, you should not have the "Cache Mode" option set to ENABLED; use a USB-attached SSD instead.

Admittedly, I have no idea what the reasoning was behind JackYaz’s decision to offer the "Cache Mode" option and what its actual effectiveness was WRT improving performance at the time the original code was written.

In any case, at this point, I really don't see valid good reasons to spend my time on this particular option.

 

[Davidncali001]

I don't know if this has to do with the latest update which happen today (12/23/2024) but this is when I first noticed it.

I'm getting SQLData.js and SQLite3 failure/errors. I uninstalled then reinstalled uiDivStats but the same errors appear.

Please see attached photos

 

[Martinski]

I don't know if this has to do with the latest update which happen today (12/23/2024) but this is when I first noticed it.

I'm getting SQLData.js and SQLite3 failure/errors. I uninstalled then reinstalled uiDivStats but the same errors appear.
...

Very good catches!!
I know where & what the problems are and have fixes already implemented.

BTW, the SQLite3 calls are now more chatty because any errors are being captured & reported on the system log. However, the "database is locked" error messages should be filtered out since those may happen intermittently and are not necessarily fatal as several background processes are accessing the database file at regular intervals to perform their assigned database operations.

I'll issue a new release later today (most likely in the evening).
Thanks.