vnStat [Release] vnStat-on-Merlin - UI, CLI and email - data use and data limit monitoring - R1 and R2

[dev_null]

vnStat-on-Merlin - Release versions - R1 and R2​

Update 18 July 2021: Upgrade! For ARM and AARCH-based routers, we've got some great news: vnStat-on-Merlin Release 2 (R2) is now available for non-MIPS devices. This version is based on vnStat 2.7, recently included as an Entware package. More details below and here.

• For supported routers, in addition to updating the underlying engine of vnStat, Jack Yaz has worked some of his charting magic, building on the existing charts which are generated by vnStati.
• More notes here (and see the changelog in the next post).

Update: 15 August 2021: minor fixes to R2 and introduction of trend lines in the interactive charts within R2. No changes in R1.

• See changelog in next post and this post: https://www.snbforums.com/goto/post?id=707674

Update: 27-Dec-2021: vnStat-on-Merlin (dn-vnstat) version 1.0.3 (R1) and 2.0.3 (R2) and later check for amtm messaging existence. If available, vnStat-on-Merlin will use amtm messaging preferentially. If amtm messaging is not configured, will continue to use Diversion (for the time being).

• See changelog in next post.

11-Feb-2024: Hotfix posted to address iCloud "weird server reply" issue.

---

Original announcement:

I'm pleased to announce the release of vnStat-on-Merlin version 1.0.2 (aka R1) - for all Merlin-supported routers and version 2.000 (R2) for ARM and AARCH architecture Merlin-supported routers.

I'd like to acknowledge @Jack Yaz who is responsible for the programming behind the concept - Jack is a true partner in every sense of the word!

The main vnStat-on-Merlin github page is located here: https://github.com/de-vnull/vnstat-on-merlin; the changelog is located here.

Background​

• This is an implementation of vnStat for use on AsusWRT-Merlin routers. This effort was started to enable accurate measurement of data use in a local database (for privacy reasons), and as an alternative the internal monitoring tool, Traffic Analyzer > Traffic Monitor, which will periodically record false “17GB” usage bursts. This "false spike", as some have called it, only occurs on some routers on some firmware (I personally experienced it with RT-AC68U firmware on RT-AC66U_B1).
• Accurate tracking of data usage became a particular concern when Comcast/Xfinity stated their intention to implement 1.2TB caps nationwide.
  • The totals are consistent with the firmware "Traffic Monitor/Traffic Analyzer".
  • The totals reported by vnStat are slightly higher than those currently being reported by Comcast (about 10-12%).

Notes about data use, units and monitoring​

• vnStat-on-Merlin R1 (in the 'Legacy' branch) uses the Entware version of vnStat, currently version 1.18. This is an older version of the application. This version has certain limitations, described here: https://github.com/de-vnull/vnstat-on-merlin/blob/main/more-info.md
• vnStat-on-Merlin R2 uses the Entware version of vnStat2, currently version 2.7. This is a recent release of the software. It is supported by ARM and AARCH architectures only.
• vnStat-on-Merlin data-use and data-limit reporting should be considered a guide, an approximation of actual use. vnStat-reported totals may or may not match that reported by your provider, your cycle may start and stop on a different day of the month, a partial month (especially the first month) or the data use reported could be affected by variables such as hardware acceleration, settings that bypass the TCP/IP stack or as mundane as scheduled reboots. You must conduct proper due diligence to determine if the usage reported by vnStat aligns with your provider.

Initial installation and updates​

• Initial installation and update from R1 - choose one of the following procedures:
  • Running amtm and installing from the menu will identify the architecture of the router and install the proper version
  • Choosing u update from within the vnStat-on-Merlin menu will identify the architecture of the router and install the proper version
  • Clicking the Update button from within the vnStat-on-Merlin UI will identify the architecture of the router and install the proper version
    • Upgrading from R1 to R2 will erase any custom vnstat configurations. You may need to redo any custom settings (e.g., date formats).
• Additional information may be found on the Github page.

Minimum requirements​

• AsusWRT Merlin version 384.19 or later for R1.
  • R1 has been tested on 384.19 through 386.2_6 release version.
    • Earlier versions (384-series only) may likewise function; kindly report any further experiences (include router model and firmware version).
    • Initial testing on John's fork appears to demonstrate expected functionality. Kindly report any further experiences (include router model and firmware version).
  • AsusWRT Merlin version 386.2 or later for R2.
    • R2 has been tested on 386.2_6 release version.
    • No version of 384.x nor John's fork have been tested (no supported hardware available) - please report success/failure.
• AMTM, Diversion* and Entware.
• Update 27/28-Dec-2021:
  • * vnStat-on-Merlin (dn-vnstat) version 1.0.3 (R1) and 2.0.3 (R2) and later checks for amtm messaging existence. If available, vnStat-on-Merlin will use amtm messaging preferentially. If amtm messaging is not configured, will continue to use Diversion.
    • The Diversion requirement is now deprecated in vnStat-on-Merlin. In a future update (TBD), only amtm messaging will be supported.
    • You should start amtm, then follow the prompts to initialize and test the emailing component (amtm > em).

​

Views​

The AddOns tab showing the UI (with the "rate" column removed via vnstat.conf setting), and the "daily report" collapsed; Asus-theme:


New (R2) JY charts:

The dn-vnstat menu (CLI via SSH):



An example of the email message output - html and plain text options:

Trend lines (R2 2.0.1 and beyond):

Updating from manual/alpha/beta 1:​

• Please refer to the Github page if you are pre-R1.

CLI version of vnStat​

• For non-UI users, the CLI (non-UI) install and usage instructions can be found on the vnStat-on-Merlin Github landing page.
• Instructions on how to install email notifications using the non-UI version are in the readme.md on the landing page.

Feedback welcomed! Please list your router model and firmware version in your posts.​

Please keep all vnStat-on-Merlin issues contained to this thread.​

If it expires, another will be opened to replace it.

 

[dev_null]

Notes, versions and errata.

• Changelog: https://github.com/de-vnull/vnstat-on-merlin/blob/main/changelog.md

Version R1 (based on vnStat 1.18):

• 11-Apr-2021: version 1.0.1. Minor changes, including:
  • New format for monthly total in output listing: i.e., yyyy-mm (dd) where dd is the start of the cycle in a given period
  • Add "Router Friendly Name" to email subject line (can be modified in Diversion email set-up)
  • Add cycle start day to usage string: i.e., "You have used XX % (YY GB) of your ZZ GB cycle allowance, the cycle starts on day 1 of the month"
• 11-Jul-2021: version 1.0.2. Minor fixes to correct an error which manifested itself as an "arithmetic error"; additional hot-fix to mitigate a related error.
• 28-Dec-2021: version 1.0.3. Updating to amtm to 3.2.1 or later, R1 1.0.3 amtm messaging is now used, thus removing the requirement for Diversion, even in R1. The existing setup (using Diversion) will work for a while still, but you should set up and test amtm messaging, then update to 1.0.3. Note: amtm messaging ('em') may not be visible on the amtm menu, but can be opened by typing em.

Version R2 (based on vnStat 2.x):

• 18-Jul-2021: version 2.0.0.Major update, moves supported router architectures (ARM and AARCH) to vnStat 2.x.
  • Upgrade to vnStat version 2.7
  • Data usage is reported in megabytes (base-1000), so no correction is necessary to monitor against ISP bandwidth usage (see this)
  • Minor changes in the built-in charts, including a "shadow" to the daily and monthly usage with projected usage
  • Interactive charts
    - Data usage, by minutes, hours and days
    - Data use comparisons (current week vs previous week, by day of week)
    - Data use trends (current week, previous week, two weeks prior) -
  • Minor changes to the text-based daily reports. These now focus on the most recent periods (last day, previous week, recent 6 months) and include yearly use.
  • Choice to save data to USB rather than JFFS
• 16-Aug-2021: version 2.0.1.Minor fixes and addition of trend lines.
  • Includes the fix for the "month start" issue: remove "month start" field from monthly usage metrics. See notes here: https://www.snbforums.com/goto/post?id=707598.
  • With this version, there is a new feature: trend lines. These are found in the interactive charts on the R2vnStat-on-Merlin UI page, and show trends in data usage over time - both download and upload.
    • More information here: https://www.snbforums.com/goto/post?id=707674
• 27-Sep-2021: version 2.0.2. Minor fix to the email symlinks to address an email piggybacking change.
  • More details here: https://www.snbforums.com/goto/post?id=715523
• 27-Dec-2021: version 2.0.3. Posted here: https://www.snbforums.com/goto/post?id=731816
  • The primary change is to check for amtm messaging existence. If available, vnStat-on-Merlin will use amtm messaging preferentially.
  • If amtm messaging is not configured, will continue to use Diversion (for now; see below).
  • Notes:
    • The Diversion requirement is now deprecated in vnStat-on-Merlin R2. In a future update (TBD), only amtm messaging will be supported. This change is only for R2.
    • vnStat-on-Merlin R1(primarily MIPS-based routers and older firmware versions) still uses Diversionmessaging credentials.
      • Also on R1, there is a "false update" message that may show for vnStat-on-Merlin if you update to amtm 3.2.1. You may be shown that v 2.0.3 is available, but when you attempt to update you are informed - correctly - that the 1.0.2 version is the latest. You should ignore this update prompt in 3.2.1. The R1 update to 1.0.3 appears to fix this notification.
  • Other minor configuration, performance and back-end tweaks to charts.
• 31-Dec.2021: v. 2.0.4. Post here: https://www.snbforums.com/goto/post?id=732843
  • End-of-year-housekeeping: this update includes minor updates to charts, interface id, and uninstall functions.
• 11-Feb-2024: v. 2.0.5. Hotfix posted to address iCloud "weird server reply" issue. Only required if you are using iCloud/me.com SMTP services.
• 29-Sep-2024: v. 2.0.6: @Martinski has submitted changes to the git that have been merged. The changes include:
  
  • Fixed error when loading the webGUI page on the latest 3006.102.1 F/W version.
  • Fixed the "missing scrollbar" issue when the webGUI page was loaded on the latest 3004.386.14 F/W version.
  • Fixes/improvements when reading & checking for configuration settings.
  • Removed/modified code related to "var $j = jQuery.noConflict();" which is now considered obsolete.
    
  Many thanks to those that flagged the issue and particularly to @Martinski !
  
  Note: I have no device that runs the .14 variant, so I cannot check the scroll bar issue. Please advise if this fix doesn't address the issue.

 

[dev_null]

FAQ, wishlist, and mea-culpas.

Errata:

• If, after updating Entware, you see errors such as:
  

  vnstati: error while loading shared libraries: libwebpdecoder.so.4.1.1: cannot open shared object file: No such file or directory vnstati: error while loading shared libraries: libwebpdecoder.so.4.1.1: cannot open shared object file: No such file or directory
  vnstati: error while loading shared libraries: libwebpdecoder.so.4.1.1: cannot open shared object file: No such file or directory
  cp: can't stat '/jffs/addons/dn-vnstat.d/images/vnstat_m.png': No such file or directory

• You should run this command: opkg remove vnstati2 libgd libwebp && opkg install vnstati2to reinstall certain necessary packages.
  • This is an Entware update issue.

Wishlist:

• Potential "trend line" for R2 interactive charts - added to v 2.0.1

S33vnstat daemon "dead" - Jan-May 2024

For those that find that the S33vnstat daemon dies after a reboot, you can use a script like this to check and restart the daemon. Since there are only a few reports of this occurring (as of May 2024), I'm not inclined to add this to the script, but can if this becomes and issue. This is not a dn-vnstat thing, it's the (Linux app) vnStat daemon dying for some reason. The author of the (Linux app) vnStat doesn't have reports of this occurring, so it may be a Merlin thing.

Here's a slightly more sophisticated version of the daemon check script (see this for context - https://www.snbforums.com/goto/post?id=910841):

#!/bin/sh
# This script created in response to reports that the S33vnstat daemon is occasionally crashing after router start or reboot
# See https://www.snbforums.com/goto/post?id=910841 for the rationale for this script
# color constants
readonly red="\033[1;31m"
readonly green="\033[1;32m"
readonly std="\033[m"
# wait 2 minutes to see if the daemon is running - adjust this if the crash occurs immediately at reboot
printf "$green"
echo Please wait while we check the status of the S33vnstat daemon, used in the dn-vnstat script... default pause is 2 minutes.
printf "$std"
sleep 12
# check status of the vnstat daemon
MSG=$(/opt/etc/init.d/S33vnstat check)
echo $MSG | grep 'dead'
if [ $? -eq 0 ]
then
printf "$red"
echo "S33vnstat daemon is dead... restarting"
printf "$std"
/opt/etc/init.d/S33vnstat restart && sleep 10 && /opt/etc/init.d/S33vnstat check
# add a log entry if restart is needed
logger -s -t vnstat-check S33vnstat daemon restart triggered
else
printf "$green"
echo "S33vnstat daemon is running"
printf "$std"
exit
fi

 

[elorimer]

Smooth update to 1.0.

1. In the UI, the cli display looks like it is trying to be centered, which is hard to read.
2. I wonder, in the default configuration file, for the colors, if it would be helpful to comment what the different lines cover, and what to use for common colors.

 

[Jack Yaz]

Smooth update to 1.0.

1. In the UI, the cli display looks like it is trying to be centered, which is hard to read.

left-aligned looks much worse and out of place imo:

 

[dev_null]

left-aligned looks much worse and out of place imo:

Agreed.

Centered.

 

[elorimer]

I must be doing something wrong. I get the centered part for the Top 10, but my CLI looks like it was barfed up:

 

[Jack Yaz]

I must be doing something wrong. I get the centered part for the Top 10, but my CLI looks like it was barfed up:
View attachment 32532

which browser/OS? can you share your vnstat.conf so I can compare if you have changed any output styles?

 

[dev_null]

I must be doing something wrong. I get the centered part for the Top 10, but my CLI looks like it was barfed up:

This is what it looks like on all my routers... hopefully Jack sees something wrong with yours (tested on Chrome, FF, Edge)

EDIT: If I (mentally) delete those semi-colons and % signs at the end of some of your lines, it all gets back into alignment. Jack, are those padding?

 

[elorimer]

which browser/OS? can you share your vnstat.conf so I can compare if you have changed any output styles?

Chrome on Windows 10Pro

vnstat.conf:

 vnStat 1.18 config file
##

# default interface
Interface "eth0"

# location of the database directory
DatabaseDir "/opt/var/lib/vnstat"

# locale (LC_ALL) ("-" = use system locale)
Locale "-"

# on which day should months change
MonthRotate 1

# date output formats for -d, -m, -t and -w
# see 'man date' for control codes
DayFormat    "%x"
MonthFormat  "%b '%y"
TopFormat    "%x"

# characters used for visuals
RXCharacter       "%"
TXCharacter       ":"
RXHourCharacter   "r"
TXHourCharacter   "t"

# how units are prefixed when traffic is shown
# 0 = IEC standard prefixes (KiB/MiB/GiB/TiB)
# 1 = old style binary prefixes (KB/MB/GB/TB)
UnitMode 0

# how units are prefixed when traffic rate is shown
# 0 = IEC binary prefixes (Kibit/s...)
# 1 = SI decimal prefixes (kbit/s...)
RateUnitMode 0

# output style
# 0 = minimal & narrow, 1 = bar column visible
# 2 = same as 1 except rate in summary and weekly
# 3 = rate column visible
OutputStyle 2

# used rate unit (0 = bytes, 1 = bits)
RateUnit 1

# number of decimals to use in outputs
DefaultDecimals 2
HourlyDecimals 1

# spacer for separating hourly sections (0 = none, 1 = '|', 2 = '][', 3 = '[ ]')
HourlySectionStyle 2

# try to detect interface maximum bandwidth, 0 = disable feature
# MaxBandwidth will be used as fallback value when enabled
BandwidthDetection 0

# maximum bandwidth (Mbit) for all interfaces, 0 = disable feature
# (unless interface specific limit is given)
MaxBandwidth 1000

# interface specific limits
#  example 8Mbit limit for 'ethnone':
MaxBWethnone 8

# how many seconds should sampling for -tr take by default
Sampletime 5

# default query mode
# 0 = normal, 1 = days, 2 = months, 3 = top10
# 4 = exportdb, 5 = short, 6 = weeks, 7 = hours
QueryMode 0

# filesystem disk space check (1 = enabled, 0 = disabled)
CheckDiskSpace 1

# database file locking (1 = enabled, 0 = disabled)
UseFileLocking 1

# how much the boot time can variate between updates (seconds)
BootVariation 15

# log days without traffic to daily list (1 = enabled, 0 = disabled)
TrafficlessDays 1


# vnstatd
##

# switch to given user when started as root (leave empty to disable)
DaemonUser ""

# switch to given user when started as root (leave empty to disable)
DaemonGroup ""

# how many minutes to wait during daemon startup for system clock to
# sync time if most recent database update appears to be in the future
TimeSyncWait 10

# how often (in seconds) interface data is updated
UpdateInterval 30

# how often (in seconds) interface status changes are checked
PollInterval 30

# how often (in minutes) data is saved to file
SaveInterval 1

# how often (in minutes) data is saved when all interface are offline
OfflineSaveInterval 5

# how often (in minutes) bandwidth detection is redone when
# BandwidthDetection is enabled (0 = disabled)
BandwidthDetectionInterval 0

# force data save when interface status changes (1 = enabled, 0 = disabled)
SaveOnStatusChange 1

# enable / disable logging (0 = disabled, 1 = logfile, 2 = syslog)
UseLogging 2

# create dirs if needed (1 = enabled, 0 = disabled)
CreateDirs 1

# update ownership of files if needed (1 = enabled, 0 = disabled)
UpdateFileOwner 1

# file used for logging if UseLogging is set to 1
LogFile "/opt/var/log/vnstat.log"

# file used as daemon pid / lock file
PidFile "/opt/var/run/vnstat.pid"


# vnstati
##

# title timestamp format
HeaderFormat "%x %H:%M"

# show hours with rate (1 = enabled, 0 = disabled)
HourlyRate 1

# show rate in summary (1 = enabled, 0 = disabled)
SummaryRate 0

# layout of summary (1 = with monthly, 0 = without monthly)
SummaryLayout 1

# transparent background (1 = enabled, 0 = disabled)
TransparentBg 0

# image colors

CBackground     "475A5F"
CEdge           "475A5F"
CHeader         "475A5F"
CHeaderTitle    "FFFFFF"
CHeaderDate     "EAE303"
CText           "FFFFFF"
CLine           "B0B0B0"
CLineL          "EAE303"
CRx             "C5C5CE"
CTx             "0EC009"
CRxD            "-"
CTxD            "-"

 

[elorimer]

This is what it looks like on all my routers... hopefully Jack sees something wrong with yours (tested on Chrome, FF, Edge)

View attachment 32533

EDIT: If I (mentally) delete those semi-colons and % signs at the end of some of your lines, it all gets back into alignment. Jack, are those padding?

Yes, those things at the end are the rate bars in the text. From the email version:

 

[Jack Yaz]

I think the best thing to do is turn off text-center as it breaks the layout if rate columns etc. are turned on @dev_null

Remove text-align center on CLI in WebUI - breaks layout with rates by jackyaz · Pull Request #10 · de-vnull/vnstat-on-merlin

github.com

 

[dev_null]

Yes, those things at the end are the rate bars in the text. From the email version:

Ah. Those were disabled by default in beta 2 and later. Really find them quite useless.

I think the best thing to do is turn off text-center as it breaks the layout if rate columns etc. are turned on @dev_null

Remove text-align center on CLI in WebUI - breaks layout with rates by jackyaz · Pull Request #10 · de-vnull/vnstat-on-merlin

github.com

Merged. Please update.

 

[QuikSilver]

Typo? "stats file present on your router." (remove from?)

 

[elorimer]

Those were disabled by default in beta 2 and later.

Holdover from being an early adopter!

Maybe, though, you are right that that element should be deprecated and go back to centering in the UI.

 

[dev_null]

Typo? "stats file present on your router." (remove from?)
View attachment 32537

It's the "queen's english" me thinks. Did your stats generate with option 1 or the UI refresh button?

 

[Jack Yaz]

It's the "queen's english" me thinks. Did your stats generate with option 1 or the UI refresh button?

nope, typo

 

[dev_null]

Holdover from being an early adopter!

Maybe, though, you are right that that element should be deprecated and go back to centering in the UI.

There are probably more than a few early adopters. But I agree that the left-alignment is not attractive.

You can drop that column by editing vnstat.conf setting OutputStyle 0.

@Jack Yaz : could we sed change that setting and go back to centering?

PS - we love early adopters! You got us where we are today! (And that is in no way, shape, or form, the queen's english).

 

[Jack Yaz]

There are probably more than a few early adopters. But I agree that the left-alignment is not attractive.

You can drop that column by editing vnstat.conf setting OutputStyle 0.

@Jack Yaz : could we sed change that setting and go back to centering?

PS - we love early adopters! You got us where we are today! (And that is in no way, shape, or form, the queen's english).

do we want to enforce the 0 setting, or reset it as a one-time thing?

 

[dev_null]

Hotfix with OutputStyle 0 and centering has been merged.

@elorimer - can you give it a few minutes to propagate and then do an update (u), then report back?