[Beta] VNSTAT on Merlin; CLI, UI and email data use monitoring (install script offline for now)

[dev_null]

Note: this version has been superseded by the beta 2 version: https://www.snbforums.com/threads/b...-monitoring-with-full-install-and-menu.70727/

I will be asking mods to close this thread (27-Feb-2021). Thanks to everyone who provided feedback on the alpha/beta versions!

Background

• This is an implementation of vnstat/vnstati for use on AsusWRT-Merlin routers. This effort was created to enable accurate measurement of data use on my local network. There were two primary drivers:

1. Supplement or replace the internal monitoring tool, Traffic Analyzer > Traffic Monitor, which suffers from (false) “17GB” data use bursts on some firmware (e.g., RT-AC68U 386.1). These ghost bursts interfered with tracking data usage (though monthly totals still appeared accurate).
   • This became a particular concern for me when Xfinity announced it was implementing 1.2GB caps nationwide in January 2021 (it was recently announced that capping is postponed in the Northeast until 3Q2021).
   • The 17GB ghosting is still being reported: https://www.snbforums.com/threads/t...14_2-was-reverted-in-386-1.69843/#post-658183
2. To monitor traffic locally (for privacy reasons).

• This was created with support from @Jack Yaz, who provided scaffold and scripting assistance to create the AddOn "Vnstat on Merlin". Many thanks, Jack!
• This concept and view was created for my personal use. I have not (yet) tried to automate the installation and set-up steps. I welcome your feedback - and more importantly your collaboration - to improve and/or automate installation.
  • A Day-2 alpha install script created by @Martineau - thanks so much!
  • A Day-3 single-line install script updated by @Martineau - really appreciate the collaboration!
    • See notes below about post-install steps required (location of db files, email configuration).

Features and benefits:

• Once set up, the recurring refreshes and other operations are automated, controlled by a few cron jobs.

CLI:

• The standard "vnstat" install from Entware will track data usage, with few additional configuration tweaks required (the configuration needs to be modified so the data isn't stored in a volatile location).
  • In my tests, the CLI version of vnstat totals are consistent with those collected by the built-in Traffic Monitor, without any 17GB ghosts.
  • Non-UI access to vnstat is via SSH.
  • The vnstat totals tend to be a bit higher (10-15%) than those currently reported by Comcast (NE region, 1Q2021).
    
    
    

UI:

• Vnstat has a corresponding application called vnstati which creates PNGs from the vnstat data totals. Thanks, Teemu Toivola!
• The "Vnstat on Merlin" UI uses the scaffolding from @Jack Yaz 's connmon script (with his permission and assistance) to create a reasonably attractive visual representation of data usage on the AddOns tab:
  
  
  
  • Each section can be collapsed to simplify the view
  • Refreshes require a cron job, through which the refresh frequency is adjustable. There doesn't seem to be a discernible resource impact.

Email summaries:

• Because I don't log on to the router every day, I wanted a "push" notification of data use. I had hacked together a script to email semi-weekly Skynet reports (as described here: https://www.snbforums.com/threads/r...urity-enhancements.16798/page-280#post-538221), so I repurposed and updated it into an aggregation and email script for vnstat statistics.
• This report is an amalgamation of monthly, weekly and daily usage. The report content is customizable.
  
  
  
  • The frequency and timing of this email is modifiable by adjusting the associated cron job.
  • This report is converted into an image (using the "imagemagick" package from Entware) and is located at the bottom of the UI page.

Requirements:

• AsusWRT-Merlin; scripting support is required.
• Entware, typically installed with Diversion.
  • To take advantage of the encrypted Diversion password, functioning Diversion "communications" (amtm > 1 (Diversion) > c (communication) > 5 (edit email settings, test email))
• One or more @Jack Yaz scripts, particularly connmon, so that the AddOns tab and underlying folder structure is set up. The vnstat-ui script does not have full AddOn functionality built in.
• The vnstat application (currently v1.18) and the "Vnstat on Merlin" UI has been tested on Merlin 384.19, 386.1 beta 1-5 and 386.1 release, on RT-AC66U_B1 (on AC68U firmware) and RT-AX86U 86.1 beta 1-5 and 386.1 release.
• Experience with SSH, scripting and comfort modifying settings on your router.

Support:

• I have reproduced the steps outlined in the repository and found that it works on the hardware available to me. Your hardware and settings may differ (most differences should be obvious and readily addressable; the community may be able to help).
• I'm providing this as-is, for the potential benefit of the community. Use is at your own risk.
• Any contributions from the community to optimize the install and performance are welcomed.

Location of the Tutorial - updated to Github:

• Please see the github repository: https://github.com/de-vnull/vnstat-on-merlin
  • The install and other scripts are being migrated to point to the github repository. The bitbucket repository is active until that is finished.
• Deprecated: Please visit the bitbucket repository, located here: https://bitbucket.org/vnstat-on-merlin-v001/vnstat_on_merlin/src/Initial/

New! Installer command (courtesy of @Martineau ) - see his post below.

• Because each change in bitbucket updates the UUID, the link needs to be updated. If you get an error, please try again later (we'll get the link fixed as soon as practical). In the meantime, see this post:

[Beta] VNSTAT on Merlin; CLI, UI and email data use monitoring (install script offline for now)

@Martineau I created a file called "vnstat-install.sh" in the bitbucket scripts folder. I don't think that having a different name makes a difference, but wanted to differentiate it from the other files called vnstat. Can you verify your script will work even with the name "vnstat-install.sh"...

www.snbforums.com

15 Feb 2021: For now the installer is off-line. I am working with @Martineau to try to get it back. Please follow the manual steps outlined in the github repository.

 

[dev_null]

Notes, versions and errata:

• Beta install script created by @Martineau! Linked in Martineau's comment below. A few caveats, including:
  • The script does not alter the Database location and cannot modify the email send-vnstatscript with the user credentials. You will need to make these changes manually.
  • You may wish to verify that the modifications, particularly the conf files, match your set-up.
• Hardware acceleration: @ColinTaylor reports that vnstat "appears to be using the regular interface stats and therefore suffers from the same issues regarding hardware acceleration (interface vlan2), namely no traffic is recorded for wired Ethernet clients". So with HA enabled, vnstat collected statistics may not be accurate or complete.
• The instructions have been updated to reflect that the Entware path is symlinked to /opt/ so you don't need to enter the full path to Entware. Thanks @ColinTaylor!
• @elorimer provided an improved email process which leverages Diversion. This adds an extra dependency but solves other issues. See this post: https://www.snbforums.com/threads/b...ing-with-install-script-too.70091/post-661889.
• Version 0.9.0 (10-Feb-2021) - beta: the tutorial and files have been migrated to github, and the tutorial has been cleaned up to reflect learnings from the alpha version (such as the db location, logger entries into syslog, entware path, etc; mostly behind the scenes stuff). If your install is running, this is not a mandatory update, except if you want to move from the send-vnstat (plain text credential storage) to use the Diversion-based email credential approach (in that case, see the post here: https://www.snbforums.com/threads/b...ing-with-install-script-too.70091/post-661889).
• UI page - image issues when interface is not eth0: If you are using an interface other than eth0 and are having issues with the images on the UI, you can either re-install or if you want to make a more targeted fix, download the updated vnstat-ww.sh from here. Save it to the /jffs/scripts folder on your device and run /opt/bin/vnstat -u && sh -x /jffs/scripts/vnstat-ww.sh from the CLI

 

[dev_null]

Reserved for FAQ, wishlist, and mea-culpas.

Wishlist/future development:

• Encrypt/support 2FA for email. Perhaps leverage Diversion email process? (@elorimer 's idea - need to follow-up) Done.
• Create an email notification when data use limits are being approached. Tried grepping vnstat monthly to grab the total column, but no success (thoughts anyone)? Edit: exploring exporting a json version of vnstat to see if that's easier to grep (comma delineated)
• Install script checks for existence of Diversion email credentials and sets up the correct script (div-email vs send-vnstat). Done (per Martineau)

 

[spanjap]

@dev_null: Thanks for this very nice implementation.

Tried to set it up as your instructions on my RT-AX88U. There was one problem I had.
Got the following error when starting "vnstat -u && sh /jffs/scripts/vnstat-ww.sh" manually:
vnstati: error while loading shared libraries: libjpeg.so.62: cannot open shared object file: No such file or directory

Installed "opkg install libjpeg-turbo" and now it's seems to be up and running.

PS: You don't need to "/mnt/path_to/entware/bin/vnstat" vnstat or vnstati is enough.

 

[dev_null]

@dev_null: Thanks for this very nice implementation.

Tried to set it up as your instructions on my RT-AX88U. There was one problem I had.
Got the following error when starting "vnstat -u && sh /jffs/scripts/vnstat-ww.sh" manually:
vnstati: error while loading shared libraries: libjpeg.so.62: cannot open shared object file: No such file or directory

Installed "opkg install libjpeg-turbo" and now it's seems to be up and running.

PS: You don't need to "/mnt/path_to/entware/bin/vnstat" vnstat or vnstati is enough.

Great, thanks for the feedback. I'll add the libijpeg note to the instructions... for me perhaps it was installed by another program or add-in.

W/r/t full path: I think if the path is added to the conf file it should be okay with just the command, but since the execution is generated by cron, I didn't want it to error out, unseen.

 

[ColinTaylor]

Good stuff @dev_null .

Two quick questions - 1) How are you determining the WAN interface to monitor? 2) Presumably on older models (like the RT-AC68U) this will produce no meaningful data if hardware acceleration is enabled?

 

[dev_null]

Good stuff @dev_null .

Two quick questions - 1) How are you determining the WAN interface to monitor? 2) Presumably on older models (like the RT-AC68U) this will produce no meaningful data if hardware acceleration is enabled?

Thanks very much.

In the instructions on bitbucket, I recommend that users verify the interface by issuing "nvram get wan0_ifname" from the cli before editing the configuration.

On my set-up I don't use HW acceleration, so I don't know how that would affect vnstat. If you could report back, I can add that as a notation in the above and on the bitbucket page.

 

[ColinTaylor]

I've just tried the command line installation of vnstat on my RT-AC68U using John's firmware. As suspected it appears to be using the regular interface stats and therefore suffers from the same issues regarding hardware acceleration (interface vlan2), namely no traffic is recorded for wired Ethernet clients.

I noticed a couple of differences in the install procedure. On my router the default Entware installation is in /opt rather the /entware shown in your instructions. Also, the startup script provided by Entware was called S32vnstat not s32vnstat.

 

[dev_null]

I've just tried the command line installation of vnstat on my RT-AC68U using John's firmware. As suspected it appears to be using the regular interface stats and therefore suffers from the same issues regarding hardware acceleration (interface vlan2), namely no traffic is recorded for wired Ethernet clients.

I noticed a couple of differences in the install procedure. On my router the default Entware installation is in /opt rather the /entware shown in your instructions. Also, the startup script provided by Entware was called S32vnstat not s32vnstat.

OK, I added your comments on HA above and to the bitbucket notes. I wonder if vlan2 can be added to the vnstat configuration (I'm not familiar enough with HA to know if this is simply another interface or if data tracking is completely bypassed) - I think vnstat can be configured for multiple interfaces.

A favor, if you're willing: install darkstat (it's available on Entware). I'm curious if that shows accurate traffic (it has it's own GUI - routerip:667 IIRC). I could never figure out how to get darkstat to tell me what the cumulative usage was, so I focused on vnstat, which could.

 

[Martineau]

• AsusWRT-Merlin; scripting support is required.
• Entware, typically installed with Diversion.
• One or more @Jack Yaz scripts, preferably connmon, so that the AddOns tab and underlying folder structure is set up. The vnstat-ui script does not have full AddOn functionality built in.
• The vnstat application (currently v1.18) and the "Vnstat on Merlin" UI has been tested on Merlin 384.19, 386.1 beta 1-5 and 386.1 release, on RT-AC66U_B1 (on AC68U firmware) and RT-AX86U 86.1 beta 1-5 and 386.1 release.
• Experience with SSH, scripting and comfort modifying settings on your router.

Support:

• I have reproduced the steps outlined in the repository and found that it works on the hardware available to me. Your hardware and settings may differ (most differences should be obvious and readily addressable; the community may be able to help).
• I'm providing this as-is, for the potential benefit of the community. Use is at your own risk.
• Any contributions from the community to optimize the install and performance are welcomed.

Location of the Tutorial:

• Please visit the bitbucket repository, located here: https://bitbucket.org/vnstat-on-merlin-v001/vnstat_on_merlin/src/Initial/

Crude install/uninstall script (uninstall isn't complete) that may assist in the take-up of this useful addition to the firmware or at least let non-techies trial vnstat.

/jffs/scripts/vnstat-install.sh

Beta vnstat-install.sh script

[Beta] VNSTAT on Merlin; CLI, UI and email data use monitoring (install script offline for now)

Note: this version has been superseded by the beta 2 version: https://www.snbforums.com/threads/beta-2-vnstat-on-merlin-ui-cli-and-email-data-use-monitoring-with-full-install-and-menu.70727/ I will be asking mods to close this thread (27-Feb-2021). Thanks to everyone who provided feedback on...

www.snbforums.com

 

[ColinTaylor]

I wonder if vlan2 can be added to the vnstat configuration...

Yes this is what I did, use vlan2. If I had left it as eth0 it would have produced even more inaccurate results.

 

[ColinTaylor]

I also noticed that vnstat couldn't correctly identify the speed of my WAN interface and set it at 10Mb which produces warnings in the syslog:

Feb  7 17:25:16 vnstatd[9284]: vnStat daemon 1.18 started. (pid:9284 uid:0 gid:0)
Feb  7 17:25:16 vnstatd[9284]: Monitoring: vlan2 (10 Mbit) br0 (1000 Mbit)
Feb  7 17:25:16 admin: Started vnstatd from .
Feb  7 17:30:16 vnstatd[9284]: Traffic rate for "vlan2" higher than set maximum 10 Mbit (60->83, r1030 t90), syncing.
Feb  7 17:31:16 vnstatd[9284]: Traffic rate for "vlan2" higher than set maximum 10 Mbit (60->83, r961 t92), syncing.

Setting BandwidthDetection to 0 in vnstat.conf solved that problem:

Feb  7 17:39:35 vnstatd[10349]: vnStat daemon 1.18 started. (pid:10349 uid:0 gid:0)
Feb  7 17:39:35 vnstatd[10349]: Monitoring: vlan2 (1000 Mbit) br0 (1000 Mbit)

 

[dev_null]

Crude install/uninstall script (uninstall isn't complete) that may assist in the take-up of this useful addition to the firmware or at least let non-techies trial vnstat.
Feel free to host it on your bitbuckit repository.

@Martineau , thank you very much for this contribution. I've followed your work for a long time, so I'm grateful for your interest and assistance.

I'll add this to the repository as an "early alpha install" and dig through it so I can understand how you've constructed it.

At first glance it's principally installing the scripts and applications, then the user would still need to modify the actual settings to reflect their set-up, correct?

I also noticed that vnstat couldn't correctly identify the speed of my WAN interface and set it at 10Mb which produces warnings in the syslog:

Feb  7 17:25:16 vnstatd[9284]: vnStat daemon 1.18 started. (pid:9284 uid:0 gid:0)
Feb  7 17:25:16 vnstatd[9284]: Monitoring: vlan2 (10 Mbit) br0 (1000 Mbit)
Feb  7 17:25:16 admin: Started vnstatd from .
Feb  7 17:30:16 vnstatd[9284]: Traffic rate for "vlan2" higher than set maximum 10 Mbit (60->83, r1030 t90), syncing.
Feb  7 17:31:16 vnstatd[9284]: Traffic rate for "vlan2" higher than set maximum 10 Mbit (60->83, r961 t92), syncing.

Setting BandwidthDetection to 0 in vnstat.conf solved that problem:

Feb  7 17:39:35 vnstatd[10349]: vnStat daemon 1.18 started. (pid:10349 uid:0 gid:0)
Feb  7 17:39:35 vnstatd[10349]: Monitoring: vlan2 (1000 Mbit) br0 (1000 Mbit)

@ColinTaylor now that I see this pointed out, I think I had the same issue with my setup (my vnstat.conf has '0' in the bandwidth detection setting, which I don't think is the default) so I'll add that to the guidance.

 

[Martineau]

At first glance it's principally installing the scripts and applications, then the user would still need to modify the actual settings to reflect their set-up, correct?

The script also modifies the base installation given most users may not be familiar with nano/WinSCP nor feel confident to manually edit configuration files,

e.g. I've edited my post to v1.02 and you can see that it now automatically applies the bug fix as identified below.

@ColinTaylor now that I see this pointed out, I think I had the same issue with my setup (my vnstat.conf has '0' in the bandwidth detection setting, which I don't think is the default) so I'll add that to the guidance.

NOTE: The script does not alter the Database location and clearly it cannot modify the email send-vnstatscript with the user credentials.

The script does still require minimal SSH experience to be able to execute a single command.

Requirements:

• AsusWRT-Merlin; scripting support is required.
• Entware, typically installed with Diversion.
• One or more @Jack Yaz scripts, preferably connmon, so that the AddOns tab and underlying folder structure is set up. The vnstat-ui script does not have full AddOn functionality built in.
• The vnstat application (currently v1.18) and the "Vnstat on Merlin" UI has been tested on Merlin 384.19, 386.1 beta 1-5 and 386.1 release, on RT-AC66U_B1 (on AC68U firmware) and RT-AX86U 86.1 beta 1-5 and 386.1 release.
• Experience with SSH, scripting and comfort modifying settings on your router.

 

[ColinTaylor]

@dev_null Can you check to contents of your /entware/etc/init.d/rc.unslung file. If it's the same as mine your startup script won't run unless the first letter is a capital S, e.g. S32vnstat.

I suggest that you also check to see if you have a symbolic link for /opt (ls -l /opt /tmp/opt) because if you don't it could cause all sort of problems with Entware packages in general.

 

[rgnldo]

Very nice!

 

[dev_null]

@dev_null Can you check to contents of your /entware/etc/init.d/rc.unslung file. If it's the same as mine your startup script won't run unless the first letter is a capital S, e.g. S32vnstat.

I suggest that you also check to see if you have a symbolic link for /opt (ls -l /opt /tmp/opt) because if you don't it could cause all sort of problems with Entware packages in general.

Indeed the init.d file is (capital) S32vnstat. I will correct the capitalization in my directions, but that file is created when Entware installs the application, so it should only be my reference that needs to be corrected.

And yes, there is a symbolic link. Which must be by default (I did not create that symlink).

username@RT-AX86U-C328:/tmp/home/root# ls -l /opt /tmp/opt
lrwxrwxrwx    1 username root             7 Jan 30 16:00 /opt -> tmp/opt
lrwxrwxrwx    1 username root            26 May  5  2018 /tmp/opt -> /tmp/mnt/my_SSD/entware

 

[ColinTaylor]

And yes, there is a symbolic link. Which must be by default (I did not create that symlink).

username@RT-AX86U-C328:/tmp/home/root# ls -l /opt /tmp/opt
lrwxrwxrwx    1 username root             7 Jan 30 16:00 /opt -> tmp/opt
lrwxrwxrwx    1 username root            26 May  5  2018 /tmp/opt -> /tmp/mnt/AWMX_120G/entware

Then I suggest you amend your documentation to use /opt instead of /entware as that is what is hard coded into the standard Entware scripts.

 

[dev_null]

Then I suggest you amend your documentation to use /opt instead of /entware as that is what is hard coded into the standard Entware scripts.

Yes, I'm just realizing that with the symlink they are pretty much the same. I'll make those changes, thanks @ColinTaylor and @Martineau.

I'm learning here, folks!

UPDATE: DONE.

 

[dev_null]

The script also modifies the base installation given most users may not be familiar with nano/WinSCP nor feel confident to manually edit configuration files,

e.g. I've edited my post to v1.02 and you can see that it now automatically applies the bug fix as identified below.


NOTE: The script does not alter the Database location and clearly it cannot modify the email send-vnstatscript with the user credentials.

The script does still require minimal SSH experience to be able to execute a single command.

@Martineau I created a file called "vnstat-install.sh" in the bitbucket scripts folder. I don't think that having a different name makes a difference, but wanted to differentiate it from the other files called vnstat. Can you verify your script will work even with the name "vnstat-install.sh"?