[Beta] VNSTAT on Merlin; CLI, UI and email data use monitoring (install script offline for now)

[Martineau]

@Martineau I created a file called "vnstat-install.sh" in the bitbucket scripts folder. I don't think that having a different name makes a difference, but wanted to differentiate it from the other files called vnstat. Can you verify your script will work even with the name "vnstat-install.sh"?

I've submitted v1.03 pull request to accommodate the rename to vnstat-install.sh

Change log

1. Add Progress messages
2. Do not remove Entware package libjpeg-turbo during the uninstall process.

I have removed the code and replaced it with

This install command

e.g. verification that the new name works....

./vnstat-install.sh install

./vnstat-install.sh: v1.03 Installing 'vnstat'....
./vnstat-install.sh:    Installing 'vnstat' Entware modules
Installing vnstat (1.18-2) to root...
Downloading http://bin.entware.net/aarch64-k3.10/vnstat_1.18-2_aarch64-3.10.ipk
Installing vnstati (1.18-2) to root...
Downloading http://bin.entware.net/aarch64-k3.10/vnstati_1.18-2_aarch64-3.10.ipk
Installing libjpeg-turbo (2.0.6-1) to root...
Downloading http://bin.entware.net/aarch64-k3.10/libjpeg-turbo_2.0.6-1_aarch64-3.10.ipk
Installing imagemagick (7.0.9-2) to root...
Downloading http://bin.entware.net/aarch64-k3.10/imagemagick_7.0.9-2_aarch64-3.10.ipk
Configuring vnstat.
Configuring libjpeg-turbo.
Configuring vnstati.
Configuring imagemagick.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   904  100   904    0     0   2597      0 --:--:-- --:--:-- --:--:--  2627
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   679  100   679    0     0   1656      0 --:--:-- --:--:-- --:--:--  1668
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4230  100  4230    0     0  12190      0 --:--:-- --:--:-- --:--:-- 12332
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1284  100  1284    0     0   3658      0 --:--:-- --:--:-- --:--:--  3689
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8233  100  8233    0     0  19235      0 --:--:-- --:--:-- --:--:-- 19417
./vnstat-install.sh:    Creating Addons GUI TAB
vnstat-ui: Mounting vnstat-ui WebUI page as user6.asp

./vnstat-install.sh:    Adding 'vnstat-ui' Addons GUI TAB creation service-event (@boot)
./vnstat-install.sh:    Adding cron jobs to services-start
./vnstat-install.sh:    Customising 'vnstat.conf' Auto detected (nvram get wan0_ifname) 'Interface eth0'
./vnstat-install.sh:    Customising 'vnstat.conf' 'BandwidthDetection 0'
./vnstat-install.sh:    Customising 'S32vnstat/vnstat-ww.sh' scripts Auto detected (nvram get wan0_ifname) 'Interface eth0'
Starting vnstatd...              done.

Deliberate attempt to install vnstat to bind to a non-existant Interface provided by the user!

./vnstat-install.sh install vlan2

./vnstat-install.sh: v1.03 Installing 'vnstat'....
./vnstat-install.sh:    Installing 'vnstat' Entware modules
Package vnstat (1.18-2) installed in root is up to date.
Package vnstati (1.18-2) installed in root is up to date.
Package libjpeg-turbo (2.0.6-1) installed in root is up to date.
Package imagemagick (7.0.9-2) installed in root is up to date.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   904  100   904    0     0   2651      0 --:--:-- --:--:-- --:--:--  2666
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   679  100   679    0     0   1962      0 --:--:-- --:--:-- --:--:--  1985
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4230  100  4230    0     0  12051      0 --:--:-- --:--:-- --:--:-- 12190
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1284  100  1284    0     0   3700      0 --:--:-- --:--:-- --:--:--  3743
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8233  100  8233    0     0  23257      0 --:--:-- --:--:-- --:--:-- 23522
./vnstat-install.sh:    Creating Addons GUI TAB
vnstat-ui: Mounting vnstat-ui WebUI page as user6.asp

./vnstat-install.sh:    Adding 'vnstat-ui' Addons GUI TAB creation service-event (@boot)
./vnstat-install.sh:    Adding cron jobs to services-start
./vnstat-install.sh:    Customising 'vnstat.conf' Manual override 'Interface vlan2'
./vnstat-install.sh:    Customising 'vnstat.conf' 'BandwidthDetection 0'
./vnstat-install.sh:    Customising 'S32vnstat/vnstat-ww.sh' scripts Manual override 'Interface vlan2'
Error: Unable to read database "/opt/var/lib/vnstat/vlan2": No such file or directory
Error: Unable to get interface "vlan2" statistics.
Only available interfaces can be added for monitoring.

The following interfaces are currently available:
    spu_us_dummy sit0 imq1 tun22 (10 Mbit) lo eth3 br0 bcmsw tun21 (10 Mbit) ifb1 imq0 eth6 eth2 ip6tnl0 ifb0 eth5 eth1 dpsta imq2 spu_ds_dummy eth4 eth0 (1000 Mbit)
Shutting down vnstatd...              done.
Starting vnstatd...              done.

./vnstat-install.sh uninstall

./vnstat-install.sh: v1.03 Uninstalling 'vnstat'....
./vnstat-install.sh:    Stopping 'vnstat'
./vnstat-install.sh:    Uninstalling 'vnstat' Entware modules
Removing package vnstat from root...
Not deleting modified conffile /opt/etc/vnstat.conf.
Removing package vnstati from root...
Removing package imagemagick from root...
./vnstat-install.sh:    Deleting directories '/jffs/addons/vnstat*' and configuration 'vnstat.conf'
./vnstat-install.sh:    Uninstalling './vnstat-install.sh'

 

[dev_null]

I've submitted v1.03 pull request to accommodate the rename to vnstat-install.sh

Change log

1. Add Progress messages
2. Do not remove Entware package libjpeg-turbo during the uninstall process.

@Martineau , thanks so much. I've approved and merged the pull. I will also update the main post.

 

[Martineau]

@Martineau , thanks so much. I've approved and merged the pull. I will also update the main post.

Until someone more familiar/experienced with bitbucket can assist, you will need to change your update in the first post....the HEX UUID will change each time you perform a bitbucket commit to vnstat-install.sh

 

[dev_null]

Until someone more familiar/experienced with bitbucket can assist, you will need to change your update in the first post....the HEX UUID will change each time you perform a bitbucket commit to vnstat-install.sh

Thanks for calling this out. I updated the command to reflect the raw path: curl -O https://bitbucket.org/vnstat-on-merlin-v001/vnstat_on_merlin/raw/7e7b8e3eeb5886dd1bcb964d2285220e36510873/scripts/vnstat-install.sh -o /jffs/scripts/vnstat-install.sh && chmod +x /jffs/scripts/vnstat-install.sh && dos2unix /jffs/scripts/vnstat-install.sh;/jffs/scripts/vnstat-install.sh install. Is that all I needed to do?

EDIT: In the email I see your suggestion that I link to your post. I'll take that approach.

 

[dev_null]

Thanks for calling this out. I updated the command to reflect the raw path: curl -O https://bitbucket.org/vnstat-on-merlin-v001/vnstat_on_merlin/raw/7e7b8e3eeb5886dd1bcb964d2285220e36510873/scripts/vnstat-install.sh -o /jffs/scripts/vnstat-install.sh && chmod +x /jffs/scripts/vnstat-install.sh && dos2unix /jffs/scripts/vnstat-install.sh;/jffs/scripts/vnstat-install.sh install. Is that all I needed to do?

EDIT: In the email I see your suggestion that I link to your post. I'll take that approach.

@Martineau, I updated the instructions and it appears to have changed the UUID (to 3092dd5b0e12dd08a55c1b9f8d0022bcf12b52d0).

That is going to be quite aggravating. Perhaps someone can recommend how best to approach this... maybe a separate repository with the scripts from the instructions (since the former is likely to be updated infrequently)? Or just let the instructions/background become inaccurate?

Or do other "bitbucket" type hosts handle the UUID situation differently? Looking for recommendations (@Jack Yaz ?)

 

[Jack Yaz]

@Martineau, I updated the instructions and it appears to have changed the UUID (to 3092dd5b0e12dd08a55c1b9f8d0022bcf12b52d0).

That is going to be quite aggravating. Perhaps someone can recommend how best to approach this... maybe a separate repository with the scripts from the instructions (since the former is likely to be updated infrequently)? Or just let the instructions/background become inaccurate?

Or do other "bitbucket" type hosts handle the UUID situation differently? Looking for recommendations (@Jack Yaz ?)

Github?

 

[elorimer]

@Martineau : I ran the install line from /home/root initially and that was where the install script downloaded; then the install failed at the chmod line because the script wasn't in /jffs/scripts. Not sure why that would be given the -o specification.

But when I ran the install line from /jffs/scripts it all went dandy.

 

[Martineau]

@Martineau : I ran the install line from /home/root initially and that was where the install script downloaded; then the install failed at the chmod line because the script wasn't in /jffs/scripts. Not sure why that would be given the -o specification.

But when I ran the install line from /jffs/scripts it all went dandy.

@elorimer thanks for the bug report.

@dev_null please remove '-O' in the curl command

i.e.

admin@RT-AC86U-6160>/tmp/home/root> ls

admin@RT-AC86U-6160>/tmp/home/root> curl https://bitbucket.org/vnstat-on-merlin-v001/vnstat_on_merlin/raw/3092dd5b0e12dd08a55c1b9f8d0022bcf12b52d0/scripts/vnstat-install.sh -o /jffs/scri
pts/vnstat-install.sh && chmod +x /jffs/scripts/vnstat-install.sh && dos2unix /jffs/scripts/vnstat-install.sh;/jffs/scripts/vnstat-install.sh install


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5050  100  5050    0     0  14265      0 --:--:-- --:--:-- --:--:-- 14852
/jffs/scripts/vnstat-install.sh: v1.03 Installing 'vnstat'....
/jffs/scripts/vnstat-install.sh:    Installing 'vnstat' Entware modules
Package vnstat (1.18-2) installed in root is up to date.
Package vnstati (1.18-2) installed in root is up to date.
Package libjpeg-turbo (2.0.6-1) installed in root is up to date.
Package imagemagick (7.0.9-2) installed in root is up to date.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   904  100   904    0     0   2243      0 --:--:-- --:--:-- --:--:--  2260
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   679  100   679    0     0   1689      0 --:--:-- --:--:-- --:--:--  1706
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4230  100  4230    0     0  10342      0 --:--:-- --:--:-- --:--:-- 10418
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1284  100  1284    0     0   3637      0 --:--:-- --:--:-- --:--:--  3689
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8233  100  8233    0     0  19509      0 --:--:-- --:--:-- --:--:-- 19649
/jffs/scripts/vnstat-install.sh:    Creating Addons GUI TAB
vnstat-ui: Mounting vnstat-ui WebUI page as user6.asp

/jffs/scripts/vnstat-install.sh:    Adding 'vnstat-ui' Addons GUI TAB creation service-event (@boot)
/jffs/scripts/vnstat-install.sh:    Adding cron jobs to services-start
/jffs/scripts/vnstat-install.sh:    Customising 'vnstat.conf' Auto detected (nvram get wan0_ifname) 'Interface eth0'
/jffs/scripts/vnstat-install.sh:    Customising 'vnstat.conf' 'BandwidthDetection 0'
/jffs/scripts/vnstat-install.sh:    Customising 'S32vnstat/vnstat-ww.sh' scripts Auto detected (nvram get wan0_ifname) 'Interface eth0'
Shutting down vnstatd...              done.
Starting vnstatd...              done.

 

[elorimer]

I have several scripts that email me reports at intervals, and I've found it convenient to piggyback on Diversion's method. I have an email script that accepts two variables, the first being the subject line and the second being the file to be sent via curl, using the email parameters stored in Diversion. Any variation in sending the email can then be handled within Diversion. Since Diversion is a prereq, you might consider something similar.

In my case then, the end of the chron job is to call

/jffs/scripts/email.sh "VNSTAT report" "/home/root/vnstat.txt"

The script here obviously doesn't work for me because I have 2FA in place.

 

[dev_null]

I have several scripts that email me reports at intervals, and I've found it convenient to piggyback on Diversion's method. I have an email script that accepts two variables, the first being the subject line and the second being the file to be sent via curl, using the email parameters stored in Diversion. Any variation in sending the email can then be handled within Diversion. Since Diversion is a prereq, you might consider something similar.

In my case then, the end of the chron job is to call

/jffs/scripts/email.sh "VNSTAT report" "/home/root/vnstat.txt"

The script here obviously doesn't work for me because I have 2FA in place.

I'd love to be able to do this, I just never figured out the "piggyback" method, and yes, 2FA would render this approach ineffective (I use a dedicated email account on my Google Workspace account for just this activity).

If you're willing (perhaps by PM) to share your piggybacking details, I'll see what can be done.

(Tangentially related: I'd also like to create an email message that comes when I approach my limits... I've tried grepping the vnstat monthly report to pull out the total column, but have had no success yet. So I have other things I'd like to be able to do.)

 

[elorimer]

I have a file in /jffs/scripts called "email.sh". This is adapted from Diversion's own email script:

#!/opt/bin/sh
#Parameters passed#
mailsubject=$1
mailbody=$2

# Email settings (mail envelope) #
. /opt/share/diversion/.conf/email.conf
PASSWORD=$(openssl aes-256-cbc -d  -in /opt/share/diversion/.conf/emailpw.enc -pass pass:ditbabot,isoi)

#Build email
    echo "From: \"$FRIENDLY_ROUTER_NAME\" <$FROM_ADDRESS>" >/tmp/mail.txt
    echo "To: \"$TO_NAME\" <$TO_ADDRESS>" >>/tmp/mail.txt
    echo "Subject: $mailsubject " >>/tmp/mail.txt
    echo "Date: $(date -R)" >>/tmp/mail.txt
    echo >>/tmp/mail.txt
    echo " $(cat $mailbody)" >>/tmp/mail.txt
  

#Send Email
#First parameter is subject, second is file to send
/usr/sbin/curl --url $PROTOCOL://$SMTP:$PORT \
        --mail-from "$FROM_ADDRESS" --mail-rcpt "$TO_ADDRESS" \
                    --upload-file /tmp/mail.txt \
                    --ssl-reqd \
                    --user "$USERNAME:$PASSWORD" $SSL_FLAG
          
rm /tmp/mail.txt

The bit in the beginning is how diversion stores its email settings and encrypts the password. That takes care of at least 4 smtp setups. The script pulls in that information, so all one needs to do is pass the two parameters to the script. Extra points for logging.

 

[ugandy]

is this utility expected to have any impact on router throughput ?
my max speeds on my 1Gbps symmetrical appear to be capped at 300Mbps after this install (previous 900)
could be something else, but i'm wondering, it looks very similar to the cap I got with CakeQoS or FlexQoS or Suricata.
[edit]: confirmed. after un installing my speed is back at 900Mbps

 

[dev_null]

I have a file in /jffs/scripts called "email.sh". This is adapted from Diversion's own email script: <snip>

The bit in the beginning is how diversion stores its email settings and encrypts the password. That takes care of at least 4 smtp setups. The script pulls in that information, so all one needs to do is pass the two parameters to the script. Extra points for logging.

@elorimer thanks for the tip. I have it working now and will run it overnight to ensure it executes from the cron job. This will be a great help for those that have Diversion credentials already stored. Thanks for pointing me in the right direction!

 

[dev_null]

is this utility expected to have any impact on router throughput ?
my max speeds on my 1Gbps symmetrical appear to be capped at 300Mbps after this install (previous 900)
could be something else, but i'm wondering, it looks very similar to the cap I got with CakeQoS or FlexQoS or Suricata.
[edit]: confirmed. after un installing my speed is back at 900Mbps

There is no reason that I can tell it should, unless for some reason hardware acceleration was disabled when installed. But I don't have a gig connection to try (and I'm envious of you who do). Perhaps others can weigh in?

Note however that ColinTaylor reported that with hardware acceleration activated, that the accounting vnstat does is likely no more accurate than with the built-in counter.

 

[Milan]

For some reason cron job for updating stats is not working for me. I see them in cru table:

"59 23 * * * /opt/bin/vnstat -u #vnstat_daily#
"/13 vnstat.txt vnstat.txt vnstat.txt /opt/bin/vnstat -u #vnstat_update#

 

[Martineau]

For some reason cron job for updating stats is not working for me. I see them in cru table:

"59 23 * * * /opt/bin/vnstat -u #vnstat_daily#
"/13 vnstat.txt vnstat.txt vnstat.txt /opt/bin/vnstat -u #vnstat_update#

@Milan Thanks for the bug report.

There are a couple of typos in the cron definitions

RT-AC86U-6160 crond[1745]: user admin: parse error at /13

that I blindly copied from the manual install/setup instructions:

However, vnstat-install.sh v1.03 is further flawed as the cron jobs will only be active following a reboot (i.e. they are only added to services-start)

FYI I think they should look like this...

*/13 * * * * /opt/bin/vnstat -u && sh /jffs/addons/vnstat.d/vnstat-ww.sh #vnstat_update#
59 23 * * * /opt/bin/vnstat -u && sh /jffs/addons/vnstat.d/vnstat-stats && sh /jffs/addons/vnstat.d/send-vnstat #vnstat_daily#

I've created vnstat-install.sh v1.04 and I will submit a pull request to @dev_null, when he has made a final decision on his repository choice.

However, in the interim given the time difference (and totally embarrassed not having checked that the cron schedules actually worked for myself) I will temporarily host a patched version

vnstat-install.sh v1.04

 

[dev_null]

vnstat-install.sh v1.04

Thanks @Martineau I think the asterisk at the front of that command was interpreted by bitbucket as a bullet at one point during formatting.

I'm working on transferring from bitbucket to github this week, so appreciate the support. I'm going to PM you with a couple of other changes, including the email process, for which I have an update.

EDIT: yup, the asterisk is there, bitbucket see's it as asking for italics... grrr.

 

[Milan]

@Milan Thanks for the bug report.

There are a couple of typos in the cron definitions

RT-AC86U-6160 crond[1745]: user admin: parse error at /13

that I blindly copied from the manual install/setup instructions:

View attachment 30516

However, vnstat-install.sh v1.03 is further flawed as the cron jobs will only be active following a reboot (i.e. they are only added to services-start)

FYI I think they should look like this...

*/13 * * * * /opt/bin/vnstat -u && sh /jffs/addons/vnstat.d/vnstat-ww.sh #vnstat_update#
59 23 * * * /opt/bin/vnstat -u && sh /jffs/addons/vnstat.d/vnstat-stats && sh /jffs/addons/vnstat.d/send-vnstat #vnstat_daily#

I've created vnstat-install.sh v1.04 and I will submit a pull request to @dev_null, when he has made a final decision on his repository choice.

However, in the interim given the time difference (and totally embarrassed not having checked that the cron schedules actually worked for myself) I will temporarily host a patched version

vnstat-install.sh v1.04

Thanks, working very well ...

 

[dev_null]

Email update: thanks to @elorimer 's script, there is now a second (better! in most cases) option for the vnstat-stats email process.

This adds another dependency - leveraging @thelonelycoder 's Diversion "communication" email process - but solves other issues, including support for platforms other than gmail, secure storage of passwords, etc.

Here are the steps to replace the existing send-vnstat process:

1. Make sure that Diversion's email communication is set up (amtm > 1 (Diversion) > c (communication) > 5 (edit email settings, test email)) with the account of your choice.
2. In /jffs/scripts, create the div-email.sh from this source. Make sure it's executable (octal 0755).
3. Update the cron job via SSH to point to this script rather than the send-vnstat script (all on a single line):
   cru a vnstat_daily "59 23 * * * /opt/bin/vnstat -u && sh /jffs/scripts/vnstat-stats && sh /jffs/scripts/div-email.sh Vnstat-stats /home/root/vnstat.txt"
4. Test by running /opt/bin/vnstat -u && sh /jffs/scripts/vnstat-stats && sh /jffs/scripts/div-email.sh Vnstat-stats /home/root/vnstat.txt
5. If not using send-vnstat, you can delete it (to secure your email details).

I'll be connecting with @Martineau to make mods to the install script he has so generously created.

---

Aside: the beauty of this div-email.sh is that you can send anything that can be put into a text file as an email message. Simply pass the script name /jffs/scripts/div-email.sh (space) "Subject line in quotes if there are spaces" (space) /path/to/text/file - that's it.

For example, here's my script to send weekly skynet updates:

#!/bin/sh
# change the number 25 below to the number of hits you want in your report
sh /jffs/scripts/firewall stats 25 | sed -r 's/'$(echo -e "\033")'\[[0-9]{1,2}(;([0-9]{1,2})?)?[mK]//g' | strings >  /home/root/skynet.txt
# the next command prunes whitespace from the top of the file
sed -i '1,25d' /home/root/skynet.txt
# sends a message to the system log
logger -s -t skynet_stats periodic stats complete

Which now is sent with the same div-email.sh script via this cron job (no additional email customization required!):
cru a skynet-wkly "2 2 * * SUN sh /jffs/scripts/skynet-stats && /jffs/scripts/div-email.sh Skynet-stats /home/root/skynet.txt"

 

[dev_null]

Request to users of Vnstat on Merlin: if anyone has installed with Martineau's script and not modified the database file location (that is, left the db - eth0 and .eth0 - in the default install location, which is probably in the /opt/ directory) - can you let me know if the statics survive reboot in this default location please.

This will simplify the installation (when I first was setting this up, I lost stats on reboot, but I'm not sure it was due to db location or some other configuration change. There were a lot of starts and stops to get this working - mostly my learning curve.)

I'm now trying to optimize the process and if the default location works, then I can remove at least one custom step.

Thanks very much.