[Beta 3] Vnstat on Merlin - UI, CLI and email - data use monitoring - now with data limit monitoring

[visortgw]

So, if Diversion is installed and then disabled, can vnstat email then be setup? Or, do need to install diversion, setup vnstat email, and then disable diversion?

Why not try it, and report back?

 

[dev_null]

The problem was on two models of routers RT-AX88U and RT-AC86U Firmware Version:386.1_2. There were no resets for a year, it is configured and works. The first installation of vnstat was from version 0.9.4 via amtm. After updating to vnstat 0.9.5, the problem was not resolved. Checked the installation of the libjpeg package was installed "Package libjpeg (9c-2) installed in root is up to date.", decided to additionally install the libjpeg-turbo package, installed and immediately the problem was fixed. I do not understand why you need a connection, but if you need it, it is 100 mbit/sec from the provider via WAN FTTb. About the add-ons, too, I do not quite understand, if you are about what I thought, then they are in the attached screenshots from both RT-AX88U and RT-AC86U routers.

I asked about your setup because I'm trying to understand why you alone have encountered this issue, what the setup environment is, so that we can figure it out and mitigate it if possible. N=1 typically becomes greater with wider adoption. The fact that this has happened on two of your devices is even more curious.

 

[dev_null]

So, if Diversion is installed and then disabled, can vnstat email then be setup? Or, do need to install diversion, setup vnstat email, and then disable diversion?

Why not try it, and report back?

Correct. These are options, but the hypothesis is sound. Please report back.

 

[JohnD5000]

Correct. These are options, but the hypothesis is sound. Please report back.

OK, so I installed Diversion Lite, and disabled it. In vnstat set emails to enabled............but, where do I enter the email address that I want the emails sent to?

 

[dev_null]

OK, so I installed Diversion Lite, and disabled it. In vnstat set emails to enabled............but, where do I enter the email address that I want the emails sent to?

You set up the communications in Diversion, tested and confirmed that the Diversion emailer was set up properly and it doesn't work in Vnstat on Merlin? Just making sure you have an actual working configuration before we go any further... The steps are listed on the github page if you need a reference (under "minimum requirements).

 

[JohnD5000]

You set up the communications in Diversion, tested and confirmed that the Diversion emailer was set up properly and it doesn't work in Vnstat on Merlin? Just making sure you have an actual working configuration before we go any further... The steps are listed on the github page if you need a reference (under "minimum requirements).

No, I just did this:

To disable Diversion without uninstalling: amtm > 1 (enter Diversion) > d (Diversion) > 1 (disable Diversion) > e (exit).

I figured that there would be an email address field that would appear someplace or enter into a config file some place. I guess it is a littl more involved. I will try and look into that github page when I have more time.
Thanks

 

[JohnD5000]

OK, so I tried setting it up and I'm getting error. Anyone point me where I'm going wrong? Using a gmail account without 2fa

  7. Edit SMTP Server:    smtp.gmail.com
  8. Edit Server port:    465
  9. Edit Protocol:       smtps
 10. Edit SSL flag:       --insecure
 11. Send testmail to confirm settings

 Enter your selection [1-11 e=Exit] 11
____________________________________________________

 This will send a testmail from:

 xxxxxxx@xxx.xxx to:
 xxxx xxxxxx <xxxxxxxxx@gmail.com>

 Continue? [1=Yes e=Exit] 1

*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (67) Login denied

  ✖  sending testmail failed

 Note the curl: error above and check your settings

  !  Press Enter to return to menu

 

[lbtboy]

I asked about your setup because I'm trying to understand why you alone have encountered this issue, what the setup environment is, so that we can figure it out and mitigate it if possible. N=1 typically becomes greater with wider adoption. The fact that this has happened on two of your devices is even more curious.

I've given you enough information or something else you need. Yes, it is on two devices with different hardware, with the same add-ons and configuration, with the same firmware. Maybe I'm not alone, just do not voice this problem users? I, too, could not voice it, but silently use it and decide not to post it in this topic by installing the additional libjpeg-turbo package.

 

[elorimer]

OK, so I tried setting it up and I'm getting error. Anyone point me where I'm going wrong? Using a gmail account without 2fa

  7. Edit SMTP Server:    smtp.gmail.com
  8. Edit Server port:    465
  9. Edit Protocol:       smtps
10. Edit SSL flag:       --insecure
11. Send testmail to confirm settings

Enter your selection [1-11 e=Exit] 11
____________________________________________________

This will send a testmail from:

xxxxxxx@xxx.xxx to:
xxxx xxxxxx <xxxxxxxxx@gmail.com>

Continue? [1=Yes e=Exit] 1

*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (67) Login denied

  ✖  sending testmail failed

Note the curl: error above and check your settings

  !  Press Enter to return to menu

These are two separate errors. The first one about deprecated key you can ignore. At some point that will be fixed in diversion but it will be a headache until @thelonelycoder figures out a smooth way of transitioning. Every confidence he will.

The second I suspect is because you have 2FA on in your gmail account. You can't use your regular password but need to get an application-specific password. Google will generate it for you and then you can use that.

 

[JohnD5000]

These are two separate errors. The first one about deprecated key you can ignore. At some point that will be fixed in diversion but it will be a headache until @thelonelycoder figures out a smooth way of transitioning. Every confidence he will.

The second I suspect is because you have 2FA on in your gmail account. You can't use your regular password but need to get an application-specific password. Google will generate it for you and then you can use that.

Thanks, but as I said, there is no 2fa on the account

 

[JohnD5000]

So, I decided to use a different Gmail account with 2fa ON and create an app password (as eloirmer suggested). The test email from diversion worked. I then disabled Diversion.
I setup vnstat with email.
Now, how can I send a test email from vnstat? Or do I just need to wait until midnight or something?

Thanks

 

[visortgw]

So, I decided to use a different Gmail account with 2fa ON and create an app password (as eloirmer suggested). The test email from diversion worked. I then disabled Diversion.
I setup vnstat with email.
Now, how can I send a test email from vnstat? Or do I just need to wait until midnight or something?

Thanks

Send a test from dn-vnstat -- it will send immediately if properly set up.

 

[Vergo]

Few comments/suggestions from the (vnStat) author as I happen to find this discussion while looking for possible missed issues before the vnStat version 2.7 release:

• It's called "vnStat" while "vnstat" is the cli command for textual output and "vnstati" for image output (that "i" is just for image)
• I've added the version 1.18 man pages available at https://humdi.net/vnstat/man/1.18/ as you appear to be using the last version from the 1.x series (which isn't anymore developed). As you've noticed, there are some differences compared to the more recent 2.x versions.
• Related to the documentation, see the "vnstat" man page for the "--reset" parameter. It wasn't there for reseting the database content and doesn't essentially do anything if the daemon ("vnstatd") is running.
• As Entware appears to be using packages created by OpenWrt, you may want to look for the "vnstat2" package as, for some reason, OpenWrt decided to go with a new package name for the 2.x versions. I did find it also in the Entware config files but it doesn't appear to be currently be build as a binary package and may not be directly available as a result. https://github.com/Entware/entware-packages/tree/master/net/vnstat2 however does exist.
• "BandwidthDetection" is a way of querying the network interface driver for the maximum bandwidth that the interface is capable. If that driver provided information isn't valid then there isn't much vnStat can do. Complain to the manufacturer if you happen to have connections that way. If the driver however opts to provide no information at all then vnStat uses "MaxBandwidth" as fallback value to act as a sanity check for the seen traffic. If the traffic is higher than the given value then it gets ignored. This is used to eliminate "ghost traffic" and other kinds of traffic spikes that may occur due to device driver implementation issues and other reasons. I'd highly recommend not setting "MaxBandwidth" to zero.
• Consider "--dumpdb" as deprecated and don't write implementation around it. Prefer --oneline, --json or --xml instead. Those will still be available in the 2.x versions while "--dumpdb", "--exportdb", "--importdb" get dropped out. However do note the version field.
• Assuming you have 1Gbit interfaces, consider setting "UpdateInterval" to 20 as that's the default value in later releases.
• The default database write interval is 5 minutes, not 30 seconds. See "SaveInterval" in the configuration file and the man page.
• Depending on the "vnstati" parameters, it's possible to have the command generate images only if the files don't exist or if the files are older than the specified number of minutes. If it feels like there are caching issues then check the parameters if the images are being generated on the fly.

While I can't provide distribution specific support, don't hesitate to contact if there are some other issues that need clarification or feature requests for future versions.

 

[JohnD5000]

Send a test from dn-vnstat -- it will send immediately if properly set up.

Yes. I was asking how to do this. I don't see anyplace that says "send test message" or similar.

Thanks

 

[visortgw]

Yes. I was asking how to do this. I don't see anyplace that says "send test message" or similar.

Thanks

In the CLI -- from amtm, launch dn-vnstat. Toggle emails for daily summary stats off (if already on), and then toggle (back) on.

 

[JohnD5000]

In the CLI -- from amtm, launch dn-vnstat. Toggle emails for daily summary stats off (if already on), and then toggle (back) on.

Thanks, that worked

 

[JohnD5000]

Correct. These are options, but the hypothesis is sound. Please report back.

Reporting back

I tested the first of these options:

1)Diversion is installed (I used Diversion Lite option)
2)Setup email and get a successful test email from Diversion
Note: The steps are listed on the github page if you need a reference (under "minimum requirements).
3) Disabled Diversion
4) Setup DN-VNStat to send emails
Note: To send a test email from dn-vnstat, In the CLI -- from amtm, launch dn-vnstat. Toggle emails for daily summary stats off (if already on), and then toggle (back) on.

 

[Jack Yaz]

Few comments/suggestions from the (vnStat) author as I happen to find this discussion while looking for possible missed issues before the vnStat version 2.7 release:

• It's called "vnStat" while "vnstat" is the cli command for textual output and "vnstati" for image output (that "i" is just for image)
• I've added the version 1.18 man pages available at https://humdi.net/vnstat/man/1.18/ as you appear to be using the last version from the 1.x series (which isn't anymore developed). As you've noticed, there are some differences compared to the more recent 2.x versions.
• Related to the documentation, see the "vnstat" man page for the "--reset" parameter. It wasn't there for reseting the database content and doesn't essentially do anything if the daemon ("vnstatd") is running.
• As Entware appears to be using packages created by OpenWrt, you may want to look for the "vnstat2" package as, for some reason, OpenWrt decided to go with a new package name for the 2.x versions. I did find it also in the Entware config files but it doesn't appear to be currently be build as a binary package and may not be directly available as a result. https://github.com/Entware/entware-packages/tree/master/net/vnstat2 however does exist.
• "BandwidthDetection" is a way of querying the network interface driver for the maximum bandwidth that the interface is capable. If that driver provided information isn't valid then there isn't much vnStat can do. Complain to the manufacturer if you happen to have connections that way. If the driver however opts to provide no information at all then vnStat uses "MaxBandwidth" as fallback value to act as a sanity check for the seen traffic. If the traffic is higher than the given value then it gets ignored. This is used to eliminate "ghost traffic" and other kinds of traffic spikes that may occur due to device driver implementation issues and other reasons. I'd highly recommend not setting "MaxBandwidth" to zero.
• Consider "--dumpdb" as deprecated and don't write implementation around it. Prefer --oneline, --json or --xml instead. Those will still be available in the 2.x versions while "--dumpdb", "--exportdb", "--importdb" get dropped out. However do note the version field.
• Assuming you have 1Gbit interfaces, consider setting "UpdateInterval" to 20 as that's the default value in later releases.
• The default database write interval is 5 minutes, not 30 seconds. See "SaveInterval" in the configuration file and the man page.
• Depending on the "vnstati" parameters, it's possible to have the command generate images only if the files don't exist or if the files are older than the specified number of minutes. If it feels like there are caching issues then check the parameters if the images are being generated on the fly.

While I can't provide distribution specific support, don't hesitate to contact if there are some other issues that need clarification or feature requests for future versions.

all good stuff, thanks! I'll build max bandwidth in as a menu / installer option which should eliminate the spikes reported by 1 user, if I understand you correctly.

re. vnstat 2 this is opened as a new package request with entware here: https://github.com/Entware/Entware/issues/612

 

[dev_null]

Few comments/suggestions from the (vnStat) author as I happen to find this discussion while looking for possible missed issues before the vnStat version 2.7 release:

• It's called "vnStat" while "vnstat" is the cli command for textual output and "vnstati" for image output (that "i" is just for image)
• I've added the version 1.18 man pages available at https://humdi.net/vnstat/man/1.18/ as you appear to be using the last version from the 1.x series (which isn't anymore developed). As you've noticed, there are some differences compared to the more recent 2.x versions.
• Related to the documentation, see the "vnstat" man page for the "--reset" parameter. It wasn't there for reseting the database content and doesn't essentially do anything if the daemon ("vnstatd") is running.
• As Entware appears to be using packages created by OpenWrt, you may want to look for the "vnstat2" package as, for some reason, OpenWrt decided to go with a new package name for the 2.x versions. I did find it also in the Entware config files but it doesn't appear to be currently be build as a binary package and may not be directly available as a result. https://github.com/Entware/entware-packages/tree/master/net/vnstat2 however does exist.
• "BandwidthDetection" is a way of querying the network interface driver for the maximum bandwidth that the interface is capable. If that driver provided information isn't valid then there isn't much vnStat can do. Complain to the manufacturer if you happen to have connections that way. If the driver however opts to provide no information at all then vnStat uses "MaxBandwidth" as fallback value to act as a sanity check for the seen traffic. If the traffic is higher than the given value then it gets ignored. This is used to eliminate "ghost traffic" and other kinds of traffic spikes that may occur due to device driver implementation issues and other reasons. I'd highly recommend not setting "MaxBandwidth" to zero.
• Consider "--dumpdb" as deprecated and don't write implementation around it. Prefer --oneline, --json or --xml instead. Those will still be available in the 2.x versions while "--dumpdb", "--exportdb", "--importdb" get dropped out. However do note the version field.
• Assuming you have 1Gbit interfaces, consider setting "UpdateInterval" to 20 as that's the default value in later releases.
• The default database write interval is 5 minutes, not 30 seconds. See "SaveInterval" in the configuration file and the man page.
• Depending on the "vnstati" parameters, it's possible to have the command generate images only if the files don't exist or if the files are older than the specified number of minutes. If it feels like there are caching issues then check the parameters if the images are being generated on the fly.

While I can't provide distribution specific support, don't hesitate to contact if there are some other issues that need clarification or feature requests for future versions.

Echo'ing Jack: thanks for this! I started this project for personal use and with other SNB member's encouragement (and tireless support!), it's become what you see. So I've been learning what this Entware version is capable of doing (and not capable)... your information is very helpful (particularly the BWDetection setting - had no idea that would help filter false spikes - very cool).

Question (and maybe you Jack and I can take this offline): is there an import function after a --oneline, --json or --xml export? I've actually used that to preserve data and move it across platforms.

The other stuff we can roll up to future releases. Still hoping Entware does incorporate 2.x. Cheers,

 

[Vergo]

Question (and maybe you Jack and I can take this offline): is there an import function after a --oneline, --json or --xml export? I've actually used that to preserve data and move it across platforms.

The output of --oneline, --json or --xml isn't intended to be imported back. If you need to move the version 1.x database around between architectures then --exportdb and --importdb are there for that. In version 2.x it's possible to move the database file directly as sqlite handles the format while version 1.x database stops working if the endianness or architecture bit width changes.

The situations when you'd want to use --oneline, --json or --xml is when you are trying to pull out some single measurement or a limited series from the database. These output allow you to get access to the values directly without having to parse the text formatted output. As an example, I found this from db-vnstat.sh:

$VNSTAT_COMMAND -m | tail -n 3 | head -n 1 | cut -d "|" -f3 | awk '{print $1}'

while it could be replaced with

$VNSTAT_COMMAND --oneline b | cut -d\; -f11

and you wouldn't even need to worry about running the command again since that drops the unit and provides the result directly in bytes. 2 commands executed instead of 10.

PS. When the daemon ("vnstatd") is running in the background, there shouldn't be a need to run "vnstat -u". When that combination is executed, the "vnstat -u" command ends up rewriting the database file based on the limit visibility it has on the current situation. Meanwhile, the daemon still stays running in the background with a cached copy of the database and will eventually overwrite the database with that cached and updated copy based on the longer duration changes it has seen. "vnstat -u" was removed from the 2.x versions to avoid this to even happen while in 1.x versions it can cause some confusing result as the database gets overwritten with cached data after a while.