[EDIT]Real-time Quota Monitoring (Download/Upload/Data Used)

[Martineau]

There was a query recently about how to detect DOWNLOAD throughput activity and BLOCK the device if deemed appropriate so I have revised my suggested solution.

Using the Traffic Analyzer reports, you can review device activity, but real-time monitoring may be useful.

So here is my shoddy QuotaMonitor.sh script to monitor/apply a DOWNLOAD (Rx) or UPLOAD (Tx) rate or Data Cap or apply QoS throttling.
EDIT: 13/12/2018 v1.04 Released see Release notes
EDIT: 30/11/2018 v1.03 Released see Release notes
EDIT: 27/11/2018 v1.02 Released see Release notes Thanks to @FreshJR for the example 'throttle' rules which is less harsh than a complete BLOCK!

It isn't foolproof by any means, and it may seemingly take a few seconds before the BLOCK is apparent - i.e. buffered streaming video will play until the next burst of the data stream is requested.

The default limits used by the script are:

Allow Download (Rx) maximum 50MB and Upload (Tx) maximum 10MB in 60 seconds with a Total Used Data cap applied of 50GB from the time the monitoring was initialised.

However, these arbitrary (unrealistic?) limits may be modified as required.

Prerequisites:

N/A​

 ./QuotaMonitor.sh -h

#======================================================================================= © 2018 Martineau v1.04
#
# Monitor device traffic quotas and apply block if limit exceeded
#
#    QuotaMonitor     [help | -h] ['ip='{[ip_address[,...] | hostname[...] | auto | all ]} ['remove']] ['init'[qos]] ['reset' | 'resetdel'] ['unblock'] ['zero']
#                     ['monitor'] ['interval='{interval_seconds}] ['dlimit={interval_limit}'] ['ulimit={interval_limit}']  ['quota='['rx'|'tx'|'used']]
#                     ['cap='{cap_limit}] ['ignorezero'] ['actionrx='{script_name}] ['actiontx='{script_name}] ['actionused='{script_name}] ['nolog'] ['qos']
#                     ['report='{path_for_quota.csv}] ['once'] ['status'[ 'verbose']]
#
#    QuotaMonitor     init
#                     Create the two Quota Monitor Chains (MyQUOTAi/MyQUOTA) for DOWNLOAD/UPLOAD
#                     Individual device Quota limits may be defined in '/jffs/configs/QuotaLimits' but will be ignored if 'dlimit='/'ulimit=' specified
#                     e.g.
#                         #Hostname      Rx     Tx     Cap
#                         HP-Envy13   300MB   20MB   100GB
#                     NOTE: Quota limits may only be applied to the same 'interval=' value
#    QuotaMonitor     status
#                     Show the contents of the two Chains. The USED var is derived from the RECV/XMIT byte count!
#    QuotaMonitor     ip=192.168.1.123,iphone,laptop
#                     Add three devices to be monitored
#    QuotaMonitor     ip=auto
#                     Attempt to auto discover all LAN devices and add them to the monitor list.
#    QuotaMonitor     monitor
#                     Start Rx/Tx and Total Used quota monitoring for the devices defined (default limits 50MB in 60 secs Capped @50GB)
#                     If the limits are exceeded, then the device is BLOCKED.
#    QuotaMonitor     monitor interval=10 dlimit=5mb cap=100MB
#                     Start Start Rx/Tx and Total Used quota monitoring for the devices defined in the Chains (uses 5MB in 10 secs Capped @100MB)
#    QuotaMonitor     monitor interval=10 dlimit=5mb cap=100MB ignorezero quota=
#                     Start Start Rx/Tx and Total Used quota monitoring for the devices, but devices which have not received/transmitted data are not displayed.
#                     This mode is useful for identifying suitable quota limits, as no quota metric (Rx,Tx or Used) is enforced
#                     Useful for testing, to prove if the Quota limits can be ENFORCED in real-time
#    QuotaMonitor     monitor actionRx=Quota_Rx.sh
#                     Start the Rx/Tx and Total Used Quota monitoring, but rather than BLOCK the device if the Rx quota is exceeded run script Quota_Rx.sh
#    QuotaMonitor     monitor qos
#                     Start the Rx/Tx and Total Used Quota monitoring, but rather than BLOCK the device if the Rx quota is exceeded apply QoS rules
#                     NOTE: 'initqos' must have been previously specified.
#    QuotaMonitor     ip=laptop unblock
#                     Unblock a specified device.
#    QuotaMonitor     ip=all unblockqos
#                     Unblock ALL devices that are currently throttled by QoS.
#    QuotaMonitor     ip=laptop remove
#                     Remove the device 'laptop' from the Quota Monitoring.
#                     NOTE: This will require a second instance of the script to allow uninterrupted monitoring
#    QuotaMonitor     reset
#                     Removes all devices from monitoring but retains the Chains.
#    QuotaMonitor     resetdel
#                     Deletes both chains and all iptables rules
#    QuotaMonitor     zero
#                     Resets the Quota counts to 0. Could be scheduled by a cron job every day @00:00 etc.
#    QuotaMonitor     monitor nolog
#                     Start the Rx/Tx and Total Used Quota monitoring, but do not record the output to Syslog
#    QuotaMonitor     monitor nolog report=/tmp/mnt/xxx/Quota.csv once quota=
#                     Start the Rx/Tx and Total Used Quota monitoring, but do not record the output to Syslog, instead write the results to '/tmp/mnt/xxx/Quota.csv'
#                     NOTE: **WARNING" If you schedule it via cron with the 'once' option YOU MUST SPECIFY 'quota=' otherwise it may apply the Quota BLOCK!

The example shows that I added 25 devices but for testing a couple of devices should suffice.

e.g. Create the monitoring rules for three devices

./QuotaMonitor.sh   init    ip=hp-envy13,hp-envy14,sga5

Image fullsize

I then started the real-time monitoring, but to speed up testing, I altered the Quota DOWNLOAD limit to max 500KB in (60 seconds), and also requested that devices that didn't receive/transmit any data during the interval would not be displayed on screen in the fixed scrollable window.

./QuotaMonitor.sh    monitor   dlimit=500k   scroll=  ignorezero

So immediately I started a web browser session on my HP-Envy13 laptop and easily exceeded the 500KB/min limit so the device was QoS throttled, and the monitor displayed the reason 'QoS - Rx' and continue to report the QoS throttling.

I then obviously removed the QoS throttling from my laptop

./QuotaMonitor.sh ip=hp-envy13 unblockqos

v1.04 Quota Monitoring.....

 Quota Monitoring clients: hp-envy13 (10.88.8.111) UnBLOCKED

NOTE: The data is is not saved to disk (or SQL) at the moment, v1.03 allows use of the 'report=' option to save a .csv but the real-time data is only valid for as long as the iptables remain intact.

As usual all/any feedback appreciated.

 

[JDB]

Neat script/idea.

Would be most useful if it could apply a throughput limit rather than block to me.

So allow bursts of small downloads, but if someone does a large one it will apply a throughout limit so as to not hog all the bandwidth for a long time.
No idea how that could be applied, but you asked for feedback!


Sent from my iPhone using Tapatalk

 

[JDB]

Looks like ‘tc’ should allow you to enforce limits. Not sure if that will play nice with QOS if it’s enabled as I know it also uses tc


Sent from my iPhone using Tapatalk

 

[Martineau]

Looks like ‘tc’ should allow you to enforce limits. Not sure if that will play nice with QOS if it’s enabled as I know it also uses tc

Using the block is a quick and definite way of testing also inviting wider real-world application i.e. those who are on a fixed dataplan (say thu' a Dual-WAN 3G/4G Seconday connection) will expect the connection to be stopped rather than allow their allowance to continue to be drained albeit at a lower rate?

Clearly if the script cannot accurately detect the throughput/data used then bandwidth shaping etc. is going to be irrelevant.

However, if my script is proven to be able to monitor traffic in real-time, then I'm sure @FreshJR's expertise will prove useful.

 

[JDB]

Yea my use case was for example:
15 yr old regularly downloads large files (games etc) and it takes a lot of the bandwidth away from the rest of the household. Traditional/Fresh QOS should handle this to some degree, but I’d like to set a hard throughout limit on the user if they are doing a large download so I know there is still X Mbps available for everyone else to stream/surf/download.

As you say, I’m sure Fresh could suggest more on how to apply that limit if the need was triggered by your existing monitoring.


Sent from my iPhone using Tapatalk

 

[Martineau]

Yea my use case was for example:
15 yr old regularly downloads large files (games etc) and it takes a lot of the bandwidth away from the rest of the household. .

Well he would be a perfect example of the ideal candidate to be summarily blocked forthwith!

 

[RMerlin]

With the amount of really useful scripts you've cooked over the years, you should put them all on a Github repo, to make it easier for people to use/update them. Users would be able to update through one single wget command for instance. Just add a copyright header to each of them to claim legal ownership over these.

 

[Martineau]

QuotaMonitoring v1.02

Fix - When 'ip= remove' requested, include removal of the BLOCK rule (if it exists) as well as the monitoring rule.
Change - Rename 'ip= flush' option to 'ip= remove'
New - The Upload (Tx) Quota is now also monitored
New - Allow clients to be dynamically added/removed from a running monitor list using another instance of the script.
New - Add option 'ulimit=' to allow overriding of the default Upload (Tx) Quota.
New - Add option '['quota='['rx'|'tx'|'used']] to specify which quotas will be enforced (Actioned).

e.g. 'quota=Used' will only apply the Quota exceeded action to the Data Usage metric.
NOTE: specify 'quota=' to use the script in tracking mode to assist in identifing appropriate Quota limits.​

New - Add options 'actionrx=','actiontx=' and 'actionused=' to specify the script to run when Quota Limit exceeded.

e.g Rather than simply BLOCK the device, possibly schedule a couple of cron jobs to suspend for a limited period
or place device in a slow bandwidth shaping pool 'tc' command etc?​

New - Add 'nolog' option to suppress writing the records to Syslog
Change - Redesigned display colour scheme/layout, i.e include Upload (Tx) Quota limit.
Change - Remove option 'noblock' as this functionality is superceded by the new 'quota=' option.​

 

[skeal]

This sounds nice!!

 

[JDB]

This sounds nice!!

Yea I was thinking I might install it just to see it as it sounds awesome!


Sent from my iPhone using Tapatalk

 

[FreshJR]

Using the block is a quick and definite way of testing also inviting wider real-world application i.e. those who are on a fixed dataplan (say thu' a Dual-WAN 3G/4G Seconday connection) will expect the connection to be stopped rather than allow their allowance to continue to be drained albeit at a lower rate?

Clearly if the script cannot accurately detect the throughput/data used then bandwidth shaping etc. is going to be irrelevant.

However, if my script is proven to be able to monitor traffic in real-time, then I'm sure @FreshJR's expertise will prove useful.

It shouldn't interfere with AdpativeQOS if you avoid name conflicts with the existing structure.

AdaptiveQOS filters data into rate guaranteed classes based on packet marks.

To be subject to different limits you are always free to create a new class (using a different name than the existing classes), eg 1:100 instead of 1:10-1:17, and have that class ceiling limited with whatever bandwidth you deem appropriate.


With the new class created, you can filter begin filtering specific data into your own class, instead of the original destined AdaptiveQOS classes by overriding the Asus assigned mark with your own via iptables

(The iptables mark override method is easier than deleting and recreating TC rules in order to insert your rule at an earlier evaulation point. Rules evaluate from top to bottom and deleting/recreating the top rule is the only way to insert your rule before it).

Edit:
Example

##upload
iptables -D POSTROUTING -t mangle -o eth0 -s 192.168.2.100/32 -m mark --mark 0x40000000/0xc0000000 -j MARK --set-mark 0x403FFFFF
iptables -A POSTROUTING -t mangle -o eth0 -s 192.168.2.100/32 -m mark --mark 0x40000000/0xc0000000 -j MARK --set-mark 0x403FFFFF
tc class add dev eth0 parent 1:1 classid 1:100 htb prio 8 rate 1500kbit ceil 1500kbit
tc qdisc add dev eth0 parent 1:100 handle 100: sfq
tc filter add dev eth0 parent 1: protocol all prio 50 u32 match mark 0x403FFFFF 0xC03FFFFF flowid 1:100

##download
iptables -D POSTROUTING -t mangle -o br0 -d 192.168.2.100/32 -m mark --mark 0x80000000/0xc0000000 -j MARK --set-mark 0x803FFFFF
iptables -A POSTROUTING -t mangle -o br0 -d 192.168.2.100/32 -m mark --mark 0x80000000/0xc0000000 -j MARK --set-mark 0x803FFFFF
tc class add dev br0 parent 1:1 classid 1:100 htb prio 8 rate 1500kbit ceil 1500kbit
tc qdisc add dev br0 parent 1:100 handle 100: sfq
tc filter add dev br0 parent 1: protocol all prio 50 u32 match mark 0x803FFFFF 0xC03FFFFF flowid 1:100

Once TC is setup, all you have to do is add/remove iptables rules to enable/disable limiting.

 

[Martineau]

It shouldn't interfere with AdpativeQOS if you avoid name conflicts with the existing structure.

AdaptiveQOS filters data into rate guaranteed classes based on packet marks.

To be subject to different limits you are always free to create a new class (using a different name than the existing classes), eg 1:100 instead of 1:10-1:17, and have that class ceiling limited with whatever bandwidth you deem appropriate.


With the new class created, you can filter begin filtering specific data into your own class, instead of the original destined AdaptiveQOS classes by overriding the Asus assigned mark with your own via iptables

(The iptables mark override method is easier than deleting and recreating TC rules in order to insert your rule at an earlier evaulation point. Rules evaluate from top to bottom and deleting/recreating the top rule is the only way to insert your rule before it).

Edit:
Example

##upload
iptables -D POSTROUTING -t mangle -o eth0 -s 192.168.2.100/32 -m mark --mark 0x40000000/0xc0000000 -j MARK --set-mark 0x403FFFFF
iptables -A POSTROUTING -t mangle -o eth0 -s 192.168.2.100/32 -m mark --mark 0x40000000/0xc0000000 -j MARK --set-mark 0x403FFFFF
tc class add dev eth0 parent 1:1 classid 1:100 htb prio 8 rate 1500kbit ceil 1500kbit
tc qdisc add dev eth0 parent 1:100 handle 100: sfq
tc filter add dev eth0 parent 1: protocol all prio 50 u32 match mark 0x403FFFFF 0xC03FFFFF flowid 1:100

##download
iptables -D POSTROUTING -t mangle -o br0 -d 192.168.2.100/32 -m mark --mark 0x80000000/0xc0000000 -j MARK --set-mark 0x803FFFFF
iptables -A POSTROUTING -t mangle -o br0 -d 192.168.2.100/32 -m mark --mark 0x80000000/0xc0000000 -j MARK --set-mark 0x803FFFFF
tc class add dev br0 parent 1:1 classid 1:100 htb prio 8 rate 1500kbit ceil 1500kbit
tc qdisc add dev br0 parent 1:100 handle 100: sfq
tc filter add dev br0 parent 1: protocol all prio 50 u32 match mark 0x803FFFFF 0xC03FFFFF flowid 1:100

Once TC is setup, all you have to do is add/remove iptables rules to enable/disable limiting.

@FreshJR

Many thanks.

I didn't expect you to take time to write the rules, but I'm sure @JDB will be happy, and may now be able to pro-actively reduce the negative impact his son's downloads appear to cause

 

[FreshJR]

Sorry off topic

@JDB feel free to change the ceil of both video streaming && file transfers QOS categories if you don’t want the network to be in a saturated state.

 

[Martineau]

QuotaMonitoring v1.03

New - Allow QoS throttling option rather than BLOCKing the device using @FreshJR's rules
New - Add options 'initqos', 'qos' and 'ip= unblockqos' to support QoS

Essentially, 'initqos' ensures 'classid/flowid 1:100' rules are created.
To use the QoS throttling rather than the default BLOCK, specify 'monitor qos'
NOTE: The order of the applied action is External Script, QoS then BLOCK.​

New - Add option 'ip=auto' to retrieve all active LAN devices.
New - Add options 'ip=all unblock' and 'ip=all unblockqos'
Change - Improve 'status' request to show the number of devices monitored and

how many are BLOCKED or QoS throttled.​

New - Add option 'report=' so the interval results will be saved to disk as a '.csv' file
New - Add option 'once' to allow creating of the .csv by cron say every 24:00

e.g.

cru a QuotaMonitor24 "0 0 * * *" "/jffs/scripts/QuotaMonitor.sh monitor nolog report=/tmp/mnt/sdxx/Quota.csv quota= once"
cru a QuotaMonitor12 "0 12 * * *" "/jffs/scripts/QuotaMonitor.sh monitor nolog report=/tmp/mnt/sdxx/Quota.csv quota= once"​

Change - Minor display layout cganges e.g. timer countdown feedback....​

 

[Martineau]

QuotaMonitoring v1.04

Fix - If client is ALREADY 'Qos throttled' the QoS is needlessly reapplied rather than being reported as 'Currently QoS throttled'
New - Allow Quota Limit to be applied per client '/jffs/configs/QuotaLimits'

e.g.

 #Hostname      Rx     Tx     Cap
 HP-Envy13   300MB   20MB   100GB

New - Cosmetic scrolling window (use both options 'scroll= ignorezero' combination to see it in action!

i.e. Any new client that exceeds the Quota limit is highlighted and HELD at the top of the scroll window for easier real-time tracking notification!.​

Change - Display Monitoring Pause/Resume countdown time in HH:MM:SS format rather than Total seconds.​

 

[msalhab]

Hi @Martineau
is this script still maintained? on my RT-AC86U version 384.13 . ./QuotaMonitor.sh init is OK, however when i try to add a new IP I got:

iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.

Quota Monitoring clients: 192.168.1.153 (192.168.1.153 ) Added

then I check iptables -L ... the quota chain is empty

 

[Martineau]

Hi @Martineau
is this script still maintained? on my RT-AC86U version 384.13 . ./QuotaMonitor.sh init is OK, however when i try to add a new IP I got:

iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.

Quota Monitoring clients: 192.168.1.153 (192.168.1.153 ) Added

then I check iptables -L ... the quota chain is empty

I think I tweaked the script to accommodate possible NULL values in the report, and fixed the 'resetdel' request but never got around to publishing v1.05.

However, could you try

modprobe -sv xt_comment.ko

then retry the script?

 

[msalhab]

I think I tweaked the script to accommodate possible NULL values in the report, and fixed the 'resetdel' request but never got around to publishing v1.05.

However, could you try

modprobe -sv xt_comment.ko

then retry the script?

it looks better now, i can see the ip added to the iptables chain.
however when i launch the script with monitor and internal 10 , and i try to download a file, the amount of data used is much less than what has been actually downloaded

 

[Martineau]

when i launch the script with monitor and internal 10 , and i try to download a file, the amount of data used is much less than what has been actually downloaded

The script was written as an alternative to 'Traffic Analyzer' i.e. eliminates the need to accept TrendMicro's EULA.

Unfortunately, if enabled, HW Acceleration will seemingly affect the accuracy of the iptables packet statistics, but if HW Acceleration is OFF, the iptables statistics are as expected and my script is able to work as designed.

However, for HND models e.g. RT-AC86U - 'Runner=ENABLED/Flow cache=ENABLED' supplements its superior CPU throughput, so disabling HW Acceleration on these models (to use my script) is probably a sacrifice too far?

 

[msalhab]

The script was written as an alternative to 'Traffic Analyzer' i.e. eliminates the need to accept TrendMicro's EULA.

Unfortunately, if enabled, HW Acceleration will seemingly affect the accuracy of the iptables packet statistics, but if HW Acceleration is OFF, the iptables statistics are as expected and my script is able to work as designed.

However, for HND models e.g. RT-AC86U - 'Runner=ENABLED/Flow cache=ENABLED' supplements its superior CPU throughput, so disabling HW Acceleration on these models (to use my script) is probably a sacrifice too far?

thanks for the reply, sounds very logical