Inaccuracy - Traffic Monitor

[ch4os]

[fixed] Inaccuracy - Traffic Monitor

************************************
This problem got fixed in new firmware, see here
************************************

Hey there,

I think I found a bug in the traffic monitor. Maybe this was already discussed ealier (in that case: sorry!), but here we go:

For testing I downloaded some stuff from usenet (around 2.80 GB).
I also made screenshots of the traffic stats before and after download (see attachments).

So, what I found out is: The per-device stats are way off compared to the global stats. If you look at the screenshots and compare global and per-device reception (PC2 was downloading), you get:

global reception:
5.62 GB - 3.27 GB = 2.35 GB

per-device reception:
3.95 GB - 2.67 GB = 1.28 GB

the real value should be something around 2.80 GB here, so even the global stats are wrong

I'm using an RT-AC66U and tried these firmwares, no difference:
RT-AC66U_3.0.0.4_270.26b, RT-AC66U_3.0.0.4_354.28-Beta1

The downloads from usenet were SSL encrypted, destination port is 443.

Thanks in advance for any help here!

Greets,
ch4os

 

[coxhaus]

I don’t know anything about ASUS routers or stats but if you have a file of a certain size to move, that file will need to be broken up and packetized to fit in a bunch of TCPIP packets to be transferred to the other machine. This packetizing will add overhead and increase the amount of data transferred.

 

[ch4os]

Hey,

thanks for your reply! That's true, segmentation and encapsulation in/of packets always produces overhead.

But here the router shows way less traffic than it actually routed, which just can't be right. And there also shouldn't be a difference between the global stats and the stats of just one IP being monitored

Something's not right here

 

[DrTeeth]

My router shows LOADS more traffic than there actually is/was, impossibly more.

You are lucky your results are so close to reality .

DrT

 

[ch4os]

Hmm... That's really too bad there are so much differences...

I just did a quick sum-up of the different per-device stats from this month.
This is the result I got:
- 68.64 GB (sum-up of all per-device stats per day)
- 98.46 GB (global stats of this month)

I'm not sure if this only happens with some connections or where the problem actually is. Do others with this firmware also have this problem?

Most of the PCs are connected on port 1 btw, with additional switches. WAN is connected to a DOCSIS 3 modem.

 

[RMerlin]

The traffic is counted by a Netfilter module. I just can't see any way how it could miscount it unless somehow a packet would go twice, or even not go at all through the Linux Netfilter stack. I have also been unable to reproduce this here, so I have no way to even being looking at this, sorry.

 

[DrTeeth]

It's a real puzzler and that's for sure.

Can you think of anyway one of us could supply data that would be of use?

TIA

DrT

 

[RMerlin]

It's a real puzzler and that's for sure.

Can you think of anyway one of us could supply data that would be of use?

TIA

DrT

Someone who has the know-how to play with iptables could keep an eye on the packet counts going through all chains while transferring a large file, and see if the packet counts do increase correctly all accross the Netfilter path, matching the large file transfer they just did. Packet and bandwidth count can be viewed with "iptables -L -v"

The ipt_account rules are located in the FORWARD chain:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  tun21  any     anywhere             anywhere            
3921K 4130M            all  --  eth0   br0     anywhere             anywhere            account: network/netmask: 192.168.10.0/255.255.255.0 name: lan 
3082K 1138M            all  --  br0    eth0    anywhere             anywhere            account: network/netmask: 192.168.10.0/255.255.255.0 name: lan 
6949K 5265M ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED 
    0     0 DROP       all  --  !br0   eth0    anywhere             anywhere            
 2251 98205 DROP       all  --  any    any     anywhere             anywhere            state INVALID 
    2   160 ACCEPT     all  --  br0    br0     anywhere             anywhere            
43751 2437K triggers   all  --  any    eth0    anywhere             anywhere            
 7834  567K TRIGGER    all  --  eth0   any     anywhere             anywhere            TRIGGER type:in match:0 relate:0 
 7795  565K ACCEPT     all  --  any    any     anywhere             anywhere            ctstate DNAT 
43751 2437K ACCEPT     all  --  br0    any     anywhere             anywhere

 

[ch4os]

Well... I tried around a bit further today.

I downloaded 2 GB from an FTP server and compared global and per-device stats. Here the result was correct, both showed around 2 GB additional traffic.

Then I tried another usenet download, this time on destination port 563 instead of 443 (just to see if the port might be the problem). Here the problem occured again:

usenet download (3132 MB) - destination port 563

global stats:
6819 MB - 4312 MB = 2507 MB new traffic

per-device stats:
4913 MB - 3592 MB = 1321 MB new traffic

I also did an "iptables -L -v" before and after the download:

before download:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
3359K 4952M            all  --  eth0   br0     anywhere             anywhere            account: network/netmask: 10.9.8.0/255.255.255.0 name: lan
1458K   60M            all  --  br0    eth0    anywhere             anywhere            account: network/netmask: 10.9.8.0/255.255.255.0 name: lan
4817K 5012M ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
    0     0 DROP       all  --  !br0   eth0    anywhere             anywhere
    1    40 DROP       all  --  any    any     anywhere             anywhere            state INVALID
  741 37544 ACCEPT     all  --  br0    br0     anywhere             anywhere
    0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/SYN limit: avg 1/sec burst 5
    0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/RST limit: avg 1/sec burst 5
    0     0 ACCEPT     icmp --  eth0   any     anywhere             anywhere            icmp echo-request limit: avg 1/sec burst 5
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere            ctstate DNAT
  218 12648 ACCEPT     all  --  br0    any     anywhere             anywhere

after download:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
5658K 8355M            all  --  eth0   br0     anywhere             anywhere            account: network/netmask: 10.9.8.0/255.255.255.0 name: lan
2453K  102M            all  --  br0    eth0    anywhere             anywhere            account: network/netmask: 10.9.8.0/255.255.255.0 name: lan
8111K 8457M ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
    0     0 DROP       all  --  !br0   eth0    anywhere             anywhere
    1    40 DROP       all  --  any    any     anywhere             anywhere            state INVALID
  842 42664 ACCEPT     all  --  br0    br0     anywhere             anywhere
    0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/SYN limit: avg 1/sec burst 5
    0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/RST limit: avg 1/sec burst 5
    0     0 ACCEPT     icmp --  eth0   any     anywhere             anywhere            icmp echo-request limit: avg 1/sec burst 5
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere            ctstate DNAT
  260 14976 ACCEPT     all  --  br0    any     anywhere             anywhere

If you compare those results you get:
8355 MB - 5012 MB = 3343 MB new traffic

3343 MB traffic could be correct (that would be around 211 MB usenet overhead).

So, I think the problem is not the iptables, it must be something else, probably inside the traffic monitor...

 

[RMerlin]

That confirms that the traffic does go through Netfilter, and properly hits the accounting rule. That means the issue could be at one of these levels:

1) Inside the ipt_account module (not properly processing all the traffic that hit its rule, not properly updating the traffic counters, etc...)
2) In the code that reads the traffic counters
3) In the code that processes the data before displaying it

The ipt_account counter can be manually read with this command:

cat /proc/net/ipt_account/lan

This will show all possible devices, with the amount of traffic the module counted since the last reboot.

 

[ch4os]

Ok, here we go
Did another usenet download (5.65 GB) with an unused IP (10.9.8.145), these are the results:

global stats in web menu:

Reception       Transmission     Total
5547.19 MB      94.41 MB         5641.61 MB

per-device in web menu:

Download        Upload           Total
3794.46 MB      70.52 MB         3864.98 MB

iptables before download:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
8630K   13G            all  --  eth0   br0     anywhere             anywhere            account: network/netmask: 10.9.8.0/255.255.255.0 name: lan
3763K  165M            all  --  br0    eth0    anywhere             anywhere            account: network/netmask: 10.9.8.0/255.255.255.0 name: lan
  12M   13G ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
    0     0 DROP       all  --  !br0   eth0    anywhere             anywhere
   80  3824 DROP       all  --  any    any     anywhere             anywhere            state INVALID
 1705 86552 ACCEPT     all  --  br0    br0     anywhere             anywhere
    1    48 ACCEPT     tcp  --  eth0   any     anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/SYN limit: avg 1/sec burst 5
    0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/RST limit: avg 1/sec burst 5
    0     0 ACCEPT     icmp --  eth0   any     anywhere             anywhere            icmp echo-request limit: avg 1/sec burst 5
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere            ctstate DNAT
 5506  322K ACCEPT     all  --  br0    any     anywhere             anywhere

iptables after download:

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  13M   19G            all  --  eth0   br0     anywhere             anywhere            account: network/netmask: 10.9.8.0/255.255.255.0 name: lan
5548K  239M            all  --  br0    eth0    anywhere             anywhere            account: network/netmask: 10.9.8.0/255.255.255.0 name: lan
  18M   19G ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
    0     0 DROP       all  --  !br0   eth0    anywhere             anywhere
   80  3824 DROP       all  --  any    any     anywhere             anywhere            state INVALID
 1713 86968 ACCEPT     all  --  br0    br0     anywhere             anywhere
    1    48 ACCEPT     tcp  --  eth0   any     anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/SYN limit: avg 1/sec burst 5
    0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/RST limit: avg 1/sec burst 5
    0     0 ACCEPT     icmp --  eth0   any     anywhere             anywhere            icmp echo-request limit: avg 1/sec burst 5
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere            ctstate DNAT
 5598  328K ACCEPT     all  --  br0    any     anywhere             anywhere

/proc/net/ipt_account/lan (before download):

ip = 10.9.8.0 bytes_src = 495260755 447861212 46705576 664548 29419 packets_src = 7014037 6830112 172560 11075 290 bytes_dst = 21456131752 21406659832 48791304 665040 15576 packets_dst = 14980197 14781052 187800 11084 261 time = 0
ip = 10.9.8.2 bytes_src = 84018113 83720721 152 297240 0 packets_src = 1913004 1908048 2 4954 0 bytes_dst = 6665108975 6664811823 152 297000 0 packets_dst = 4483168 4478216 2 4950 0 time = 1
ip = 10.9.8.11 bytes_src = 14118062 14117758 304 0 0 packets_src = 54655 54651 4 0 0 bytes_dst = 180265751 180265447 304 0 0 packets_dst = 134956 134952 4 0 0 time = 505
ip = 10.9.8.50 bytes_src = 1047754 1045322 2432 0 0 packets_src = 15786 15754 32 0 0 bytes_dst = 638138 634922 2432 784 0 packets_dst = 12115 12069 32 14 0 time = 0
ip = 10.9.8.51 bytes_src = 209886 208670 1216 0 0 packets_src = 2042 2026 16 0 0 bytes_dst = 112977 111761 1216 0 0 packets_dst = 1956 1940 16 0 0 time = 6
ip = 10.9.8.145 bytes_src = 37833 37113 0 720 0 packets_src = 190 178 0 12 0 bytes_dst = 43312 42592 0 720 0 packets_dst = 185 173 0 12 0 time = 0

/proc/net/ipt_account/lan (after download):

ip = 10.9.8.0 bytes_src = 569243055 521839852 46705576 668208 29419 packets_src = 8799510 8615524 172560 11136 290 bytes_dst = 27754999936 27705524356 48791304 668700 15576 packets_dst = 19208098 19008892 187800 11145 261 time = 0
ip = 10.9.8.2 bytes_src = 84049962 83750710 152 299100 0 packets_src = 1913159 1908172 2 4985 0 bytes_dst = 6665252403 6664953391 152 298860 0 packets_dst = 4483344 4478361 2 4981 0 time = 2
ip = 10.9.8.11 bytes_src = 14124284 14123980 304 0 0 packets_src = 54699 54695 4 0 0 bytes_dst = 180274547 180274243 304 0 0 packets_dst = 134995 134991 4 0 0 time = 227
ip = 10.9.8.50 bytes_src = 1050638 1048206 2432 0 0 packets_src = 15841 15809 32 0 0 bytes_dst = 640710 637494 2432 784 0 packets_dst = 12164 12118 32 14 0 time = 1
ip = 10.9.8.51 bytes_src = 213318 212102 1216 0 0 packets_src = 2100 2084 16 0 0 bytes_dst = 115797 114581 1216 0 0 packets_dst = 2009 1993 16 0 0 time = 3
ip = 10.9.8.145 bytes_src = 73975746 73973226 0 2520 0 packets_src = 1785351 1785309 0 42 0 bytes_dst = 6298741793 6298739273 0 2520 0 packets_dst = 4227734 4227692 0 42 0 time = 0

all other IPs in that /24 subnet didn't change during my test, so I left it out

 

[RMerlin]

This is very useful information - thanks you very much for taking the time to compile them! I will have to analyze it to determine at which level the data suddenly becomes wrong. If I remember correctly ipt_account even allows you to load your own set of data, so I could try loading your values, which will help in at least reproducing the issue. I'll post back once I get any new info.

 

[DrTeeth]

Though I have mentioned what follows to Merlin, I am positing it here as it is pertinent.

I had my N66u reporting I had downloaded 68 GB in less than an hour during the time my ISP meters/monitors data usage. No such use was monitored by them. On another occasion, MY pc was deemed to have d/l 21GB, whist my ISP's figure was 4GB for all users in the house (including two teenage YouTube fans).

My monthly allowance for peak use is 50GB and is only rarely exceeded. The per IP would be very useful to me as I would then know which of my children to blame for excessive usage.

The usage seems to tally at very low figures - soon after a reboot then seem to get more out of kilter as time progresses.

Unfortunately, troubleshooting this is beyond my Linux skills .

 

[RMerlin]

Think I spotted at least one part of the problem. The webend code was handling byte counters as 32-bit values. That means it was unable to properly report byte counters beyond 4 GB (or 2 GB depending on whether it was using signed or unsigned values).

I'll need to go through the whole data path to ensure those get handled as 64-bit values throughout the whole chain.

EDIT:

Changing some of the webend backend code to 64-bit.

Before the change:

iptraffic=[ ['192.168.10.10', 2003774497, 73975746, 4227692, 1785309, 0, 0, 42, 42, 0, 0]];

After the switch to 64-bit:

iptraffic=[ ['192.168.10.10', 6298741793, 73975746, 4227692, 1785309, 0, 0, 42, 42, 0, 0]];

Getting pretty close now

 

[ch4os]

There's also another strange thing, maybe this is related to this problem, I'm not sure. I also didn't have the time so far to troubleshoot that any further.

In my household we have another notebook (my parents are using this from time to time) which hasn't been shown up in the per-IP stats for 3 days now. They don't use the internet that much but I asked them and they assured me they were browsing some websites just yesterday.

Not sure why there's no traffic for that notebook then, I set static DHCP leases for every PC.

BTW: Thank you so much RMerlin for taking the time to make this firmware even better than it is already! I recently bought this router and I think I wouldn't have if I didn't find your custom firmware earlier. There were some mixed reviews about this router on amazon.de, mainly pointing out weaknesses in the stock firmware

 

[JoeSchmoe007]

There's also another strange thing, maybe this is related to this problem, I'm not sure. I also didn't have the time so far to troubleshoot that any further.

In my household we have another notebook (my parents are using this from time to time) which hasn't been shown up in the per-IP stats for 3 days now. They don't use the internet that much but I asked them and they assured me they were browsing some websites just yesterday.

Not sure why there's no traffic for that notebook then, I set static DHCP leases for every PC.

BTW: Thank you so much RMerlin for taking the time to make this firmware even better than it is already! I recently bought this router and I think I wouldn't have if I didn't find your custom firmware earlier. There were some mixed reviews about this router on amazon.de, mainly pointing out weaknesses in the stock firmware

Just a crazy thought: does this laptop show up in the list of connected clients? What if it connects to some other router (open WiFi at neighbors location for instance)?

 

[ch4os]

Just a crazy thought: does this laptop show up in the list of connected clients? What if it connects to some other router (open WiFi at neighbors location for instance)?

Haha, good point. I'm pretty sure I set the notebook to ask before joining new networks first but I can't guarantee my parents didn't accidentally still chose another network. The DHCP lease list doesn't show the notebook but they didn't use it for 24hs so the lease time would be expired anyway; and system log doesn't go back that far due to router reboot unfortunately.

I'll check WiFi settings when I'm back home, thanks for that idea

*EDIT*
Ok, just checked the notebook by myself and it instantly sent a DHCP request and traffic showed up in the webend. Not sure what went wrong then... Maybe they really connected to another network... I think I'm gonna ignore that "problem" for now

 

[RMerlin]

More 64-bit related fixes were done tonight, this time to the cstats binary itself. Now I am able to transfer 5 GB of data in a short period of time, and the router will be able to properly keep track of it.

In my test there was a small percentage of difference between global and per device traffic, but it was the exact same percentage when I did the same transfer a second time, so I suspect this was some protocol overhead that iptables doesn't count but the global monitoring code does.

While at it I want to take a look at the random issues people often experience when trying to (re)create their data files. I think I got an idea as to where the issue lies.

 

[RMerlin]

Think I spotted one last cause of mis-calculation that could occur under certain circumstances. Once I figured out to trigger it, I was able to transfer 5 GB, trigger the situation, and end up with 30 GB of total traffic. I just need to finalize a fix for this issue, and then I can take a look at the quirky new file creation.

I have to say, the code handling the Per device monitoring had a lot of bugs here and there, despite having been written nearly a year ago for Tomato. I suspect it never got fully debugged at the time when its author switched the internal data management to use linked lists, and suddenly stopped all Tomato development.

 

[ch4os]

New firmware available!
In case someone here missed it - like me - I was only subscribed to this topic here

Thanks Merlin!