Extracting Real-time Stats

[Glenn Freidet]

Make sure you enable IPTraffic under Tools -> Other Settings. This will load the ipt_account module, and set it up to monitor your LAN subnet.

Yep, already doing that. I'll figure it out. Thanks Merlin.

 

[Glenn Freidet]

Yes. Asuswrt's httpd only allows one concurrent user.

Got it. Thanks.

 

[Glenn Freidet]

No idea what darkstat is, but I assume it's using its own web server and not Asus's, so it will be fine authentication-wise.

Must have been about 10 years since I last looked at NTOP + Netflow - sorry...

Darkstat is an Entware package.

OK. I'm all set. Final thanks.

 

[Glenn Freidet]

FYI, both cstats and /proc/net/ipt_account/lan worked out really well. I have more than enough of what I need. I really appreciate your help.

 

[ColinTaylor]

That good news Glenn. Would it be possible for you to explain how you extracted the information from cstats or what the record format is? Is it some sort of "standard" RRD format perhaps?

It's just that this question has been asked many times and as far as I know no one has worked out the format. Until now.

 

[Glenn Freidet]

That good news Glenn. Would it be possible for you to explain how you extracted the information from cstats or what the record format is? Is it some sort of "standard" RRD format perhaps?

It's just that this question has been asked many times and as far as I know no one has worked out the format. Until now.

Sure, no problem. I started with a Python script that I found somewhere. Then I used cstats.[ch] from the latest release and a hex editor. I'm not sure why, but the format of the file doesn't exactly match the code.

The output format is customized for my use, so you'll have to modify it to get what you need.

Please let me know if you make any changes that I might be able to use.

https://drive.google.com/file/d/0B7AAdqGpSE8ZS1ZoX1lMZkpyNGc/view?usp=sharing

 

[ColinTaylor]

That's great. Thank you very much.

Personally I don't have a use for it at the moment but I'm sure it will be helpful to others.

I've not been able to run your test program myself because your gzip function relies on Python 2.7+ I believe and I only have 2.6 on my Centos box.

The only pedantic thing I could spot in your program was on line #19 (Count of loaded day records) where the indentation is wrong. But that's nitpicking.

Now I just need to think of a use for this.... Hmmm.... Maybe if I rewrote it in Perl.....

 

[Glenn Freidet]

The only pedantic thing I could spot in your program was on line #19 (Count of loaded day records) where the indentation is wrong. But that's nitpicking.

Ah. Good catch.

 

[Zirescu]

Sure, no problem. I started with a Python script that I found somewhere. Then I used cstats.[ch] from the latest release and a hex editor. I'm not sure why, but the format of the file doesn't exactly match the code.

The output format is customized for my use, so you'll have to modify it to get what you need.

Please let me know if you make any changes that I might be able to use.

https://drive.google.com/file/d/0B7AAdqGpSE8ZS1ZoX1lMZkpyNGc/view?usp=sharing

Python script might have been from here: http://pastebin.com/bZxLpmQL

 

[ColinTaylor]

Python script might have been from here: http://pastebin.com/bZxLpmQL

Ah. That looks to be a slightly older version of this: https://github.com/emjelde/rstats/blob/master/RStatsHistory.py

(Which, together with RStatsOutput.py, were the only previously known programs that read the rstats file, but not the cstats)

 

[Glenn Freidet]

Ah. That looks to be a slightly older version of this: https://github.com/emjelde/rstats/blob/master/RStatsHistory.py

(Which, together with RStatsOutput.py, were the only previously know programs that read the rstats file, but not the cstats)

You're right, they look very similar, but I still don't remember where the cstats version came from. Weird.

I'll let you know when I'm done with the script for /proc/net/ipt_account/lan. For now, I'm playing with fprobe --> ntop and PRTG.

 

[wheelq]

Tried using cstats and...it fails:

python cstats.py /mnt/sda2/cstats/tomato_rstats_1cb72cc379c0.gz

>>> Warning!

Read 0 bytes.

Expected to read 2112 bytes.

Left to read 2112 bytes

 

[ColinTaylor]

You have to use it against a cstats file, not rstats. A cstats file is created when you enable "per IP monitoring" (IPTraffic).

 

[wheelq]

You have to use it against a cstats file, not rstats. A cstats file is created when you enable "per IP monitoring" (IPTraffic).

Screenshot attached. And the tomato_rstats_1cb72cc379c0.gz is 65 bytes in size all the time.
Also, the 'TrafficAnalyzer_Statistic.asp' is able to display all the stats... so either something is wrong or I'm missing something...

 

[wheelq]

Ok so I have tried those scripts. cstats.py -> fails. rstats.py -> works, but it only provides the In/Out data, no IPs assigned...

 

[ColinTaylor]

Screenshot attached. And the tomato_rstats_1cb72cc379c0.gz is 65 bytes in size all the time.
Also, the 'TrafficAnalyzer_Statistic.asp' is able to display all the stats... so either something is wrong or I'm missing something...

Did you select "Create or reset IPTraffic data files"?

 

[wheelq]

Yes I did. As you can see file is being created and I am able to reads the data, so it seems like this option creates rstat file with In/Out data only...just wondering where is the data stored with IPs and hosts etc...the one that is being used in the 'TrafficAnalyzer_Statistic.asp' section

 

[ColinTaylor]

Yes I did. As you can see file is being created and I am able to reads the data,

No I don't see that. You posted output using an rstats file, not a cstats file.

so it seems like this option creates rstat file with In/Out data only...just wondering where is the data stored with IPs and hosts etc...the one that is being used in the 'TrafficAnalyzer_Statistic.asp' section

The cstats file contains the per IP information, not rstats.

I don't use the Traffic Analyser but I think its data file is under /jffs. I believe it's some sort of SQL database file.

 

[wheelq]

Oh, ok. So, there are rstats and cstats, only after I have checked both:
"Create or reset data file" and "Create or reset IPTraffic data files".

Summary:
rstats contains info about In/Out only.
cstats contains info about In/Out per IP.

There are multiple python scripts on github, but only one works properly (attached) for cstats (screenshot attached).
Still no data there..I'll wait couple hours, maybe it is being cached or sth.

But now I have also realised those stats are still missing destination IP/Hosts which are visible on the Traffic Analyser site.
So I have checked the /jffs and there is a file called 'traffic.db' which contains all the information I needed and it is in sql format

So I guess there is no point in generating rstats and cstats then...

 

[RMerlin]

Traffic Analyzer is unrelated to rstats or cstats. It's based on Trend Micro's DPI engine, which is a completely separate thing.