Restore Traffic Stats
- Thread starter sbsnb
- Start date
Jeffrey Young
Very Senior Member
I am co
rstat is associated with Traffic Monitor, not Traffic Analyzer. I am confused now. From your last few posts, it seems to be traffic analyzer that you actually wanted.
rstat is associated with Traffic Monitor, not Traffic Analyzer. I am confused now. From your last few posts, it seems to be traffic analyzer that you actually wanted.
My preference is to have Traffic Analyzer data for as long as I want to keep it. I am assuming since that's Trend Micro code there's no way to change its behavior. So, not being able to achieve my first preference, I would like to join my disjointed collection of rstats data from the past 2 years (I think that's the limit of RS01) and at least have that data available.
After examining your code and the rstats files, it looks to be fairly easy for me to manually "merge" my backups by manually creating my own rstats file in a hex editor. I wouldn't want to do it on a regular basis, but as a one-time exercise it shouldn't be too bad.
If I understand the format right, it just looks like the monthly data starts at offset 0x5F9 and is two bytes: the decimal month (e.g., 06 for July), and the year since 1900 (e.g., 2021 is 0x79). Then comes the downloaded byes in little endian (e.g., 1GB would be recorded as 0x00CA9A3800000000), followed by the upload bytes in the same format. So I would enter 1GB upload and 1GB download for July 2021 as:
If I understand the format right, it just looks like the monthly data starts at offset 0x5F9 and is two bytes: the decimal month (e.g., 06 for July), and the year since 1900 (e.g., 2021 is 0x79). Then comes the downloaded byes in little endian (e.g., 1GB would be recorded as 0x00CA9A3800000000), followed by the upload bytes in the same format. So I would enter 1GB upload and 1GB download for July 2021 as:
Code:
00 06 79 00 00 00 00 00 00 CA 9A 38 00 00 00 00 00 CA 9A 38 00 00 00 00
Last edited:
Upon further examination, I think my rstats files had complete daily data, but the Excel script didn't pick up on them because it presumes them to be in chronological order in the rstats file. It doesn't appear that the daily data are in chronological order. For example, from my rstats backup:
Translated to human-readable:
However, the Excel sheet only picks up on the data from 3/28-3/31.
What's also interesting is the disparity between rstats and Traffic Analyzer. For example, rstats reports 13,104,873,056 bytes downloaded on 3/30. However, Traffic Analyzer reports 12,857,189,520 bytes downloaded on 3/30.
Code:
1C 02 7A 00 00 00 00 00 52 D6 2B 91 02 00 00 00 4E AE 0E 1A 00 00 00 00
1D 02 7A 00 00 00 00 00 5F 6E D5 97 03 00 00 00 5E 63 C6 1D 00 00 00 00
1E 02 7A 00 00 00 00 00 60 7E 1C 0D 03 00 00 00 C9 10 15 B9 03 00 00 00
1F 02 7A 00 00 00 00 00 E8 B1 B0 9B 00 00 00 00 EF 95 AC A3 08 00 00 00
1D 00 7A 00 00 00 00 00 B5 36 FC EB 03 00 00 00 DC 79 FB 20 00 00 00 00
1E 00 7A 00 00 00 00 00 01 F3 62 A1 02 00 00 00 DF AD 08 5C 00 00 00 00
...Translated to human-readable:
Code:
3/28/2022 11025503826 437169742
3/29/2022 15432248927 499540830
3/30/2022 13104873056 15990067401
3/31/2022 2612048360 37105735151
1/29/2022 16844076725 553351644
1/30/2022 11297551105 1544072671
...However, the Excel sheet only picks up on the data from 3/28-3/31.
What's also interesting is the disparity between rstats and Traffic Analyzer. For example, rstats reports 13,104,873,056 bytes downloaded on 3/30. However, Traffic Analyzer reports 12,857,189,520 bytes downloaded on 3/30.
Last edited:
Jeffrey Young
Very Senior Member
That is entirely possible. It has been a couple of years since I wrote that code. I wrote it with Excel 2007 Pro, which does not have an unsigned long data type hence why some of the code is the way it is (newer versions of Excel has a "long long" data type for an unsigned long variables. I probably did write it for sequential dates as it is the most logical to assume. For myself, I still use it to transfer my usage to an excel sheet monthly .
I am glad you were able to get the data out. I've attached the structure of the file.
Now, that said, is it the Traffic Monitor data you are after. From one of your posts above, I got the impression that the aggregate data is not want you wanted, but rather data by application, which would be traffic analyzer. Or is your interest in both now?
I am glad you were able to get the data out. I've attached the structure of the file.
Now, that said, is it the Traffic Monitor data you are after. From one of your posts above, I got the impression that the aggregate data is not want you wanted, but rather data by application, which would be traffic analyzer. Or is your interest in both now?
Code:
# RStats log format:
# First four bytes are magic string that indicate version
# Next four bytes are empty, presumably for QWORD alignment
# Next 24 bytes are empty, for future usage?
# Next are 61 3-tuples for day-level stats,
# contains date, download and upload totals (8 byte/value)
# Next is one-byte counter that should match actual number of previous 3-tuples
# Next seven bytes are empty, presumably for QWORD alignment
# Next 24 bytes are empty, again for future usage?
# Next are 12 3-tuples for month-level stats,
# contains date, download and upload totals (8 byte/value)
# Version 1 logfiles hold another 12 month-level stat entries
# Last is one-byte counter that should match actual number of previous 3-tuples
year = ((num >> 16) & 0xFF) + 1900
month = ((num >> 8) & 0xFF) + 1
day = (num & 0xFF) or 1Jeffrey Young
Very Senior Member
Now that you have pointed that out, I will have to go back and look at it again. I believe that if you use the option 1, the script should just dump the rstat file as is, without looking at any chronological dates. That is option 2, which I done to keep a running year over year data tracking. Option 1 just simply reads an rstat file and dumps it to a pre built template from within the excel sheet.
Jeffrey Young
Very Senior Member
Would you mind PM me your rstat file? I would like to use it as a test file. Albeit, I may be a few weeks before I can look at it again.
I will PM the data.
It looks like once the rstats daily data reaches the limit it just rolls over and starts overwriting the oldest daily data and advances from there.
I was able to put everything together by cutting/pasting 24-byte blocks between rstats files in a hex editor.
The only hiccup I noticed was that if you
It looks like once the rstats daily data reaches the limit it just rolls over and starts overwriting the oldest daily data and advances from there.
I was able to put everything together by cutting/pasting 24-byte blocks between rstats files in a hex editor.
The only hiccup I noticed was that if you
killall -SIGTERM rstats it does write out all its files in /var/lib/misc, but when you start rstats again it has "lost" all the "Last 24 hours" data. It picks up all the other data from past days and months, but today's data up until the time the process was killed is lost.
PR3MIUM
Senior Member
LOL -> FAIL , "killall" wont work
It worked fine.LOL -> FAIL , "killall" wont work
Code:
admin@RT-AX86U-AR28:/var/lib/misc# killall -SIGTERM rstats
admin@RT-AX86U-AR28:/var/lib/misc# pidof rstats
admin@RT-AX86U-AR28:/var/lib/misc#I have combined the data from three different rstats backups into one rstats file and they were successfully loaded in the UI.
It's actually really easy once you understand the RS01 file format.
The issue with losing today's stats happens even if you don't do anything with the data files. Just look at your "Last 24 hours" data and then execute:
Code:
killall -SIGERM rstats
service start_rstatsNow go look at your "Last 24 hours" and see that it's empty.
ColinTaylor
Part of the Furniture
Yes it does. Maybe you don't know what the killall command is. LOLLOL -> FAIL , "killall" wont work
That's just cosmetic as the "last 24 hours" graph data is held in memory. The data is still recorded in the rstats file up until the last save - which is why you should set the save frequency to 1 hour.
I thought it was to kill any processes matching the name given as the argument?
I wish it would save that state, too. Of course it's not a big deal but it is a little annoying.
EDIT - Sorry, I didn't see that comment wasn't directed at me.
I found a python script that spits out comma-separated human-readable data from an rstats file in case anyone is interested. It can be handy if you want to keep data longer term than the 24 months that the rstats format will allow.
github.com
GitHub - VREMSoftwareDevelopment/rstats-cstats: Tomato USB Python scripts to dump RStats and CStats information
Tomato USB Python scripts to dump RStats and CStats information - VREMSoftwareDevelopment/rstats-cstats
github.com
Jeffrey Young
Very Senior Member
That is what I am thinking as well. I am interested in looking at your rstat file to see what value is written in the byte the specifies how many valid entries there actually are. I suspect it is 3, which is why my script only printed out the first three lines. The rest of the entries were considered stale and was a happy happenstance for you that they were there.
Jeffrey Young
Very Senior Member
I found a python script that spits out comma-separated human-readable data from an rstats file in case anyone is interested. It can be handy if you want to keep data longer term than the 24 months that the rstats format will allow.
GitHub - VREMSoftwareDevelopment/rstats-cstats: Tomato USB Python scripts to dump RStats and CStats information
Tomato USB Python scripts to dump RStats and CStats information - VREMSoftwareDevelopment/rstats-cstats
I've seen this script. I tried to port it at one time to Windows using a windows python compiler, but had no luck. It is what prompted me to write the Excel script for windows.
Last edited:
I was able to run it using python under cygwin.
Jeffrey Young
Very Senior Member
I'll have to try again. I wrote the option 2 portion of my script to keep a month over month and day over day beyond the 61 day / 24 months for long term tracking of my usage to make sure I am getting bang for the buck.
@sbsnb, I looked at your rstat file, and the byte value representing the actual number of days of data available was actually 4. That is why my script stopped where it did. It considered the rest of the entries as stale/invalid.
Jeffrey Young
Very Senior Member
Judging on the rstats file format, it would only require about 88 kb to store a decade's worth of daily and monthly stats uncompressed and about half that or less gzipped. Changing the code to allow that is trivial, but what I don't know is if there are other considerations I may not be aware of. Is the graphing library used in the UI sophisticated enough to handle longer datasets, e.g., will it add scrollbars or have multiple pages instead of cramming it all in one page?
Similar threads
Similar threads
Similar threads
-
Backup and Restore Procedures for Diversion and SkyNet
- Started by dragon84
- Replies: 3
-
RT-AX86U Pro - no auto-reconnect after WAN connection restore?
- Started by Rici
- Replies: 3
-
-
Using VPN Director to route only torrent traffic through VPN
- Started by goldnet
- Replies: 9
-
Asus RT-AX58U v2 Merlin: Route all traffic to Pi-hole (running directly on router itself)
- Started by hamensman
- Replies: 12
-
How can I monitor all Ip traffic from my wyze camera in Merlin?
- Started by lenovomen
- Replies: 2
-
-
-
-