Restore Traffic Stats

[Jeffrey Young]

The traffic analyser database and the rstats file are two completely different and unrelated things.

Yeah, that is what I thought too. More of a hail mary

 

[RMerlin]

Found it in /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db

Traffic Analyzer (Trend Micro) is not the same thing as Traffic Monitor (rstats).

You guys need to make sure you are all talking about the same thing.

 

[sbsnb]

The issue is what when you stop rstats, rm /var/lib/misc/rstats*, rm /tmp/mnt/sda1/stats/tomato* (where I've configured stats to be saved), and then start rstats you will still see the past month's data in the UI. I assumed since TrafficAnalyzer.db had information that also dated from 3/1 onwards that this is where the data was coming from. If not, can someone please assist in finding the other data repository for rstats?

I hadn't been aware that the traffic analyzer database would be active with it turned off in the UI.

 

[ColinTaylor]

What do you get from this command:

nvram show | grep ^rstats

 

[sbsnb]

admin@RT-AX86U-AR28:/tmp# nvram show | grep ^rstats
rstats_bak=0
rstats_data=
rstats_enable=1
rstats_exclude=
rstats_new=0
rstats_offset=1
rstats_path=/mnt/sda1/stats/
rstats_stime=12
size: 77934 bytes (53138 left)

 

[ColinTaylor]

That looks fine. I just wanted to check that nothing there was corrupted.

I would change the Save frequency from 12 to 1 hour to avoid problems in the future though.

How did you "stop rstats" in post #43? If you were still able to switch between the daily and monthly Traffic Monitor displays and still see data then the rstats process must have still been running

 

[sbsnb]

I don't want to write to the USB a lot and wear it out.

I stopped rstats with pidof rstats and kill [pid]

 

[ColinTaylor]

I don't want to write to the USB a lot and wear it out.

Writing <2kB of data once an hour is a drop in the ocean compared to other write activity on the USB drive. Seriously, what kind of USB stick can't cope with writing 17MB of data a year.

I stopped rstats with pidof rstats and kill [pid]

I suggest you try again but this time confirm that the process was killed successfully with ps w | grep rstats.

 

[RMerlin]

you will still see the past month's data in the UI.

Please show us which router page you are looking at.

 

[sbsnb]

 

[ColinTaylor]

View attachment 40529

That screenshot is the Traffic Analyzer (TrafficAnalyzer.db) not the Traffic Monitor (rstats).

 

[sbsnb]

It appears so. I wasn't aware that there were two separate and distinct sources of traffic history maintained in parallel on the filesystem.

So when I say I'm losing my traffic history on upgrade, I'm talking about the Traffic Analyzer history. As far as I recall, that didn't used to be the case until 386.3_2. The aggregated history isn't very useful to me as my historical research usually consists of something like, "Did Netflix always consume 100 GB per month?"

It very well could be that it's not related to upgrading and the new behavior of trimming everything but the current month is just new and I haven't noticed for a couple of months. Anyway, I'd like to stop that. I don't have any problem with letting TrafficAnalzyer.db grow indefinitely. I have 40 MB free on /jffs. Even better if I created a link to my USB device with 64 GB of space and let that database be 500 MB if it wants to.

 

[ColinTaylor]

It appears so. I wasn't aware that there were two separate and distinct sources of traffic history maintained in parallel on the filesystem.

So when I say I'm losing my traffic history on upgrade, I'm talking about the Traffic Analyzer history. As far as I recall, that didn't used to be the case until 386.3_2. The aggregated history isn't very useful to me as my historical research usually consists of something like, "Did Netflix always consume 100 GB per month?"

It very well could be that it's not related to upgrading and the new behavior of trimming everything but the current month is just new and I haven't noticed for a couple of months.

So if you change the "Last date:" field to 16th March on the monthly Traffic Analyzer graph what does it look like?

 

[sbsnb]

Using sqllite to do a SELECT * FROM traffic shows that there is indeed no data prior to 3/1/2022 00:00.

 

[ColinTaylor]

OK. As you said that when you examined your backups of TrafficAnalyzer.db this truncation was taking place at the start or end of each month I would look for third-party scripts that run on that schedule. This truncation has not happened this month so perhaps it was a script that you had running in the past but not anymore.

 

[sbsnb]

OK. As you said that when you examined your backup files this truncation was taking place at the start or end of each month I would look for third-party scripts that run on that schedule. This truncation has not happened this month so perhaps it was a script that you had running in the past but not anymore.

I don't have anything installed other than those in my signature and custom DDNS and NTPMerlin. Unless one of those covertly trims TrafficAnalyzer.db, I don't think I have anything that could even be a suspect.

I suspect trimming doesn't happen right at the turn of the month as my backup from March 2nd has data from 2/1/2022 00:00, but my backup from 3/19 only has from 3/1/2022 00:00.

 

[ColinTaylor]

Sounds like this problem: http://www.snbforums.com/threads/data-missing-from-traffic-analyzer.74712/

The only auto-trim function that I'm aware of runs each hour (rather then each month) and just truncates the size of file when it exceeds a certain limit.

 

[sbsnb]

After pulling backups from my old RT-AC88U, it appears this has been going on for some time. What I don't understand is how many, many times in the past I have browsed to past months and even years of usage data.

I wonder if I can use sqlite to union all of my backups and then view that in the UI before the hourly process runs and destroys it again.

 

[ColinTaylor]

What are the sizes of your TrafficAnalyzer.db files?

 

[sbsnb]

The largest in my backups is about 12MB.

It looks like each hour's activity requires about 15,000 bytes (the way the db is written is NOT space efficient - some minor changes to the format of the data in the DB could cut that by 75%). 30MB should allow me to get to almost 3 months worth of data.