Traffic Analyzer-Statistics: database erased very frequently

[JIPG]

I have noticed with the last update to 3004.388.9 FW (but probably was the same with 388.8_4) that the statistics graphics shown under Traffic Analyzer-Statistics, do not last too much and the data is lost. The data is reseted sometimes in a couple of days or sometimes even two or three times a day.
I have no reboots nor memory problems:

Traffic history location in settings is in its default location: RAM.
I have tried switching it OFF and ON again to reset the module, but to no avail.
Taking a look on the log section, I have found the following entries that occur at the same time the data is lost:

Apr 14 14:00:20 BWDPI: [BWANA] Revert process (pid=24381)
Apr 14 14:00:20 BWDPI: [sql_remove_journal(74)] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db-journal
Apr 14 14:00:20 BWDPI: [BWANA] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.dbApr 17 00:48:42 BWDPI: [BWMON] Revert process (pid=1144)
Apr 16 15:00:20 BWDPI: [BWANA] Revert process (pid=20115)
Apr 16 15:00:20 BWDPI: [sql_remove_journal(74)] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db-journal
Apr 16 15:00:20 BWDPI: [BWANA] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db
Apr 16 17:00:20 BWDPI: [BWANA] Revert process (pid=2116)
Apr 16 17:00:20 BWDPI: [sql_remove_journal(74)] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db-journal
Apr 16 17:00:20 BWDPI: [BWANA] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db
Apr 16 20:00:20 BWDPI: [BWANA] Revert process (pid=23741)
Apr 16 20:00:20 BWDPI: [sql_remove_journal(74)] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db-journal
Apr 16 20:00:20 BWDPI: [BWANA] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db
Apr 19 12:00:20 BWDPI: [BWANA] Revert process (pid=18220)
Apr 19 12:00:20 BWDPI: [sql_remove_journal(74)] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db-journal
Apr 19 12:00:20 BWDPI: [BWANA] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db
Apr 25 19:00:20 BWDPI: [BWANA] Revert process (pid=17814)
Apr 25 19:00:20 BWDPI: [sql_remove_journal(74)] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db-journal
Apr 25 19:00:20 BWDPI: [BWANA] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db
Apr 25 22:00:20 BWDPI: [BWANA] Revert process (pid=7432)
Apr 25 22:00:20 BWDPI: [sql_remove_journal(74)] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db-journal
Apr 25 22:00:20 BWDPI: [BWANA] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db
Apr 26 14:00:20 BWDPI: [BWANA] Revert process (pid=28034)
Apr 26 14:00:20 BWDPI: [sql_remove_journal(74)] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db-journal
Apr 26 14:00:20 BWDPI: [BWANA] remove /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db

It seems that a process in the router is deleting the database. As you can see, there is no rule in the timing to erase it ...
Is there any way (e.g. setting or command) that can allow me to have it longer?

 

[FTC]

I have noticed with the last update to 3004.388.9 FW (but probably was the same with 388.8_4) that the statistics graphics shown under Traffic Analyzer-Statistics, do not last too much and the data is lost. The data is reseted sometimes in a couple of days or sometimes even two or three times a day.
I have no reboots nor memory problems:

View attachment 65295
Traffic history location in settings is in its default location: RAM.
I have tried switching it OFF and ON again to reset the module, but to no avail.
Taking a look on the log section, I have found the following entries that occur at the same time the data is lost:


It seems that a process in the router is deleting the database. As you can see, there is no rule in the timing to erase it ...
Is there any way (e.g. setting or command) that can allow me to have it longer?

Hi. It seems that I am facing the same problem (using Merlin 3004_388.9_2 fw). Did you find a solution ?

 

[JIPG]

Hi. It seems that I am facing the same problem (using Merlin 3004_388.9_2 fw). Did you find a solution ?

No, the database is erased randomly, sometimes 2-3 times a day, and sometimes every 1-2 days. There is no specic problem of available memory, but maybe ASUS wants to be safe in keeping the free memory available for other uses, althouh this makes a litle bit useless this function. Perhaps your problem with the graphs of AiProtection is similar, having the database erased daily.

 

[FTC]

No, the database is erased randomly, sometimes 2-3 times a day, and sometimes every 1-2 days. There is no specic problem of available memory, but maybe ASUS wants to be safe in keeping the free memory available for other uses, althouh this makes a litle bit useless this function. Perhaps your problem with the graphs of AiProtection is similar, having the database erased daily.

Thanks for the info. The problem is not related to the AiProtection graphical glitch that I reported in another thread, since the AiProtection db file was not being erased by the system.
This deletion of the traffic analyzer database seems more like caused by a related failing process, and in my case looks like related to volume of the Traffic database under /jffs/.sys/TrafficAnalyzer, i.e once it gets very big, some kind of error is produced that the system decides to 'restart' the database. Does not look normal.. Seems more like a bug somewhere.

 

[JIPG]

Thanks for the info. The problem is not related to the AiProtection graphical glitch that I reported in another thread, since the AiProtection db file was not being erased by the system.
This deletion of the traffic analyzer database seems more like caused by a related failing process, and in my case looks like related to volume of the Traffic database under /jffs/.sys/TrafficAnalyzer, i.e once it gets very big, some kind of error is produced that the system decides to 'restart' the database. Does not look normal.. Seems more like a bug somewhere.

Do you have the same entries in the log?. In my case, the database size was small, so It has no to much sense.

 

[FTC]

Do you have the same entries in the log?. In my case, the database size was small, so It has no to much sense.

Yes, same entries. Then it could be related to some event that triggers the error,.. a bit more unlikely in my configuration so that the DB has had more time to grow. At one point I suspected that my /jffs/ nvram was flaky and whenever the growing files 'stepped' over the bad cells, the error was produced, but it seems strange not seeing a log entry with a write or read error in that case...

 

[Crimliar]

Okay guys I'm confused!

It's not something that I really bother with, but if you need/want it you can change the storage settings to a usb drive from the monitoring settings tab. *Only three days of data, I presume because of the Merlin update, and that I just let it sit in RAM.

 

[FTC]

Okay guys I'm confused!
View attachment 65510
It's not something that I really bother with, but if you need/want it you can change the storage settings to a usb drive from the monitoring settings tab. *Only three days of data, I presume because of the Merlin update, and that I just let it sit in RAM.

Thanks for the tip, however there is no 'monitoring settings' tab in the firmware for RT-AX88U. (I do not remeber to have seen that tab ever for this router) . Maybe there is a way through nvram however... can you run 'nvram show | grep whatever-your-TrafficAnalyzer-path-is' and post the results ?

 

[JIPG]

Thanks for the tip, however there is no 'monitoring settings' tab in the firmware for RT-AX88U. (I do not remeber to have seen that tab ever for this router) . Maybe there is a way through nvram however... can you run 'nvram show | grep whatever-your-TrafficAnalyzer-path-is' and post the results ?

This monitoring settings tab is new in the last beta on the 3006 branch. But in our case (3004 branch), it is in Tools - Other settings, Traffic History location. I have it selected to "RAM (default)", but no reboot in my case during the time the database has been erased.

 

[Crimliar]

Before doing anything else, are you looking in the correct location on the GUI?

 

[JIPG]

Before doing anything else, are you looking in the correct location on the GUI?
View attachment 65511

See my previous post. In our firmware branch (3004.388.9_2), this part of the GUI is in Tools- Other settings.

 

[dave14305]

OP is talking about data on the Statistics tab. Traffic Monitor is a different thing altogether.

 

[FTC]

OP is talking about data on the Statistics tab. Traffic Monitor is a different thing altogether.

Yes, that is right. The location to be configured ubder tools->Other settings is for Asus traffic data. What we want to change is the TrafficAnalyzer database location (Trendmicro), that does not seem like being configurable for the RT-AX88U.

 

[JIPG]

OP is talking about data on the Statistics tab. Traffic Monitor is a different thing altogether.

Dave14305 is right. I referred originally to that:

 

[dave14305]

Yes, that is right. The location to be configured ubder tools->Other settings is for Asus traffic data. What we want to change is the TrafficAnalyzer database location (Trendmicro), that does not seem like being configurable for the RT-AX88U.

You can set bwdpi_ana_path in nvram to move it elsewhere. This would be the directory path including the filename.

asuswrt-merlin.ng/release/src/router/bwdpi_source/asus_include/bwdpi_common.h at 050c38208b6e7f5d51101c20ddb950e30a99fbe0 · RMerl/asuswrt-merlin.ng

Third party firmware for Asus routers (newer codebase) - RMerl/asuswrt-merlin.ng

github.com

Also, the DB size seems to have a 14MB limit.
#define BWDPI_ANA_DB_SIZE "14336"

 

[FTC]

You can set bwdpi_ana_path in nvram to move it elsewhere. This would be the directory path including the filename.

asuswrt-merlin.ng/release/src/router/bwdpi_source/asus_include/bwdpi_common.h at 050c38208b6e7f5d51101c20ddb950e30a99fbe0 · RMerl/asuswrt-merlin.ng

Third party firmware for Asus routers (newer codebase) - RMerl/asuswrt-merlin.ng

github.com

Also, the DB size seems to have a 14MB limit.
#define BWDPI_ANA_DB_SIZE "14336"

Well, I just finished testing this just to see if moving the TrafficAnalyzer db file to my flashdrive would cese the spurious database deletions, which would have indicated some problem with my router's /jffs/ nvram partition. However, also in this case the TrafficAnalyzer decides to erase the database without reason, so I definetly think now that this is some kind of bug in recent versions of the GPL on which Merlin is based (experienced this with 388_9 and 388_9_2) .
Remember that to verify this bug you must see in the logs similar lines to :

May 10 11:00:21 BWDPI: [BWANA] Revert process (pid=31415)
May 10 11:00:21 BWDPI: [sql_remove_journal(74)] remove /tmp/mnt/sdb1/TrafficAnalyzer.db-journal
May 10 11:00:21 BWDPI: [BWANA] remove /tmp/mnt/sdb1/TrafficAnalyzer.db

 

[CaptnDanLKW]

FWIW, I've had this issue (feature?) with Traffic-Statistics for many years across multiple device models. It's not a file size thing, mine clears, minimally at the start of every month, or 30days from last reboot, or something like that. My current RT-AX86U Pro has this issue. My previous RT-AC86U had this issue, but my RT-AC68U (original - not the last version with the totally different hardware) did NOT - statistics were retained even after a reboot. Asus has no intention of fixing this, so I gave up a long time ago. Sad, because there is value in knowing how traffic breaks down by app.
I use the vnStat addon and the other Traffic statistics to feed my data analysis addiction.

 

[donat76]

RT-AX88U

Had httpd crash with kernel tainted... when switching between Traffic Analyser history and real time traffic. I haven't tracked down exactly what happens but when I switch to monthly or weekly for the very first time when I go to that page and then go right to real-time traffic analyzer it crashes httpd and then I just noticed for the first time that I got the BWDPI BWANA error. Deleted traffic analyser DB and DB journal files off JFFS. I'm on the newest BIOS non beta for my router all settings were put in manually after I updated. All traffic data gone obviously since it deleted the files. My 5 GHz radio was acting goofy until I shut the power off and restarted it that way. Probably not related. Maybe a glitch but I couldn't just reset it, I had to completely remove power from it by turning it off the power switch.

 

[FTC]

RT-AX88U

Had httpd crash with kernel tainted... when switching between Traffic Analyser history and real time traffic. I haven't tracked down exactly what happens but when I switch to monthly or weekly for the very first time when I go to that page and then go right to real-time traffic analyzer it crashes httpd and then I just noticed for the first time that I got the BWDPI BWANA error. Deleted traffic analyser DB and DB journal files off JFFS. I'm on the newest BIOS non beta for my router all settings were put in manually after I updated. All traffic data gone obviously since it deleted the files. My 5 GHz radio was acting goofy until I shut the power off and restarted it that way. Probably not related. Maybe a glitch but I couldn't just reset it, I had to completely remove power from it by turning it off the power switch.

In my case I have observed in the log file that the db erase thing happens always at timestamps that are at an exact hour plus 20 seconds. i.e the last 3 occurences happened yesterday at 11:00h:20 sec, and today at 18:00h 20 sec and 23:00h 20 sec. This makes me think that the problem happens in the process that refreshes the traffic count, which happens hourly.

 

[dave14305]

Every hour, the hour_monitor process runs these 2 commands:

TrafficAnalyzer -e
TrafficAnalyzer -d 14336

The first one saves the stats, the second one prunes the DB file.

You can activate a certain amount of debugging with this command:

touch /tmp/BWSQL_LOG /tmp/BWMON_LOG

Then, the bwdpi processes will write extra output to:

/tmp/BWSQL.log
/tmp/BWMON.log

If you experience the problem, maybe these logs will reveal additional clues as to what is happening before “Revert” and delete.

Stop the logging with:

rm /tmp/BWSQL_LOG /tmp/BWMON_LOG

The SQL log will likely grow faster than the MON file, so watch the free space in /tmp after a few hours.