JFFS2 space issue, rebuilt, and questions

[shama_001]

Hello folks,

I faced something similar https://www.snbforums.com/threads/kernel-jffs2-error-garbage-collecting-node-at-00f7fcac.85675/ and would like some info/advice about the way to avoid it or what I did wrong. I checked this topic too during the debug https://www.snbforums.com/threads/solved-rt-ax88u-jffs-ubi-brain-mix.90342/.

#Setup
3x XT8 nodes with Gnuton 388.2-2 (same FW as the other topic, weird?) Edit: I'm on 388.8-2, with an attached USB key.
I know this last one is controversial an will come back on it later

#Issue:
I do not have a capture of my own log (rebuilt done already) but the JFFS partion was reported with GC (garbage collection) impossible. The size of "main_checker.sh_bk" was indeed problematic

#Remediation steps achieved
I noticed this issue just after a vacation week. In the 3 to 4 weeks before below occurred:
- Update to Gnuton 388.2-2 EDIT: 388.8-2, last one
- Install uidivstat (amtm etc. fully updated)
- VPN activation, holiday in mind
- Backup through GUI of the JFFS and settings
At this moment the log and behavior was fine, nothing weird. The issue started at the end of the vacation week.

As mentioned I used a brand new Samsung USB key so a failure of it was my first idea. I restarted all nodes with no luck then checked https://www.snbforums.com/threads/solved-rt-ax88u-jffs-ubi-brain-mix.90342/ and indeed the file "/jffs/.asdbk/main_checker.sh_bk" was the one taking all the size. As steps to remediate were not posted I hard reset all XT8 nodes (WPS method), removed the USB key, and ordered USB case and SSD to replace the USB key supposed wrong.

During the setup I noticed the JFFS2 errors were still there (??) even after 2 hard reset of the router
I tested the USB key with h2testw and it's fine (as file copy, format, etc).
A soft reset (factory reset in the GUI) removed the error and I'm now online without SSD (not yet there)

#Question
a) How can the error survive after the hard reset, removal of the USB key?
b) Why does the soft reset worked?
c) It seems I faced something really similar than the first topic posted, different router but same FW. Something to worry about @RMerlin ?
d) Once the SSD will be there I intend to make the exact same setup (spdmerlin, diversion, uidivstat, swap on SSD). Anything I should avoid?

Thank reading me and have a great day!

 

[Crimliar]

You are concentrating on the wrong issue here, as JFFS is "on router" and does not exist on the external (SSD) drive. The threads that you reference, already seem to give possible solutions!

 

[shama_001]

You are concentrating on the wrong issue here, as JFFS is "on router" and does not exist on the external (SSD) drive. The threads that you reference, already seem to give possible solutions!

Unknown area for me indeed, I was thinking JFFS2 was maybe on it but the lack of any failure on my USB key gave me this assumption too. Anyway replacing it with an SSD was in the plan, but yeah maybe nothing to do with the JFFS corruption out of the blue. Thx

The question remains...how a router super stable for years sees its internal (seems the case) JFFS corrupted during the least busy week of the year (my holiday). The VPN access is rare, and occured during this week to but It's hard to believe such create a problem, common usage

 

[Ripshod]

@ColinTaylor you saved an RT-AX88U, do you think you can help rescue this model?

 

[ColinTaylor]

@ColinTaylor you saved an RT-AX88U, do you think you can help rescue this model?

Maybe. I'm not familiar with the layout of the XT8 so I'd need to see the complete unedited System Log from power up as well as some other diagnostic information.

 

[shama_001]

@Ripshod @ColinTaylor
Hi and thx. At this time the error is no more present, sorry if I made this unclear in my first post (or I missed your point?)
The WPS hard reset of all noded didn't make a difference but the soft reset did so at this time I'm with a 24h router freshly reconfigured, waiting for the SSD delivery. I'm still with Gnuton 388.8-2, no change on this side.

The log are clean, but wasn't after the hard reset on all node with an error present, the SSH key not changed (normal?) and some clue on remaining configuration that should not be there. At first glance it looked like reset, but the error log with the exact same JFFS full message and for example a port forwarding present (after hard reset) was odd. Then The factory reset in the GUI make it. The USB key failing was not the root cause it seems (see Crimliar post).

Do you think something needs to be done in the actual situation? If so we can investigate of course, thanks considering it!

Edit:
The JFFS is realy slowly growing, nothing to consider I think yet.

 

[Ripshod]

@shama_001 that's looking real good now. Nice one!