Asus RT-AC86U - Out of memory / Settings don't save / Log "Argh. No free space left for GC."

[slamdunk]

Hi Everyone

I often dip in and out of SNB to research and solve my own problems but I am really struggling with this one.

I noticed certain settings that I was adding were just not saving (port forwarding etc...) and checked the logs which are repeatedly filling up with something like this.

Jan 12 11:29:26 kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
Jan 12 11:29:26 kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28
Jan 12 11:29:26 kernel: jffs2: Error garbage collecting node at 00b18ee4!


I tracked this down to possibly an issue with Traffic Analyser which I have turned on in the past but hasn't been active for ages.

Tried the usual update firmware, reboot etc...

The router also reboots at set intervals anyway.


Would like to avoid a full factory reset if possible as worry about rebuilding static ips etc..

Any suggestions for someone running stock ASUS firmware? I saw with Merlin you can format jffs which might fix the problem but not obviously possible via ASUSWRT

Note I have AIMESH nodes so gather still need to be on stock firmware for that....

 

[ColinTaylor]

I can't find any reference to an "RT-AC82U" on the internet.

If I were you I would backup my settings, do a hard factory reset and then reload the settings.

 

[slamdunk]

I can't find any reference to an "RT-AC82U" on the internet.

If I were you I would backup my settings, do a hard factory reset and then reload the settings.

Sorry that was a typo - changed to 86U!


Yes I agree that is an option but also read restoring settings can restore problems...?

 

[ColinTaylor]

... but also read restoring settings can restore problems...?

It depends on what the problem is. Your problem appears to be that you've run out of space in the jffs partition. So the least destructive option would be to backup and restore your current settings while wiping jffs in between. If the hard reset doesn't clear the partition then try the using the GUI option to factory reset with "initialize" instead.

 

[slamdunk]

Thanks - will give it a shot when I can find a gap in all the home learning/schooling!

 

[L&LD]

Release - [Test] Asuswrt-Merlin 384.19 - OpenVPN test builds

If you replace ovpn-up, then you become responsible for doing everything it was previously doing. https://github.com/RMerl/asuswrt-merlin.ng/blob/master/release/src/router/libovpn/openvpn_control.c#L237 And also updating resolv.conf/dnsmasq's config...

www.snbforums.com

This may be your issue above?

The links below are what I would suggest and to also consider RMerlin 386.1 Beta 4b (in the link Pre-beta test builds (occasionally available, hosted on Onedrive), from the main download page Download | Asuswrt-Merlin (asuswrt-merlin.net) to have access to AiMesh v2.0 and all the RMerlin goodness too.

New M&M 2020

How to Connect an AiMesh Node

And while RMerlin suggests that an AiMesh node is fine on stock firmware (as long as it's using firmware from the same code base), I would suggest flashing all nodes to the same RMerlin firmware too.

The following link may also be helpful to get your router back to a good/known state if all the information above doesn't help by itself.

Fully Reset Router and Network

 

[slamdunk]

I haven't put any Merlin builds on there which is why I am struggling to find a definitive reason (would understand if I had been tinkering and it went wrong)

Thanks for the detailed guide on how properly factory reset !

 

[slamdunk]

Just a quick update to say I managed to do a hard reset on the main router.

Interestingly after the hard reset I was still getting the out of memory issues in the log (immediately with out restoring any config) so I also then ran through the "initialise" option and this cleared that problem.

I was able to re-load my settings onto the device and the memory errors in the log have not return as yet!

 

[OzarkEdge]

Just a quick update to say I managed to do a hard reset on the main router.

Interestingly after the hard reset I was still getting the out of memory issues in the log (immediately with out restoring any config) so I also then ran through the "initialise" option and this cleared that problem.

I was able to re-load my settings onto the device and the memory errors in the log have not return as yet!

Initialize clears logged data, so it must have cleaned up that storage. I prefer to reset with Initialize when I can.

OE

 

[vaboro]

I'm also getting the garbage collecting errors:

Jan 20 11:12:15 RT-AC86U kernel: jffs2: read_cache_page() returned error: -5
Jan 20 11:12:15 RT-AC86U kernel: jffs2: Error garbage collecting node at 00bb12c0!
Jan 20 11:12:17 RT-AC86U kernel: jffs2: Data CRC 73d2e638 != calculated CRC 0c3820b2 for node at 00bb12c0

I googled the errors and found out that they might be related to Linux kernel. If this is correct I presume they should be patched by Asus and incorporated into the next Merlin firmware release.

 

[ColinTaylor]

I googled the errors and found out that they might be related to Linux kernel. If this is correct I presume they should be patched by Asus and incorporated into the next Merlin firmware release.

No it's not something that would be patched because it is not a bug. They are caused by a lack of free space. See posts #4 and #8.

 

[vaboro]

No is not something that would be patched because it is not a bug. They are caused by a lack of free space. See posts #4 and #8.

Please, have a look at this one.
I have lots of free space in jffs:

admin@RT-AC86U:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         77.2M     60.6M     16.6M  79% /
devtmpfs                214.9M         0    214.9M   0% /dev
tmpfs                   215.0M    200.0K    214.8M   0% /var
tmpfs                   215.0M    680.0K    214.4M   0% /tmp/mnt
mtd:bootfs                4.4M      3.3M      1.1M  75% /bootfs
tmpfs                   215.0M    680.0K    214.4M   0% /tmp/mnt
mtd:data                  8.0M    624.0K      7.4M   8% /data
tmpfs                   215.0M    680.0K    214.4M   0% /tmp
/dev/mtdblock9           47.0M     11.0M     36.0M  23% /jffs
/dev/sda1                14.4G      3.5G     10.1G  26% /tmp/mnt/Storage
/dev/mtdblock9           47.0M     11.0M     36.0M  23% /www/Main_LogStatus_Content.asp
tmpfs                   215.0M    680.0K    214.4M   0% /www/require/modules/menuTree.js

Yet, the problem persists.
I haven't yet found a better way but to reboot the router on a regular basis.

 

[ColinTaylor]

I have lots of free space in jffs:

admin@RT-AC86U:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         77.2M     60.6M     16.6M  79% /
devtmpfs                214.9M         0    214.9M   0% /dev
tmpfs                   215.0M    200.0K    214.8M   0% /var
tmpfs                   215.0M    680.0K    214.4M   0% /tmp/mnt
mtd:bootfs                4.4M      3.3M      1.1M  75% /bootfs
tmpfs                   215.0M    680.0K    214.4M   0% /tmp/mnt
mtd:data                  8.0M    624.0K      7.4M   8% /data
tmpfs                   215.0M    680.0K    214.4M   0% /tmp
/dev/mtdblock9           47.0M     11.0M     36.0M  23% /jffs
/dev/sda1                14.4G      3.5G     10.1G  26% /tmp/mnt/Storage
/dev/mtdblock9           47.0M     11.0M     36.0M  23% /www/Main_LogStatus_Content.asp
tmpfs                   215.0M    680.0K    214.4M   0% /www/require/modules/menuTree.js

Yet, the problem persists.
I haven't yet found a better way but to reboot the router on a regular basis.

Sorry, I've just re-read your post and your error is different than the one discussed in this thread.

This thread was about the "No free space left for GC" message. Your error is different: "Data CRC 73d2e638 != calculated CRC 0c3820b2 for node at 00bb12c0".

So it looks like a corruption in the jffs partition. What router and firmware do your have? If you have an RT-AC86U and installed 384.19 or later did you reformat the jffs partition as required?

 

[vaboro]

Sorry, I've just re-read your post and your error is different than the one discussed in this thread.

You're right. The errors might not be the same. Yet, I decided to post here because I thought they might be related. If you have a look here you may notice that the two errors can occur one after the other:

The "Data CRC" message is issued by jffs2_read_dnode() which returns
-EIO (= -5).
...
The garbage collection thread is then aborted:
jffs2: No space for garbage collection. Aborting GC thread
...

So it looks like a corruption in the jffs partition. What router and firmware do your have? If you have an RT-AC86U and installed 384.19 or later did you reformat the jffs partition as required?

Maybe you are right. Yet, I have indeed an RT-AC86U with 384.19 Stable and I reformatted the jffs at least a couple of times; however, the problem persists.

 

[ColinTaylor]

Maybe you are right. Yet, I have indeed an RT-AC86U with 384.19 Stable and I reformatted the jffs at least a couple of times; however, the problem persists.

Same error reported here with 384.19. I think the solution is to backup your settings, do a factory reset with initialize, and then restore your settings. If that doesn't work you might have to do a factory reset with initialize followed by a manual setup.

 

[vaboro]

Same error reported here with 384.19. I think the solution is to backup your settings, do a factory reset with initialize, and then restore your settings. If that doesn't work you might have to do a factory reset with initialize followed by a manual setup.

Thank you for your help. I will look into the discussion you advised.
One thing I noticed is that the GC error pops up here and there from time to time and every time it is assumed that jffs code is working correctly like you said that it's not a Linux kernel bug.

 

[vaboro]

The problem was in the TrendMicro’s license. As soon as I withdrew and then accepted the license again, the GC errors stopped. I suppose license agreement doesn’t survive either jffs or settings restore.