Issue with GT-AXE11000 firmware not saving. /data partition mounting in read-only.

[unsynaps]

Been having some, weirdness with my router the last week or so and its come to a head.

@GT-AXE11000-2730:/tmp/home/root# nvram commit
wlcsm_nvram_commit:641 could not open nvram file
copy_file:217  Could not open file:/data/.kernel_nvram.setting

UIhhhhhhhhhhhhh,,,

@GT-AXE11000-2730:/tmp/home/root# touch /data/.kernel_nvram.setting
touch: /data/.kernel_nvram.setting: Read-only file system

Read-only? Huh?

 

[JGrana]

Try this from the command line:

ls -al /data/kernel

The .kernel_nvram.setting file permissions should be:

rwxr-xr-x

If your missing the first “w” - I would reboot and try again.

 

[unsynaps]

ls -al /data/kernel

crkinard@mpu:/tmp/home/root# ls -al /data/kernel
ls: /data/kernel: No such file or directory

 

[JGrana]

crkinard@mpu:/tmp/home/root# ls -al /data/kernel
ls: /data/kernel: No such file or directory

Oops, sorry it’s an invisible file and its name is longer


Try this:

ls -al /data/.kernel*

 

[unsynaps]

Yeah thats there but i cant modify it in any way. Tried 'touching' it and it said it was a read only flie system.

It seems that there might be something wrong with the firmware. Flashing back to latest factory one all problems went away even after many reboots. Flash latest merlin and /data mounts as read only due to some error.

 

[unsynaps]

Not 100% sure yet but I think there is something screwed up in the firmware for this router past version 388.2_0

Can mess around in 388.2_0 till im blue in the face. Change settings. Reboot out the wazoo. Set it back up after a 'nvram erase'. All with no issue. Router seems to also reboot a hell of a lot faster.

Second I flash 388.7_0 the router boots with /data in ro mode and cant do a thing that lasts over a reboot. Cant even setup the router. Saves NON of the settings and goes back into first setup mode after reboot.

 

[northumberland]

Any other drives plugged in?

 

[unsynaps]

The issue remanifested in firmware 3004.388.4_0.

No issues with 388.2_0 or 388.2_2.

No USB devices. Other network ports filled yes. But never had this issue till recently. Or at least never noticed it till recently. I dont change settings often which is the only way through the web interface you might notice there is a problem. When settings don't survive reboots. The only things i have tinkered with over the last few months are scripts which are not related to this issue.

So yeah, in short the issue seems to start in 3004.388.4_0.
Can not even set up the router in 3004.388.4_0 as /data ALWAYS boots in ro mode.

388.2_4 (17-Aug-2023)
- NOTE: This release is only for the GT-AXE11000.

What is this about? Testing 388.2_4 now. Skipped it and went right to 3004.388.4_0 cause i got tired of rebooting.

 

[unsynaps]

OK, the story so far.

It seems that on the GT-AXE11000 starting in firmware 3004.388.4_0 the /data mount (where nvram settings are stored) mounts in read-only mode for some unknown reason. No matter how many times you reboot or power cycle this happens. Can not even factory default because, well duh.

There seems to be no issue with the current official firmware.

Trued all 388.2 versions and they seem OK.

As soon as 3004.388.4_0 or higher is flashed after the reboot /data is back in read-only more. Meaning you can not even do first time setup. Well you can but it does not survive reboot.

So at this point I have no idea how to proceed. Other than go back to factory or run 388.2_4 which is latest version that works properly.

EDIT:
Abbbbd just had /data mount as ro on 388.2_4.

 

[hoang51]

I had a similar issue with my RT-AX88U where any time there was a power outage or a reboot, my router would revert back to previous stored settings (like certain VPN services was enabled and WiFi channels were preselected), which was weird. I would have to keep changing the wireless channels to auto and redo my VPN services. It got annoying because there were short burst power outage due to weather and power lines not playing nice with trees. I started to poke around and got simlar errors (first line is different #, but same concept error):

wlcsm_nvram_commit:644 could not open nvram file
copy_file:217 Could not open file:/data/.kernel_nvram.setting

In the end, I just ended my Google research and poking around time to just do a factory default/restore with the check box checked for "Initialize all the settings, and clear all the data log for AiProtection, Traffic Analyzer, and Web History." I then set up the router back to the way it was, tested by doing a manual reboot and all my customized settings were retained. Yay. Problem fixed by just starting clean. I guess doing many dirty Merlin firmware upgrades (currently on 3004.388.7; last time factory reset was probably 386 code branch) probably broke something. Another thing I found strange that got fixed was that Trend Micro: Signature version has the correct Updated date instead of reverting back to a previous date in year 2022 even though the update was known to be in year 2024.

 

[unsynaps]

It seems like the data or nvram portions are getting corrupted for some reason and the system cannot fix it. Thus it mounts in ro.

This happens all the time on reboot now it seems. But I sort of found a way to make it fix itself on reboot.

mount -t ubifs -o remount,rw /data
cd /data
cd
mount -t ubifs -o rw /dev/ubi2_0 /proc/nvram
cd /proc/nvram
cd
nvram commit
reboot

No idea why but when I run all this the router will come back from reboot with data mounted properly and everything working as it should.

So now the only thing I can figure is my nand chips are going bad or something. because after i boot the router and it boots in ro i run the above commands and it reboots in rw then i reboot again it reboots back to ro.

Is there any way to test all the nand in the router?

Also a bit curious how you even did a factory reset if you can not commit changes. When data goes ro factory reset does nothing because it cant write the factory defaults to data.

 

[unsynaps]

Well it just seems I am out a $500 router.

I can not find any reason why this is happening. The router is not stuck in RO mode for /data. I can not change any settings or even factory default it.
So yeah. Its now stuck the way its set up.

Yay.

 

[jerry6]

how long have you had this router ? Any chance you can get a replacement from Asus or covered under extrawarranty if bought with credit card ?odd as I have had no trouble with my AXE1100

 

[laracroftonline]

have you tried putting the router in rescue mode? without any usb plugged into the router.
flash the newest merlin that way and restore back to factory settings in the same rescue mode after flashing.

 

[ExtremeFiretop]

I had this exact issue in January on my first AXE11000 I bought from ebay.
I tried many things, including flashing directly using commands:

And was unable too as seen above.
There are other threads around here of the same issue such as: https://www.snbforums.com/threads/data-kernel_nvram-setting-not-editable-on-gt-axe11000.85458/

I saw the error whenever I tried to make a NVRAM change as seen below:

It randomly showed up and left, sometimes with no action by me.

No flash attempts it would show up, continue to ignore it, and it would disappear again. Most of the time, I could fix it temporarily with multiple factory resets or multiple flashes.
To me, it looked like issues with the NAND flash on the router.

Looking at the logs I constantly saw errors related to UBIFS like you:
Feb 7 17:16:14 kernel: UBIFS warning (ubi1:0 pid 1): ubifs_ro_mode: switched to read-only mode, error -30

That lead me to some kinda NAND driver bug or bad hardware, kinda reconfirmed my theory on bad NAND flash.

UBIFS do_writepage error

Dear sir, I'm using an i.mx6ull (SDK 2.0.0) with a NAND flash. But, The following error occurs only on certain boards, and the file system is changed to read only. It will be recovered upon reboot, but the same phenomenon will be repeated soon. What's the cause? UBIFS error (ubi0:0 pid 6)...

community.nxp.com

Here is a copy of my log dump I still had:

Log Dump Rog

Log Dump Rog. GitHub Gist: instantly share code, notes, and snippets.

gist.github.com

Me and Martinski discussed this at length for some time, I even tried to adapt the steps found here:

jffs partition is read-only and cannot format-restore

Interesting - the mtd drivers says ok, the ubi volume manager says ok. Hopefully it is just a filesystem issue - might be able to recover at that level. Just wondering - have you tried different factory resets - like the hold WPS button or press reset for a long time? I am surprised (like you!)...

www.snbforums.com

To wipe my data partition.

cd /
umount /data
ubinfo --devn=1 --vol_id=0
ubirmvol /dev/ubi1 --vol_id=0
ubimkvol /dev/ubi1 --vol_id=0 -s 6.7MiB -N data
mount -t ubifs /dev/ubi1_0 /data

Actual commands used: BE WARNED!

Admin@GT-AXE11000-DBF0:/tmp/home/root# cd /
Admin@GT-AXE11000-DBF0:/# umount /data
Admin@GT-AXE11000-DBF0:/# ubinfo --devn=1 --vol_id=0
Volume ID: 0 (on ubi1)
Type: dynamic
Alignment: 1
Size: 56 LEBs (7110656 bytes, 6.7 MiB)
State: OK
Name: data
Character device major/minor: 251:1
Admin@GT-AXE11000-DBF0:/# ubirmvol /dev/ubi1 --vol_id=0
Admin@GT-AXE11000-DBF0:/# ubimkvol /dev/ubi1 --vol_id=0 -s 6600KiB -N data
Volume ID 0, size 54 LEBs (6856704 bytes, 6.5 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "data", alignment 1
Admin@GT-AXE11000-DBF0:/# mount -t ubifs /dev/ubi1_0 /data
Admin@GT-AXE11000-DBF0:/# nvram erase
Admin@GT-AXE11000-DBF0:/# nvram commit

And then use the tfpd tool to restore instead of the ASUS rescue tool. https://pjo2.github.io/tftpd64/
That worked for some time until the next firmware release I believe, and flashing that, it eventually came back.

The only "solve" in the end, was to get another AXE11000, it had no issues at all and still doesn't to this day.

Which is why I ran a AXE11000 as primary and another (the buggy one) as my test node.
If it broke I didn't care. Now I have a BE98 Pro as primary and the working AXE11000 as my test device/node.

 

[ExtremeFiretop]

You can always check the health using @JGrana 's script: https://www.snbforums.com/threads/a...available-for-select-models.81644/post-801367

GitHub - JGrana01/mtd_check: Program to display information about the NAND flash on Asuswrt routers

Program to display information about the NAND flash on Asuswrt routers - JGrana01/mtd_check

github.com

My results below showing bad blocks on the NAND:
MTD Devices Below:

Admin@GT-AXE11000-DBF0:/tmp/home/root# mtd_check /dev/mtd2 -v
Flash type of /dev/mtd2 is 4 (NANDFLASH)
Flash flags are 0x400 - WRITEABLE
Block size 131072 (128K) Page size 2048 (2K) OOB size 108
Device: 8388608 (8M) bytes, 64 blocks
MTD Stats: Bad Blocks: 6 ECC Corrected: 0 ECC Uncorrected: 0

No Region Information on device
B Bad block; . Empty; - Partially filled; * Full; R Reserved (BBT); S has a JFFS2 summary node

-----------B*-*---------*-*-----**---B--*-B-*--------*-----BBB--
Summary /dev/mtd2:
Empty Blocks: 0, Full Blocks: 9, Partially Full: 49, Bad Blocks: 6, Reserved Blocks (BBT) 0

Admin@GT-AXE11000-DBF0:/tmp/home/root# mtd_check /dev/mtd6 -v
Flash type of /dev/mtd6 is 4 (NANDFLASH)
Flash flags are 0x400 - WRITEABLE
Block size 131072 (128K) Page size 2048 (2K) OOB size 108
Device: 268435456 (256M) bytes, 2048 (2K) blocks
MTD Stats: Bad Blocks: 6 ECC Corrected: 0 ECC Uncorrected: 0

No Region Information on device
B Bad block; . Empty; - Partially filled; * Full; R Reserved (BBT); S has a JFFS2 summary node

********-----------****************************-----------**********************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************-***********------------------------------------
--------------------------------------------------------------------------------
-----------****************************-----------******************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
************************-***********--------------------------------------------
--------------------------------------------------------------------------------
----------*------------*--------------------------------**.**...*.*...*.........
..............********....................*...*.*****.****.****......*.*........
.....................*..**.**...................................................
..................................................****..***..*.***..***.........
*..............**..*............................*.......-----------B*-*---------
*-*-----**---B--*-B-*--------*-----BBB--....RRRR
Summary /dev/mtd6:
Empty Blocks: 268, Full Blocks: 1381, Partially Full: 389, Bad Blocks: 6, Reserved Blocks (BBT) 4

Admin@GT-AXE11000-DBF0:/tmp/home/root# mtd_check /dev/mtd7 -v
Flash type of /dev/mtd7 is 4 (NANDFLASH)
Flash flags are 0x400 - WRITEABLE
Block size 131072 (128K) Page size 2048 (2K) OOB size 108
Device: 268435456 (256M) bytes, 2048 (2K) blocks
MTD Stats: Bad Blocks: 6 ECC Corrected: 0 ECC Uncorrected: 0

No Region Information on device
B Bad block; . Empty; - Partially filled; * Full; R Reserved (BBT); S has a JFFS2 summary node

********-----------****************************-----------**********************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************-***********------------------------------------
--------------------------------------------------------------------------------
-----------****************************-----------******************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************
************************-***********--------------------------------------------
--------------------------------------------------------------------------------
----------*------------*--------------------------------**.**...*.*...*.........
..............********....................*...*.*****.****.****......*.*........
.....................*..**.**...................................................
..................................................****..***..*.***..***.........
*..............**..*............................*.......-----------B*-*---------
*-*-----**---B--*-B-*--------*-----BBB--....RRRR
Summary /dev/mtd7:
Empty Blocks: 268, Full Blocks: 1381, Partially Full: 389, Bad Blocks: 6, Reserved Blocks (BBT) 4

 

[NotoriousNate]

Well it just seems I am out a $500 router.

I can not find any reason why this is happening. The router is not stuck in RO mode for /data. I can not change any settings or even factory default it.
So yeah. Its now stuck the way its set up.

Yay.

coming to that same conclusion

 

[unsynaps]

have you tried putting the router in rescue mode? without any usb plugged into the router.
flash the newest merlin that way and restore back to factory settings in the same rescue mode after flashing.

Yes. No matter what I do the utility can not see the router even though I am almost certain it is properly in recovery mode.

 

[unsynaps]

how long have you had this router ? Any chance you can get a replacement from Asus or covered under extrawarranty if bought with credit card ?odd as I have had no trouble with my AXE1100

Do not remember when I got it but I have had it since before Merlin supported it.

 

[jerry6]

Yes. No matter what I do the utility can not see the router even though I am almost certain it is properly in recovery mode.

now I am starting to worry about my AXE1100 , it works well with 388.8 so think that is the last update for a while . just looked at the cost of replacing it $746.00 Canadian, $ no thanks will not spend that on a 2 year old router . Hope it lasts as it has worked well the last 2 years .
Will be following to see if the problem is resolved or it is just faulty hardware .