jffs partition is read-only and cannot format-restore

[JGrana]

Interesting - the mtd drivers says ok, the ubi volume manager says ok.
Hopefully it is just a filesystem issue - might be able to recover at that level.
Just wondering - have you tried different factory resets - like the hold WPS button or press reset for a long time?

I am surprised (like you!) that a factory device reset doesn’t reinitialize jffs. If true, I would think that’s a serious bug/issue with Asus ubifs implementation.

 

[puremind]

Interesting - the mtd drivers says ok, the ubi volume manager says ok.
Hopefully it is just a filesystem issue - might be able to recover at that level.
Just wondering - have you tried different factory resets - like the hold WPS button or press reset for a long time?

I am surprised (like you!) that a factory device reset doesn’t reinitialize jffs. If true, I would think that’s a serious bug/issue with Asus ubifs implementation.

Yep, perhaps for whatever reason the ubifs are not formatted upon restart.
Tried WPS hold... had to manually power off and on after that -> no joy
Seems is impossible to wipe the partition using regular methods.

There must be a command or something to just remove/reformat that /jffs partition to wipe the logical contents and move it on from there.

 

[JGrana]

No doubt. But from what I understand, it’s a two step process. First, redo the ubi volume, then format the ubifs partition.
The bigger issue is all the command line arguments needed to correctly setup ubi/ubifs.
I would think that any incorrect values could brick the router.

Unfortunatly it’s not as easy as formatting a regular mass storage device…

I wonder if there are some scripts in the source code that does what you are looking for?

As an owner of one of these new class routers, I also would like to see whats going on, especially with the factory device reset.

 

[puremind]

No doubt. But from what I understand, it’s a two step process. First, redo the ubi volume, then format the ubifs partition.
The bigger issue is all the command line arguments needed to correctly setup ubi/ubifs.
I would think that any incorrect values could brick the router.

Unfortunatly it’s not as easy as formatting a regular mass storage device…

I wonder if there are some scripts in the source code that does what you are looking for?

Yep, I am reading around. It would be safer if the command only affected the specific partition for jffs and not the others.

Won't run anything yet, perhaps Merlin knows where that code is or something that might let me just delete the file that is

 

[puremind]

Wondering these two commands should do... but there's not much documentation

admin@GT-AX11000_Pro:/bin# ubirmvol --help
ubirmvol version 2.0.2 - a tool to remove UBI volumes.

Usage: ubirmvol <UBI device node file name> [-n <volume id>] [--vol_id=<volume id>]

         [-N <volume name>] [--name=<volume name>] [-h] [--help]

Example: ubirmvol /dev/ubi0 -n 1 - remove UBI volume 1 from UBI device corresponding
         to /dev/ubi0
         ubirmvol /dev/ubi0 -N my_vol - remove UBI named "my_vol" from UBI device
         corresponding to /dev/ubi0

-n, --vol_id=<volume id>   volume ID to remove
-N, --name=<volume name>   volume name to remove
-h, -?, --help             print help message
-V, --version              print program version



admin@GT-AX11000_Pro:/bin# ubimkvol --help
ubimkvol version 2.0.2 - a tool to create UBI volumes.

Usage: ubimkvol <UBI device node file name> [-h] [-a <alignment>] [-n <volume ID>] [-N <name>]
            [-s <bytes>] [-S <LEBs>] [-t <static|dynamic>] [-V] [-m] [-k]
            [--alignment=<alignment>][--vol_id=<volume ID>] [--name=<name>]
            [--size=<bytes>] [--lebs=<LEBs>] [--type=<static|dynamic>] [--help]
            [--version] [--maxavsize] [--skipcheck]

Example: ubimkvol /dev/ubi0 -s 20MiB -N config_data - create a 20 Megabytes volume
         named "config_data" on UBI device /dev/ubi0.

-a, --alignment=<alignment>   volume alignment (default is 1)
-n, --vol_id=<volume ID>      UBI volume ID, if not specified, the volume ID
                              will be assigned automatically
-N, --name=<name>             volume name
-s, --size=<bytes>            volume size volume size in bytes, kilobytes (KiB)
                              or megabytes (MiB)
-S, --lebs=<LEBs count>       alternative way to give volume size in logical
                              eraseblocks
-m, --maxavsize               set volume size to maximum available size
-t, --type=<static|dynamic>   volume type (dynamic, static), default is dynamic
-k, --skipcheck               skip the CRC check done at volume open time
-h, -?, --help                print help message
-V, --version                 print program version

 

[puremind]

Guessing that if this command

ubinfo --devn=0 --vol_id=13

renders this:

Volume ID:   13 (on ubi0)
Type:        dynamic
Alignment:   1
Size:        413 LEBs (52441088 bytes, 50.0 MiB)
State:       OK
Name:        jffs2

Then the 2 below commands should do the work, no?
type and alignment are the default.

Delete volume

ubirmvol /dev/ubi0 --vol_id=13

Create volume

ubimkvol /dev/ubi0 --vol_id=13 -s 50MiB -N jffs2

Or

ubimkvol /dev/ubi0 --vol_id=13 --maxavsize -N jffs2

Will wait for a Merlin recommendation unless anyone has tested this in the field and can confirm it will work.

 

[RMerlin]

I can't help you with this. I have no experience at all with UBIFS.

 

[puremind]

I can't help you with this. I have no experience at all with UBIFS.

Hi RMerlin:

Thank you anyways, @JGrana indirectly helped with the suggested commands.

I can confirm the developed steps + commands below do work to completely wipe the ubi:jffs2 Filesystem that mounts to /jffs and then rebuild correctly to achieve a new brand empty clean and writeable /jffs partition.

Disable jffs scripts usage with the GUI
Administration -> System -> Enable JFFS custom Configs and scripts OFF
http://www.asusrouter.com/Advanced_System_Content.asp

ssh to the router and (use at user's own risk)

cd /

#Make sure jffs does umount
umount /jffs

#Confirm that jffs does umount without error

#Confirm volume 13 is the jffs one
ubinfo --devn=0 --vol_id=13

#This should return:
: '
Volume ID:   13 (on ubi0)
Type:        dynamic
Alignment:   1
Size:        413 LEBs (52441088 bytes, 50.0 MiB)
State:       OK
Name:        jffs2
'
#Otherwise stop

#Delete volume 13 (we can't use it anyways)
ubirmvol /dev/ubi0 --vol_id=13

#Recreate volume 13 (with the same MiB as listed above!)
ubimkvol /dev/ubi0 --vol_id=13 -s 50MiB -N jffs2

#Mount again the /jffs folder
mount -t ubifs /dev/ubi0_13 /jffs

If it mounts, and /jffs is empty, I would suggest

reboot

df -h
ubi:jffs2                44.5M      7.6M     34.5M  18% /jffs

Re-enable jffs scripts usage with the GUI
Administration -> System -> Enable JFFS custom Configs and scripts ON
http://www.asusrouter.com/Advanced_System_Content.asp


Note:
When I recreated the volume with:
ubimkvol /dev/ubi0 --vol_id=13 --maxavsize -N jffs2
I got a bigger than normal jffs, just on the side of caution, deleted it and recreated with the normal 50MiB.

Perhaps this could lead you to forge a safe script or GUI addition to force format the "jffs" partition when the router has ubifs , seems pretty safe to do it when everything matches. Anyway, at owners risk, as usual.
Other people wouldn't be able to figure this out. Others have here suggested me to "return the router" ...

Using the very useful Restore/save/Upload menu:

http://www.asusrouter.com/Advanced_SettingBackup_Content.asp

Just restoring my backup_jffs.tar has replaced everything there and writeable as usual, thank god!

 

[puremind]

No doubt. But from what I understand, it’s a two step process. First, redo the ubi volume, then format the ubifs partition.
The bigger issue is all the command line arguments needed to correctly setup ubi/ubifs.
I would think that any incorrect values could brick the router.

Unfortunatly it’s not as easy as formatting a regular mass storage device…

I wonder if there are some scripts in the source code that does what you are looking for?

As an owner of one of these new class routers, I also would like to see whats going on, especially with the factory device reset.

Thank you for the suggestions and the commands, they've helped figure out the problem.

I'll now gonna move some stuff I had using the jffs to do it on the usb drive, given this problem can happen again...

 

[sfx2000]

May 5 06:05:22 kernel: UBIFS error (ubi0:13 pid 258): ubifs_assert_failed: UBIFS assert failed: dirty == LPROPS_NC || dirty >= 0, in fs/ubifs/lprops.c:557
May 5 06:05:22 kernel: UBIFS warning (ubi0:13 pid 258): ubifs_ro_mode.part.0: switched to read-only mode, error -22

You need to clean up that file system or it's going to remain as R/O

UBIFS is doing the right thing here...

 

[Viktor Jaep]

Thank you for the suggestions and the commands, they've helped figure out the problem.

I'll now gonna move some stuff I had using the jffs to do it on the usb drive, given this problem can happen again...

Excellent sleuthing, @puremind!

 

[drinkingbird]

Thank you for the suggestions and the commands, they've helped figure out the problem.

I'll now gonna move some stuff I had using the jffs to do it on the usb drive, given this problem can happen again...

Now that you've gotten it hopefully writable I'd go for the hard factory reset and hopefully the firmware will format and rebuild the filesystem from scratch.

 

[JGrana]

Great jobs @puremind , nicely done and I have squirreled away your script in case of emergency.

With that said, Im still concerned that the ubifs manager did not take care of the obvious error.

To quote kernel.org

Similarly to JFFS2, UBIFS is tolerant of unclean reboots and power-cuts.It does not need stuff like fsck.ext2. UBIFS automatically replays itsjournal and recovers from crashes, ensuring that the on-flash datastructures are consistent.

And the general opinion is that ubifs should never be corrupted makes me concerned. It might have been the ubi volume manager or even the mtd layer - but both said "ok".

I hope this is a one off case...

 

[drinkingbird]

Great jobs @puremind , nicely done and I have squirreled away your script in case of emergency.

With that said, Im still concerned that the ubifs manager did not take care of the obvious error.

To quote kernel.org



And the general opinion is that ubifs should never be corrupted makes me concerned. It might have been the ubi volume manager or even the mtd layer - but both said "ok".

I hope this is a one off case...

Of course the hope is that there isn't some portion of the flash that is hosed hence resulting in this issue, and may result in it again. I'm not sure what kind of flash management these have in them, my guess it is probably like a thumb drive, hopefully eventually marking a cell as bad, but not as reliable as an SSD with advanced controller.

Just something to keep an eye on for the OP I guess. I think I'd at least want to do a reset and make sure the firmware erases and rebuilds the JFFS this time. I'm not sure how far it is supposed to go, whether it repartitions it or just formats the existing partitions. I sort of suspect the latter.

 

[puremind]

What the error message suggests is a corrupted file descriptor pointing to an inode with some error state the normal “recovery” wasn’t able to fix, perhaps caused by a restart while data was updated in there, so it goes to read only mode waiting for something else to fix it.

There are many log files there the system is writing on a flash. No option to write to the usb or disable. I think the affected file was one of the curl* logs the router keeps writing there.

The point here is fact there is no other binary to fix anything on the fs, if you check in /bin all the ubi* binaries are there . The only option I could think of was to delete the whole volume used for jffs, recreate and remount. Removing the filesystem was probably going to brick the router. Mitigating options are to try to avoid to write in there logs that, in any case, should go to the usb imo.


The fact the ubi developers “assume” there will be no case of corruption seems overly positive and proven wrong. If you read the section with the error message, it is clear there is a logical issue with the inode descriptor, then the log says “recovery needed” but there is no “recovery” or binary to do so. The only way to solve was to delete and recreate the volume.

Sounds like a bug that will express itself again , very randomly , but is there.
At a minimum I would expect another binary with the option to list and delete the affected file(s) for this specific error.

Definitely the forced read-only state is also something the UI could read and suggest course of action.
There is a file there named
./sys/kernel/debug/ubifs/ubi0_13/ro_error which value is 1 when the issue was happening. (If jffs is your ubi volume 13) now it reads “0” because there is no issue

 

[puremind]

Of course the hope is that there isn't some portion of the flash that is hosed hence resulting in this issue, and may result in it again. I'm not sure what kind of flash management these have in them, my guess it is probably like a thumb drive, hopefully eventually marking a cell as bad, but not as reliable as an SSD with advanced controller.

Just something to keep an eye on for the OP I guess. I think I'd at least want to do a reset and make sure the firmware erases and rebuilds the JFFS this time. I'm not sure how far it is supposed to go, whether it repartitions it or just formats the existing partitions. I sort of suspect the latter.

What I would assume is the jffs partition would be erased/formatted by any of the resets, but wasn’t the case. There was no way to delete that partition or it’s contents but the one I discovered.

If you have there any secrets… will stay there forever lol.

Hoping an option can be added to the UI for all the people that won’t get this figured out… or even if you wanted to sell your router. Doesn’t make sense the soft or hard reset to erase the volume ?

 

[puremind]

Now that you've gotten it hopefully writable I'd go for the hard factory reset and hopefully the firmware will format and rebuild the filesystem from scratch.

There was no need. Deleting the volume and creating it was enough. After mounting I was able to write in jffs and restored my .tar backup with the UI. As good as new.

 

[puremind]

You need to clean up that file system or it's going to remain as R/O

UBIFS is doing the right thing here...

We all knew that. How do you suggest that to happeb?
Have you figured out a way or method to “clean” the file system that is not what I have discovered and shared here ?

 

[drinkingbird]

There was no need. Deleting the volume and creating it was enough. After mounting I was able to write in jffs and restored my .tar backup with the UI. As good as new.

Understood but the thinking is that now that it is writable, the hard factory reset should be able to format it and do whatever else it does to hopefully make sure it is set up correctly.

 

[puremind]

Understood but the thinking is that now that it is writable, the hard factory reset should be able to format it and do whatever else it does to hopefully make sure it is set up correctly.

There is nothing else to do after the volume is recreated.