Tutorial Fix full /jffs without factory reset

[toaruScar]

I recently ran into a full JFFS2 system at /jffs due to bloated /jffs/.sys/TrafficAnalyzer folder.
df reports that /jffs is 100% used on my router, and due to the design of JFFS, files can't be removed or edited if the disk if full. Therefore following commands are useless in this case:

TrafficAnalyzer -d 30720
rm -rf /jffs/.sys/TrafficAnalyzer/
echo "" > /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db

.

The right way to fix it is to reconstruct a JFFS2 system.

Requriement
SSH access to the router is enabled before /jffs is full.
Merlin is not required.

Dump the /jffs
Get an external storage device, be it a USB stick or HDD, connect it to the router. Then this storage device should be automatically mounted in /mnt. For the sake of discussion, let's say it's mounted as /mnt/External.
Then we dump the filesystem to a dd image.
To do so, we need to find the corresponding block device on which /jffs resides. To do so, run cat /proc/mounts, and observe the output:

$ cat /proc/mounts
...
/dev/mtdblock9 /jffs jffs2 rw,noatime 0 0

. Then we can dump the file system with dd if=/dev/mtdblock9 of=/mnt/External/original.jffs2.

Fetch the dd image
Now on your own PC running some version of linux, fetch the original.jffs2 with scp -i [ssh_key] -P [ssh port] [username]@[router's ip]:/mnt/External/original.jffs2 . command.
For example, if the private key is in ~/.ssh/id, and ssh server on the router is running on port 78, and the username (same as GUI login name) is admin, router's IP is 192.168.50.1, then the command would be scp -i ~/.ssh/id -P 78 admin@192.168.50.1:/mnt/External/original.jffs2 ..
Alternatively, just use sneakernet.

Mount the dd image
Now that the image file is on your PC, you can mount it. To play with JFFS, you will need to install mtd-utils, on Debian, it's done by running apt install mtd-utils.
The tutorial can be found here https://wiki.emacinc.com/wiki/Mounting_JFFS2_Images_on_a_Linux_PC (the "Mounting JFFS2 Images using RAM" section).

# cleanup if necessary
umount /dev/mtdblock0
modprobe -r mtdram
modprobe -r mtdblock

modprobe mtdram total_size=50000 erase_size=128
modprobe mtdblock
dd if=original.jffs2 of=/dev/mtdblock0
mkdir jffs_mount
mount -t jffs2 /dev/mtdblock0 jffs_mount

Here are explanations of some magic numbers:
50000: This is the number that's slightly bigger than than size of original.jffs2, in KiB. For example, the size of original.jffs2 in my case is 49283072B, or 48128KiB, so I used 50000.
128: This can be obtained from running cat /proc/mtd on router, read the tutorial for more information.

Remove unnecessary files
Now, the /jffs on your router should be available in the jffs_mount folder, and you can remove the bloated TrafficAnalyzer by running rm -r jffs_mount/.sys/TrafficAnalyzer.

Create new JFFS
Then we can recreate a slim JFFS2 volume from jffs_mount.

mkfs.jffs2 -r jffs_mount --eraseblock=0x20000 --pad=0x2f00000 -o trimmed.jffs2

The value for eraseblock and pad sould be copied from the output running cat /proc/mtd on router.
For example, on my router, the output is

cat /proc/mtd
dev:    size   erasesize  name
mtd0: 05f00000 00020000 "rootfs"
mtd1: 05f00000 00020000 "rootfs_update"
mtd2: 00800000 00020000 "data"
mtd3: 00100000 00020000 "nvram"
mtd4: 05f00000 00020000 "image"
mtd5: 05f00000 00020000 "image_update"
mtd6: 10000000 00020000 "dummy1"
mtd7: 10000000 00020000 "dummy2"
mtd8: 00100000 00020000 "misc3"
mtd9: 02f00000 00020000 "misc2"
mtd10: 00800000 00020000 "misc1"
mtd11: 05638000 0001f000 "rootfs_ubifs"
mtd12: 0001f000 0001f000 "METADATA"
mtd13: 0001f000 0001f000 "METADATACOPY"
mtd14: 003421d0 0001f000 "filestruct_full.bin"
mtd15: 006c8000 0001f000 "data"

We know that /jffs is on /dev/mtdblock9, then according to mtd9: 02f00000 00020000 "misc2", the eraseblock is set to 0x20000, and the pad is set to 0x2f00000.

Restore the new JFFS to the router
After this command, there should be a trimmed.jffs2 in the current directory, and its size should be exactly the same as original.jffs2.

Then just use scp to copy trimmed.jffs2 back to the storage device, and use dd again to dump the image to the block device
dd if=/mnt/External/trimmed.jffs2 of=/dev/mtdblock9.

Then do a reboot, and /jffs should be writable again.

 

[ColinTaylor]

Good stuff @toaruScar. Thanks for posting.

I would still suggest people also consider @Yousif Kelaita's approach here because it is also didn't result in data loss and doesn't require a Linux PC.

 

[toaruScar]

Good stuff @toaruScar. Thanks for posting.

I would still suggest people also consider @Yousif Kelaita's approach here because it is also didn't result in data loss and doesn't require a Linux PC.

I do not have physical access to the router. Once the router is reset, I will lose access to it because it cannot connect to the Internet via PPPoE anymore. So I must use this method that fixes the problem without resetting.

 

[Matthew Patrick]

Oh my gosh. I literally just had to deal with traffic analyzer over 30mb and filling up my JFFS on .19 on AC86U. I was getting the full notification in my webui. Removed thru ssh and done. What the heck

 

[toaruScar]

Oh my gosh. I literally just had to deal with traffic analyzer over 30mb and filling up my JFFS on .19 on AC86U. I was getting the full notification in my webui. Removed thru ssh and done. What the heck

In your case your JFFS is almost full, so you can still remove files.
My JFFS is full to to the brim, so I had to remake the filesystem.

 

[toaruScar]

Oh my gosh. I literally just had to deal with traffic analyzer over 30mb and filling up my JFFS on .19 on AC86U. I was getting the full notification in my webui. Removed thru ssh and done. What the heck

Also in my case /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db alone took up 30MB, and the journal for this database took up addtional 15MB as the last nail on the coffin.

I think some past ASUS firmware update might have been shipped with a regression that prevented traffic ayalyzer's database from being purged periodically. Then this regression slipped into in Merlin. As time passed by, this regression will manifest itself as a full /jffs partitions on more and more router.

 

[Matthew Patrick]

Also in my case /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db alone took up 30MB, and the journal for this database took up addtional 15MB as the last nail on the coffin.

I think some past ASUS firmware update might have been shipped with a regression that prevented traffic ayalyzer's database from being purged periodically. Then this regression slipped into in Merlin. As time passed by, this regression will manifest itself as a full /jffs partitions on more and more router.

Never had this problem before tho. First time having this problem. And I'm on .19 so idk if it's a bug or what. But yeah i will make a Cron job to clear it every day. Since I am only interested in the usage for today usually.

 

[toaruScar]

Never had this problem before tho. First time having this problem. And I'm on .19 so idk if it's a bug or what. But yeah i will make a Cron job to clear it every day. Since I am only interested in the usage for today usually.

In my case, it took the router 4 months to fill /jffs with TA’s data.

 

[MaziahBebop]

In my case, it took the router 4 months to fill /jffs with TA’s data.

Noob question coming at you: is this TA writing to /jffs a default setting for ASUS routers on Merlin builds? Wouldn't it wear out the flash over a long time? I'd be interested in disabling that feature if its possible.

 

[ColinTaylor]

Noob question coming at you: is this TA writing to /jffs a default setting for ASUS routers on Merlin builds? Wouldn't it wear out the flash over a long time? I'd be interested in disabling that feature if its possible.

This is the default behaviour with stock firmware. Flash wear has been discussed many times, it is not a problem.

 

[lightloch]

Thank you for the tutorial. This was ultimately the solution to my original problem where I couldn't rename any network clients. Now with this I also fixed the endless error log problem!