[Solved] RT-AX88U jffs/ubi brain mix

[Wendigogo]

Hello there,

I got issue with my RT-AX88U (NON pro version) with my /jffs mount point and I had read a lot of threads here and there about it. Probably too much as everything is mixed in my brain.

First, I had a Garbage Collector issue and I discovered there is still space left on my /jffs. The problem is taht it's on Read Only mode while it was mounted with RW.

RT-AX88U:/tmp/home/root# touch /jffs/test
touch: /jffs/test: No space left on device

RT-AX88U:/tmp/home/root# df -h /jffs/
Filesystem                Size      Used Available Use% Mounted on
/dev/mtdblock9           63.0M     48.1M     14.9M  76% /jffs

RT-AX88U:/tmp/home/root# mount
ubi:rootfs_ubifs on / type ubifs (ro,relatime)
[snip]
/dev/mtdblock9 on /jffs type jffs2 (rw,noatime)

First I restart my router but it doesn't help.
Next, I restore my /jffs partition through GUI : no issue during the process (while it is RO ) but it doesn't help too.
I then thought about wiping it but the option is not available anymore.

So I dig around and found a post from @RMerlin stating that :

It's never been there. These router uses the UBIFS filesystem, not JFFS. And since the router now stores a lot of its configuration there, erasing its content makes no sense unless you were doing a factory default reset.

The last router that used JFFS was the RT-AC5300.

(source: https://www.snbforums.com/threads/asuswrt-merlin-3004-388-7-is-now-available.89811/post-907028)

Then I started to see "mount" results showing /jffs as ubi mounted point (example) and my brain explode!

Is my model (AX88U) should use the UBI filesystem for /jffs?
- If not, why the "jffs wipe" option is not available?
- If yes, why "mount" return doesn't show ubi fs for /jffs mounting point? And as ubi is a RO filesystem how user scripts are managed?

So my questions are:
- in what configuration my router should be? jffs2, ubi?
- I probably messed up things on it. How could I get back on track? A full wipe? But if I restore my configuration (including /jffs part) will it go mad again?

Thanks for your help and sorry for this long post.

Enjoy!

PS : I'm on latest firmware (3004.388.7)

 

[ColinTaylor]

There is nothing in your output that says the /jffs filesystem is read-only. In fact it actually says it's read/write.

Although your router uses UBI for its root filesystem (which is read-only) it still uses standard MTD for /jffs.

I suggest you backup your current settings and the JFFS partition. Then do a hard factory reset. Restore your old settings followed by restoring the JFFS. Reboot and see if the problem persists. If it does then examine the router's System Log.

[Wireless Router] ASUS router Hard Factory Reset - Method 2 | Official Support | ASUS Global

 

[Wendigogo]

There is nothing in your output that says the /jffs filesystem is read-only. In fact it actually says it's read/write.

Although your router uses UBI for its root filesystem (which is read-only) it still uses standard MTD for /jffs.

I suggest you backup your current settings and the JFFS partition. Then do a hard factory reset. Restore your old settings followed by restoring the JFFS. Reboot and see if the problem persists. If it does then examine the router's System Log.

[Wireless Router] ASUS router Hard Factory Reset - Method 2 | Official Support | ASUS Global

Thanks for your support.

That's exactly the point : /jffs is mounted RW but I can't write on it. It says that there's no space left but "df" shows there is around 15MB free. So I assumed that it as fall back to RO mode for whatever reason.

I'll try to do a hard factory reset and see if problem is solved (hoping I won't loose anything during the process).

 

[Ripshod]

Thanks for your support.

That's exactly the point : /jffs is mounted RW but I can't write on it. It says that there's no space left but "df" shows there is around 15MB free. So I assumed that it as fall back to RO mode for whatever reason.

I'll try to do a hard factory reset and see if problem is solved (hoping I won't loose anything during the process).

Wow, only 15MB free? I still have 63MB free with all my scripts. Are you running any of the Trend Micro stuff?
Login on ssh and paste this - then give us the results

find /jffs -printf '%s %p\n'| sort -nr | head -10

 

[Wendigogo]

Wow, only 15MB free? I still have 63MB free with all my scripts. Are you running any of the Trend Micro stuff?
Login on ssh and paste this - then give us the results

find /jffs -printf '%s %p\n'| sort -nr | head -10

Here you are :

# find /jffs -printf '%s %p\n'| sort -nr | head -10
43160576 /jffs/.asdbk/main_checker.sh_bk
13021184 /jffs/.sys/TrafficAnalyzer/TrafficAnalyzer.db
1921024 /jffs/.sys/diag_db/conn_diag_1670112000.db
1245184 /jffs/.sys/diag_db/conn_diag_1670198400.db
1106853 /jffs/signature/rule.trf
353197 /jffs/master.zip
262205 /jffs/syslog.log-1
219536 /jffs/.acme.sh/acme.sh
208126 /jffs/letsencrypt/acme.sh-master/acme.sh
127649 /jffs/syslog.log

I dig into this /jffs/.asdbk/main_checker.sh_bk file and it looks like it's all my router has done since november (at least) and all in readable state. Including password for inadyn stuff, RSA private key from acme.sh script, amtm events and scripts and so on.
What the hell that kind of data are in a 666 file (read/write access for EVERYONE) ???

I just hard reboot my router and quickly check dmesg. It's already full of "no free space/garbage collecting" error but I also got a red line :

[ERROR pktrunner] runnerUcast_inet6addr_event,187: Could not rdpa_system_ipv6_host_address_table_find ret=-5

Enjoy !

 

[ColinTaylor]

There are multiple reports of main_checker.sh_bk becoming problematically large. See here for example.

This file should be a copy of something else that asd is checking for malware. So do you recognise the contents - an archive file for example?

In the meantime just delete it. But unless you can identify the source data the issue is likely to come back.

P.S. There is no problem with this file having 666 permissions as everything runs as root. The router isn't a multiuser Linux desktop or server.

 

[Ripshod]

Straight away the asd file seems like just a backup, so you may simply be able to delete that - wait for confirmation though.
You have Statistics enabled in Traffic Analyzer. Unless you're actively looking for something specific that can be disabled to get you another 13MB back (you may have to manually delete that .db file).

*EDIT* Pipped!

 

[Wendigogo]

There are multiple reports of main_checker.sh_bk becoming problematically large. See here for example.

This file should be a copy of something else that asd is checking for malware. So do you recognise the contents - an archive file for example?

In the meantime just delete it. But unless you can identify the source data the issue is likely to come back.

This file is kind of a merge of anything my router has done. And a simple cat give access to anything in clear: some binary-like parts and some text parts:

./scripts/firewall-start@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@0000777@0000000@0000000@00000000147@141
15502737@014406@ 0@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ustar  @admin@@@@@@@@@@@@@@@@@@@@@@@@@@@root@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@#!/bin/sh
iptables -I INPUT -p tcp --dport 80 -j ACCEPT
iptables -I INPUT -p tcp --dport 443 -j ACCEPT
[snip]
/scripts/ddns-start.dyndns@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@0000644@0000000@0000000@00000001566@14325615276@015034@ 0@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ustar  @admin@@@@@@@@@@@@@@@@@@@@@@@@@@@root@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@#!/bin/shM
M
###M
# Git development:M
# https://gist.github.com/atais/9ea6595072096ab8077f619bd3648da8M
# Based onM
# https://github.com/RMerl/asuswrt-merlin.ng/wiki/Custom-DDNS#google-domainsM
# https://github.com/RMerl/asuswrt-merlin.ng/wiki/Custom-DDNS#bind9-ddns-using-nsupdateM
###M
M
#set -uM
M
# USER=YOUR USER IN DDNS CONFIGM
# PASS=YOUR PASSWORD IN DDNS CONFIGM
# HOST=mydomain.comM
M
USER=YES they're clearly readable here !
PASS=YES they're clearly readable here !
HOST=YES they're clearly readable here !
M

[snip]

Y��V��C�@���k�#f�S�4�]'D��XLb��Bk�fϭ�c��ښ��;�=Lo�D�ڞDѤb�"X��-�A{/[��w?q�ErY���u~4UE��C(�O��C��S[_{]\K� |��),"*n�i�!�����"���7���__8@�   �V�s�ȑA[g�|�O�Ld��"l=ݥ0M���f9K|�3��5�磖JY�3y�UN<�g�%|\M��.�Ď�u$
��\�)�!�Rȅ�5fr"+j÷!KQl�--�!ZXV�pZI�T\��Bݺ����z��vj�ZK��U��:u�ل��f�j�f3E)�W_T�k�Lc�P[��ذ����XėU�sg,b��X:�{Wa
P�SQf|P+v\k�@�M|ѵ��9'E)D�GnK���^MjB��zf�)&�g/Q��N2#R�QFQ=��2WA�(E�����/�(�Ql��']&�{F���OPE`
�A7*�CEe�{\:PWNH�?A[�ٝ�_/����O���]�ۂ��8K�v�����(ɆK&�\VU5���@igc=W�O}rz��9        Ү�^Z8o�;�0k�CI,0}O^dVܜ�r�������Z��3�����#O_?�F�O�?$���_�R�{��Yx����������{�������﯇_�??�����?��_L��ߟ������{����������{���z����G
ҿ?���ID!0T'�O�_�Qo��g���{����������{���������Gҿ?���G�GP�{��Yx����������{���������7�x㯃�DN-�z@T@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@./wlcnt.json@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@0000644@0000000@0000000@00000013520@14613720732@012046@ 0@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ustar
  @admin@@@@@@@@@@@@@@@@@@@@@@@@@@@root@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@{"24:41:8C:AC:D3:75":{"start":1713974807,"end":1714397657,"alive":0},"02:11:32:2B:EC:1B":{"start":1707253248,"end":1714397657,"alive":0},"02:C8:85:C3:44:2D":{"start":171433
2713,"end":0,"alive":1},"E6:4B:7C:BB:98:F1":{"start":1688223404,"end":1714397657,"alive":0},"00:11:32:AE:CA:FF":{"start":1687167625,"end":0,"alive":1},"48:B0:2D:37:86:35":{"start":1710786434,"end":0,"alive":1},"
BC:CE:25:25:DD:66":{"start":1709982598,"end":1714397657,"alive":0},"90:E8:68:BC:CB:6C":{"start":1701267954,"end":0,"alive":1},"5C:FB:3A:BB:82:51":{"start":1714367026,"end":1714397657,"alive":0},"F0:86:20:4A:14:3
6":{"start":1692993428,"end":0,"alive":1},

[snip]

./addons/amtm/entware.mod@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@0000666@0000000@0000000@00000031604@14526505235@014411@ 0@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ustar  @admin@@@@@@@@@@@@@@@@@@@@@@@@@@@root@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@#!/bin/s
h
#bof
check_ps_version(){
        psVersion="$(pixelserv-tls -v | awk 'NR == 1 {print $2}')"
        [ -z "$psVersion" ] && psVersion="likely v.Kk"
        case $psVersion in
          v*|*v.K*) ;;
          * ) psVersion="$psVersion" ;;
        esac
}

entware_installed(){

etc.

I can't delete it as the /jffs mounting point seems to be read only atm (see my first post).

Enjoy!

 

[Ripshod]

read only

My bad, I forgot. Can still disabled Traffic Analyzer's Statistics and see what happens after a reboot.

 

[ColinTaylor]

This file is kind of a merge of anything my router has done. And a simple cat give access to anything in clear: some binary-like parts and some text parts:

That is a tar file. Are you running some sort of backup script on your router.

I can't delete it as the /jffs mounting point seems to be read only atm (see my first post).

You said you rebooted the router but did you try the hard reset like I suggested?

 

[Ripshod]

That is a tar file

Agreed. Some backup may have failed, or it's being written to with each backup.
Also agree with a factory reset at this stage.
In future the safest way I've found to do backups is with BACKUPMON.

 

[Wendigogo]

My bad, I forgot. Can still disabled Traffic Analyzer's Statistics and see what happens after a reboot.

It's already OFF. But I may have enabled it at some point in the past.

That is a tar file. Are you running some sort of backup script on your router.

A tar file! It's obvious once said.
No backup script on my router. I only back up through the GUI atm. But I saw BACKUPMON when exploring this forum.

# ll /jffs/scripts/
-rwxr-xr-x    1 Boss     root           229 May 25  2023 ddns-start*
-rw-r--r--    1 Boss     root           886 Oct 25  2022 ddns-start.dyndns
-rw-rw-rw-    1 Boss     root           934 May 25  2023 ddns-start.dyndns.v2
-rw-r--r--    1 Boss     root           173 Sep  6  2021 ddns-start.old
-rwxrwxrwx    1 Boss     root           103 Sep  6  2021 firewall-start*
-rwxr-xr-x    1 Boss     root            69 Jan  4  2023 post-mount*
-rwxr-xr-x    1 Boss     root           207 Dec 20 08:42 services-start*
-rwxr-xr-x    1 Boss     root            58 Jan  4  2023 services-stop*

Yes, I messed up with ddns stuff ^_^.

# cat /jffs/scripts/services-start
#!/bin/sh
cru a "acme.sh" '0 2 24 * * /jffs/.acme.sh/acme.sh --cron --home "/opt/scripts/acme.sh" --server  letsencrypt > /dev/null'
cru a "restart httpds every 8 hours" '0 */8 * * * service restart_httpds'

Here httpds is eating my ram.

# cat /jffs/scripts/post-mount
#!/bin/sh
. /jffs/addons/diversion/mount-entware.div # Added by amtm

THIS is strange as Diversion is not installed. I double-checked in amtm entware installed packages.

You said you rebooted the router but did you try the hard reset like I suggested?

Not yet, but it's planned asap.

 

[Ripshod]

Not yet, but it's planned asap.

That would be a good opportunity to install BACKUPMON. It'so powerful it even restored my WiFi MACs when they were zeroed.

 

[ColinTaylor]

Not yet, but it's planned asap.

After you use the GUI to backup your JFFS partition you can use something like 7-Zip to remove the main_checker.sh_bk file from the archive. That way, when you restore the archive you don't need to go in and manually delete that file from the router.

 

[RMerlin]

Try this to remove the file on a filled up volume:

: > /jffs/.asdbk/main_checker.sh_bk

It will truncate the file, which will work even when the rm command won't work.

 

[Wendigogo]

Try this to remove the file on a filled up volume:

: > /jffs/.asdbk/main_checker.sh_bk

It will truncate the file, which will work even when the rm command won't work.

I still have space left but:

# : > /jffs/.asdbk/main_checker.sh_bk
-sh: can't create /jffs/.asdbk/main_checker.sh_bk: No space left on device

I guess my jffs partition ( /dev/mtdblock9 on my AX88U) is locked in Read Only mode but I can't find the origin of this behavior.

I hope a hard reset will solve this once and for all.

Enjoy!

 

[Ripshod]

I still have space left but:

# : > /jffs/.asdbk/main_checker.sh_bk
-sh: can't create /jffs/.asdbk/main_checker.sh_bk: No space left on device

I guess my jffs partition ( /dev/mtdblock9 on my AX88U) is locked in Read Only mode but I can't find the origin of this behavior.

I hope a hard reset will solve this once and for all.

Enjoy!

How did you get on @Wendigogo ?

 

[Wendigogo]

How did you get on @Wendigogo ?

Hi,

I still don't wipe my router : too many people depending on this connection at home. As I'm alone today, I'll probably do it today, after a second backup and screenshots of all UI pages...

I'll report here if everything gone right.

Enjoy!

 

[Wendigogo]

@Ripshod @ColinTaylor @RMerlin : unfortunately, the hard wipe doesn't solve my issue.

That was really strange because I only restored configuration backup file just from greeting Asus screen after reset but my whole /jffs partition is still here ! Without restoring the jffs tar file.
Like if it didn't wipe it: still "read only", still this big "main_checker.sh_bk" file in /jffs/.asdbk etc.

Some of you already had seen this before? How can I could wipe it? Through GUI Administration>Restore/Save/Upload Setting>Factory default or on switching Administration>System>Enable JFFS custom scripts and configs to "No"?

Thanks for your help.

 

[L&LD]

If you're restoring backup config files, you're not doing a full reset.