Release Asuswrt-Merlin 384.19 is now available

[gattaca]

Possible there's too much data on the restored backup, or a script in there that's filling jffs up too quick.

I do not think so. The booted linux OS, it thinks there 40.5M FREE on /jffs. The feels like a NAND level lack of space for its GC to use to move files around in whatever size erase blocks it uses a/o those bad blocks are making it impossible to run the GC. I really don't know anymore.

df -hT
...
/dev/mtdblock9 jffs2 47.0M 6.5M 40.5M 14% /jffs

I need to have my pastebin account reset.. basically, there are no huge files in the listing.. The only way I think to fix this at this point is a nuclear factory reset or some mtd format of the /jffs. I attached a txt file below. I'm not the only one with this issue after 384.19. I'm going to try some several weeks older jffs backups just to see, I'd love to be wrong.
The listing I posted is working with no CRC GC issues. I’ll do another later when family is more offline.

Update 07 Sep 2020 21:37 - I cannot reboot anymore tonight or I might lose some vital body part(s).

I blew away the JFFS again, rebooted and then restored my JFFS from a 3 week old backup vs. the one I took a few hours before the 384.19 upgrade. I am slowly am moving the J* utilities back to the JFFS to see if one triggers or any. So far, no CRC or GC errors on the first. I am flabbergasted at this point. I gotta hit the bed, check again tomorrow. So far the JFFS is at 47.0M and 8.3M used... - will see.

Update 08 Sep 2020- 06:55 - BLUF: The CRC/GC errors keep returning whenever there's heavier I/O to the JFFS (connman for instance) so moving the J1* scripts to USB SSD will buy time. A nuclear reset is on the menu and then if the AC86U displays any CRC/GC errors, it's going to become a simple WAP or get booted to the electronics recycling center. I've spent 2+ days chasing, debugging and performing problem determination on this 384.19 (dirty) + AC86U ASUS JFFS resize *@(*(@. I have little confidence my AC86U router is going to play nice with 384.19 + AMTM + J1 scripts on JFFS or maybe anything down the road.

The JFFS CRC/GC errors returned soon after switching spdMerlin to JFFS and reinstalling connmon to USB, followed by the GC errors - along with some interesting traps I do not recall seeing while running spdMerlin.

Doing anything else to this AC86U is solely for problem determination.

Errors returned after reinstalling connman on JFFS

Sep  7 21:31:06 connmon: Welcome to connmon v2.6.0, a script by JackYaz
Sep  7 21:31:07 connmon: Checking your router meets the requirements for connmon
Sep  7 21:31:07 connmon: Installing required packages from Entware
Sep  7 21:31:09 connmon: New version of connmonstats_www.asp downloaded
Sep  7 21:31:09 connmon: Mounting connmon WebUI page as user1.asp
Sep  7 21:41:00 spdMerlin: Starting speedtest using auto-selected server for WAN interface
Sep  7 21:41:00 kernel: jffs2: Data CRC bdf8b642 != calculated CRC aeffdec3 for node at 02997594
Sep  7 21:41:00 kernel: speedtest[4145]: unhandled level 3 translation fault (7) at 0x005f4000, esr 0x92000007
Sep  7 21:41:00 kernel: pgd = ffffffc007921000
Sep  7 21:41:00 kernel: [005f4000] *pgd=000000001913b003, *pud=000000001913b003, *pmd=0000000019153003, *pte=0000000000000000
Sep  7 21:41:00 kernel: CPU: 1 PID: 4145 Comm: speedtest Tainted: P           O    4.1.27 #2
Sep  7 21:41:00 kernel: Hardware name: Broadcom-v8A (DT)
Sep  7 21:41:00 kernel: task: ffffffc01e86a080 ti: ffffffc00a7e8000 task.ti: ffffffc00a7e8000
Sep  7 21:41:00 kernel: PC is at 0x565dac
Sep  7 21:41:00 kernel: LR is at 0x565d5c
Sep  7 21:41:00 kernel: pc : [<0000000000565dac>] lr : [<0000000000565d5c>] pstate: 80000000
Sep  7 21:41:00 kernel: sp : 0000007fc02bb000
Sep  7 21:41:00 kernel: x29: 0000007fc02bb000 x28: 00000000005c1580
Sep  7 21:41:00 kernel: x27: 000000000000001b x26: 0000000000634308
Sep  7 21:41:00 kernel: x25: 0000000000634300 x24: 0000007fc02bb068
Sep  7 21:41:00 kernel: x23: 000000000000139c x22: 0000000000000000
Sep  7 21:41:00 kernel: x21: 00000000006342f8 x20: 000000000000001b
Sep  7 21:41:00 kernel: x19: 00000000005f4000 x18: 0000000000000000
Sep  7 21:41:00 kernel: x17: 0000000000000000 x16: 0000000000000000
Sep  7 21:41:00 kernel: x15: 0000000000000000 x14: 0000000000000000
Sep  7 21:41:00 kernel: x13: 0000000000000000 x12: 0000000000638ba0
Sep  7 21:41:00 kernel: x11: 0000000000000001 x10: 00000000005833fa
Sep  7 21:41:00 kernel: x9 : 0000000012db74c4 x8 : 0000007fc02bc870
Sep  7 21:41:00 kernel: x7 : 0000000000000000 x6 : 0000000000000010
Sep  7 21:41:00 kernel: x5 : 000000000000001b x4 : 000000000050da58
Sep  7 21:41:00 kernel: x3 : 00000000ffffffff x2 : 00000000005c1580
Sep  7 21:41:00 kernel: x1 : 000000000050da58 x0 : 0000000000000038
Sep  7 21:41:00 spdMerlin: Speedtest results -  -
Sep  7 21:50:36 uiDivStats: Stats updated successfully
Sep  7 21:50:45 uiDivStats: Stats updated successfully
Sep  7 22:00:08 uiDivStats: Stats updated successfully
Sep  7 22:05:01 Diversion: found 1 new YouTube hosts, total is 1440 (counter at 18 of 30)
Sep  7 22:11:00 spdMerlin: Starting speedtest using auto-selected server for WAN interface
Sep  7 22:11:00 kernel: jffs2: Data CRC bdf8b642 != calculated CRC aeffdec3 for node at 02997594
Sep  7 22:11:00 kernel: speedtest[22946]: unhandled level 3 translation fault (7) at 0x005f4000, esr 0x92000007
Sep  7 22:11:00 kernel: pgd = ffffffc013d1c000
Sep  7 22:11:00 kernel: [005f4000] *pgd=000000000204c003, *pud=000000000204c003, *pmd=0000000019284003, *pte=0000000000000000
Sep  7 22:11:00 kernel: CPU: 0 PID: 22946 Comm: speedtest Tainted: P           O    4.1.27 #2
Sep  7 22:11:00 kernel: Hardware name: Broadcom-v8A (DT)
Sep  7 22:11:00 kernel: task: ffffffc013c5e100 ti: ffffffc0015b4000 task.ti: ffffffc0015b4000
Sep  7 22:11:00 kernel: PC is at 0x565dac
Sep  7 22:11:00 kernel: LR is at 0x565d5c
Sep  7 22:11:00 kernel: pc : [<0000000000565dac>] lr : [<0000000000565d5c>] pstate: 80000000
...
Sep  7 22:11:01 kernel: x1 : 000000000050da58 x0 : 0000000000000038
Sep  7 22:11:01 spdMerlin: Speedtest results -  -
Sep  7 22:41:00 spdMerlin: Starting speedtest using auto-selected server for WAN interface
...
Sep  8 00:11:01 spdMerlin: Speedtest results -  -
Sep  8 00:37:03 kernel: jffs2: Data CRC bdf8b642 != calculated CRC aeffdec3 for node at 02997594
Sep  8 00:37:03 kernel: jffs2: read_cache_page() returned error: -5
Sep  8 00:37:03 kernel: jffs2: Error garbage collecting node at 02997594!
Sep  8 00:37:03 kernel: jffs2: No space for garbage collection. Aborting GC thread
Sep  8 00:37:59 kernel: jffs2: Data CRC bdf8b642 != calculated CRC aeffdec3 for node at 02997594
Sep  8 00:37:59 kernel: jffs2: read_cache_page() returned error: -5
Sep  8 00:37:59 kernel: jffs2: Error garbage collecting node at 02997594!
Sep  8 00:38:59 kernel: jffs2: Data CRC bdf8b642 != calculated CRC aeffdec3 for node at 02997594
Sep  8 00:38:59 kernel: jffs2: read_cache_page() returned error: -5
Sep  8 00:38:59 kernel: jffs2: Error garbage collecting node at 02997594!
Sep  8 00:39:59 kernel: jffs2: Data CRC bdf8b642 != calculated CRC aeffdec3 for node at 02997594
Sep  8 00:39:59 kernel: jffs2: read_cache_page() returned error: -5
...
Sep  8 00:40:02 kernel: jffs2: Error garbage collecting node at 02997594!
Sep  8 00:40:59 kernel: jffs2: Data CRC bdf8b642 != calculated CRC aeffdec3 for node at 02997594
Sep  8 00:40:59 kernel: jffs2: read_cache_page() returned error: -5
Sep  8 00:40:59 kernel: jffs2: Error garbage collecting node at 02997594!
Sep  8 00:41:00 spdMerlin: Starting speedtest using auto-selected server for WAN interface
Sep  8 00:41:00 kernel: jffs2: Data CRC bdf8b642 != calculated CRC aeffdec3 for node at 02997594
Sep  8 00:41:00 kernel: speedtest[15133]: unhandled level 3 translation fault (7) at 0x005f4000, esr 0x92000007
Sep  8 00:41:00 kernel: pgd = ffffffc001966000
Sep  8 00:41:00 kernel: [005f4000] *pgd=0000000008715003, *pud=0000000008715003, *pmd=0000000019153003, *pte=0000000000000000
....
<EOF>

 

[wuckawucka]

As this is a bit off topic, if you want to trouble shoot further, we can create a separate thread.

(regarding RSTAT counting VLAN tagged packets) - yes, I'll keep out of this thread now, will try some of your suggestions. Thanks.

 

[faria]

Upgraded my RT86U today without USB and rebooted with USB plugged in. Noticed that jffs partition is gone and when I ssh to the router I cannot invoke amtm utility. Tried restoring my jffs backup but this does nothing. What do you guys recommend I try next to get this thing working again?

Take a gander at the first page

 

[Smokey613]

They didn't increase it, they allocated it. There wasn't any before.

That is what I meant, I just phrased it incorrectly. At any rate, I am very happy with my two RT-AC86U routers and have no plans to replace them. Once I stopped using anything requiring the TM agreements, it runs much better. I did discover one thing while running the Asus RC2 AiMesh 2 firmware. I can’t run an Asus router without Merlin.

 

[dugaduga]

Working Flawlessly, great work Merlin!

 

[gattaca]

Rather than stringing this out over several posts.. I put all the latest test info in this earlier posting.. -> https://www.snbforums.com/threads/r...19-is-now-available.65801/page-31#post-617153

BLUF: The CRC/GC errors return whenever there's heavy I/O to the JFFS (using spdMerlin or connman, ... for instance). Relocating the J1* script's output to the USB SSD buys time as the GC/CRC errors stop immediately. A nuclear reset in on the menu. Then if my AC86U keeps displaying CRC/GC errors after a full reset, it will become a simple WAP with no routing duties or get booted to the electronics recycling center. I've spent 2+ days chasing, debugging and performing problem determination on this 384.19 (dirty upgrade) + ASUS's AC86U JFFS resize *@(*(@. I have little confidence my AC86U router will play nice with 384.19 + AMTM + J1* scripts hitting JFFS or maybe anything down the road.

Let me be clear - the JFFS resizing is something ASUS did to help ASUS debug the AC86U down the road. Yes, I think ASUS's JFFS resizing is related to what several of us are reporting with JFFS issues such as GC/CRC errors after upgrading to 384.19. (See earlier posts).

I do not think any of the JFFS GC/CRC issues are related to Jack's J* awesome scripts, Merlin's excellent work or TLC's AMTM family of tooling - far from it! You guys rock!

I'm moving the J* scripts back to USB and rebooting as the work week is starting. Stay safe, stay alive! Peace.

 

[Adamm]

Rather than stringing this out over several posts.. I put all the latest test info in this earlier posting.. -> https://www.snbforums.com/threads/r...19-is-now-available.65801/page-31#post-617153

BLUF: The CRC/GC errors keep returning whenever there's heavier I/O to the JFFS (spdMerlin or connman for instance) so moving the J1* scripts to USB SSD will buy time. A nuclear reset in on the menu and then if the AC86U keeps displaying CRC/GC errors after a full reset, it's going to become a simple WAP or get booted to the electronics recycling center. I've spent 2+ days chasing, debugging and performing problem determination on this 384.19 (dirty) + AC86U ASUS JFFS resize *@(*(@. I have little confidence my AC86U router is going to play nice with 384.19 or maybe anything down the road.

Let me be clear - the JFFS resizing is something ASUS did to help ASUS debug the AC86U down the road. Yes, I think it is related to what several of us are reporting with JFFS GC/CRC errors after upgrading to 384.19. (See earlier posts).

I do not think any of these JFFS GC/CRC issues are related to Yaz J1* awesome scripts, Merlin's excellent work or the AMTM family of tooling - far from it!

I'm moving the J* scripts back to USB as the work week is starting and it's gotta be rock stable. Stay safe, stay alive! Peace.

Reading your posts the common denominator seems to be you are restoring an old backup after formatting your JFFS partition. The logical troubleshooting step seems to be a completely fresh format without any old backups and see if your issue persists.

 

[gattaca]

Reading your posts the common denominator seems to be you are restoring an old backup after formatting your JFFS partition. The logical troubleshooting step seems to be a completely fresh format without any old backups and see if your issue persists.

10-4... I have not tried a JFFS format + all new AMTM + manually moving my custom scripts yet. 100% correct. Great point!
The Diversion + SkyNet whitelists + my NextDNS setup scripts (not the client) are the most critical elements. (mainly /jffs/configs)
I could move those back into JFFS manually if the initial setup of the AMTM J* scripts proved no GC/CRC errors on a "fresh" format. I'll have to try that and post back. One of my weaknesses is I don't like to get beat by things like this... sometimes my better half has to say, "call it / game over"... for me to back down. Sometimes it's just more cost effective to chunk the "whatever" and start over. And she's usually always right about that one! (growl...)
I think after that, it's a full factory nuclear reset path. Then if it is bad/worn NAND, it's not going to matter b/c I'm going to see this same sort of process b/c the router creates the MTD filesystem each boot pass. BTW, the /JFFS lives on MTD9 "Misc2" and there's an ASUS command to erase just "Misc2".. .

May 5 01:05:11 kernel: nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
May 5 01:05:11 kernel: bcm63xx_nand ff801800.nand: Adjust timing_1 to 0x65324458 timing_2 to 0x80040e54
May 5 01:05:11 kernel: bcm63xx_nand ff801800.nand: detected 256MiB total, 128KiB blocks, 2KiB pages, 16B OOB, 8-bit, BCH-4
May 5 01:05:11 kernel: Bad block table found at page 131008, version 0x01
May 5 01:05:11 kernel: Bad block table found at page 130944, version 0x01
May 5 01:05:11 kernel: nand_read_bbt: bad block at 0x0000032e0000
May 5 01:05:11 kernel: nand_read_bbt: bad block at 0x000007b00000
May 5 01:05:11 kernel: nand_read_bbt: bad block at 0x00000f880000
May 5 01:05:11 kernel: >>>>> For primary mtd partition rootfs, cferam/vmlinux.lz mounted as JFFS2, vmlinux fs mounted as UBIFS <<<<<
May 5 01:05:11 kernel: Secondary mtd partition rootfs_update detected as JFFS2 for cferam/vmlinux source and UBIFS for vmlinux filesystem
May 5 01:05:11 kernel: setup_mtd_parts: misc indx 2 name misc3 nvram configured size 1
May 5 01:05:11 kernel: setup_mtd_parts: name misc3 configured size 0x00100000 offset 0xF600000
May 5 01:05:11 kernel: setup_mtd_parts: misc indx 1 name misc2 nvram configured size 47
May 5 01:05:11 kernel: setup_mtd_parts: name misc2 configured size 0x02f00000 offset 0xC700000
May 5 01:05:11 kernel: setup_mtd_parts: misc indx 0 name misc1 nvram configured size 8
May 5 01:05:11 kernel: setup_mtd_parts: name misc1 configured size 0x00800000 offset 0xBF00000
May 5 01:05:11 kernel: Creating 11 MTD partitions on "brcmnand.0":
May 5 01:05:11 kernel: 0x000006460000-0x00000bf00000 : "rootfs"
May 5 01:05:11 kernel: 0x000000560000-0x000006000000 : "rootfs_update"
May 5 01:05:11 kernel: 0x00000f700000-0x00000ff00000 : "data"
May 5 01:05:11 kernel: 0x000000000000-0x000000100000 : "nvram"
May 5 01:05:11 kernel: 0x000000100000-0x000006000000 : "image_update"
May 5 01:05:11 kernel: 0x000006000000-0x00000bf00000 : "image"
May 5 01:05:11 kernel: 0x000006000000-0x000006460000 : "bootfs"
May 5 01:05:11 kernel: 0x000000100000-0x000000560000 : "bootfs_update"
May 5 01:05:11 kernel: 0x00000f600000-0x00000f700000 : "misc3"
May 5 01:05:11 kernel: 0x00000c700000-0x00000f600000 : "misc2"
May 5 01:05:11 kernel: 0x00000bf00000-0x00000c700000 : "misc1"

 

[rdallen]

I just updated my RT-AC86U from 384.17. All working fine. But I notice I have a weird option down the left in Chinese - from a bit of Googling it looks like UU Accelerator - whatever that is.

Mine is an imported model, so am used to seeing a bit of Chinese (only when I factory reset though). Have I missed something?

 

[RMerlin]

Mine is an imported model, so am used to seeing a bit of Chinese (only when I factory reset though). Have I missed something?

No, this is a feature available only on Chinese models. It gets shown/hidden based on the router's country code, and the code controlling this is closed source.

 

[rdallen]

No, this is a feature available only on Chinese models. It gets shown/hidden based on the router's country code, and the code controlling this is closed source.

Thanks Eric. All makes sense.

 

[QuikSilver]

Anyone getting or seen these?

Spoiler

Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 1 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 2 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 3 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 4 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 5 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 6 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 7 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 8 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 9 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 10 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 11 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 12 has been set to DISABLE
Sep 8 10:49:51 RT-AC86U-AEA8 kernel: The For ALL DEVICES flag of Prof 13 has been set to ENABLE

 

[L&LD]

@gattaca, what is 'BLUF'?

 

[visortgw]

@gattaca, what is 'BLUF'?

Bottom Line Up Front.

 

[L&LD]

Thanks, but now, what is that?

 

[QuikSilver]

Thanks, but now, what is that?

Had to look it up myself....

"A BLUF is a paragraph where the conclusions and recommendations are placed at the beginning of the text, rather than the end, in order to facilitate rapid decision making. "

 

[L&LD]

Ah! I thought someone was trying to pull something over my eyes.

 

[ttgapers]

Ok,

I have been following the thread, and I as well noticed the jffs2 issue on two routers. Myself I followed the steps below when I initially upgraded:

1. Full backup via nsru
2. Installed upgrade
3. Full format of jffs
4. Reboot
5. Restore jffs
6. Noticed error shortly thereafter

Since then I am hesitant to go through the exercise again (especially since I already formatted). I have gone ahead and done the following with no errors since (2 days now). I will continue to monitor.

1. rm /jffs/.sys/nc/nt_center.db (backup it up if you are paranoid - I have seen zero impact on removal)
2. Reboot

My last error was shortly before the steps above (on both routers).

Hope it helps some people (with a more complex setup).

 

[JacquesR]

I said this on page 16 (I think, but a while ago), but I've had zero problems on my 86U with 384.19 after reformatting JFFS and simply reinstalling all the scripts I was running (no restoring from backup). Most/all of the ones I use allowed for backups, so it was 10 minutes work at most.

 

[ttgapers]

I said this on page 16 (I think, but a while ago), but I've had zero problems on my 86U with 384.19 after reformatting JFFS and simply reinstalling all the scripts I was running (no restoring from backup). Most/all of the ones I use allowed for backups, so it was 10 minutes work at most.

Yup I did note your post as well and that's why I listed what I did, rather than re-installing all the scripts. I'd love if other users test the steps I went through above. It is worth a try before the nuclear option(s). Perhaps it may zero in that the culprit of all these issues is the nt_center.db going out of control.

Cheers.