Release Asuswrt-Merlin 384.19 is now available

[bbunge]

Well, I really feel I gave 384.19 another good try. But I had another time when the router appeared to be connected and had no throughput. Back to stock Asus and rock solid!

 

[gattaca]

Does this give us any clues as to what is happening?

It's the same symptom I saw after I let the CRC errors run for maybe 12 hours when they first appeared. They seem to eventually "fill" the JFFS (or think it's full) and all the J* scripts start failing to run. You cannot create any files or directories on the JFFS. A reboot will give you relief but the CRC JFFS errors will return as long as the J* scripts are hitting the JFFS. You are headed to the same place... Is your AC86U log full of "CRC" errors?

I'll have more bandwidth over the weekend to maybe take a run at a couple of things. At some point I'm going to cut my losses and follow @L&LD nuke procedure or/and go back to 384.18. The gotcha on going back is the *@(* JFFS has supposedly now been resized to 47M down from 48M in 384.18, so IDK what the firmware will think about that. I think this little Asus gift is going to catch up with us sooner than later.

First on my agenda will be to do what I described about format followed by multiple clean 10 minute reboots... if that still proves to have JFFS errors (after putting the scripts back) then we are outta runway... there's no clean formatting the JFFS with 384.19 or my nand media is failing or both.

I think, now that I moved the J* scripts to USB drive, I could leave it and it might be OK as I've had no JFFS errors in 4 days.. but I feel it's only a matter of time with other things hitting the JFFS, only not has hard. YMMV.

 

[L&LD]

Well, I really feel I gave 384.19 another good try. But I had another time when the router appeared to be connected and had no throughput. Back to stock Asus and rock solid!

I feel you haven't tried hard enough!

 

[Centrifuge]

The gotcha on going back is the *@(* JFFS has supposedly now been resized to 47M down from 48M in 384.18, so IDK what the firmware will think about that. I think this little Asus gift is going to catch up with us sooner than later.

Good point of caution for anybody getting itchy to take the leap to .19 with a 86U. Patiently awaiting your next installment.

 

[SomeWhereOverTheRainBow]

Please explain "use the erase option" is there a command to "erase" the JFFS manually? I would think the JFFS has to be unmounted to perform a format etc ... on it. which is why ASUS has you reboot the router... it has code to "format the JFFS" before releasing it is my guess during boot time.

This is exactly what be i did. I disabled and formatted jffs with a reboot. I felt that this was self explanatory that is why i did not go through the motions of breaking it down that much.

 

[SomeWhereOverTheRainBow]

The flash is performing garbage collection (GC) on the media. It is getting CRC errors which means the data at that storage location is corrupted. See my several posts over the past few pages.

What I've done is move the J* AMTM scripts to use the USB drive fore now. I rebooted the router b/c at this point the JFFS flash is likely "full or not writable" I then moved all of the J* amtm scripts to use the USB... (open the menu on each, there is an option) and then rebooted the router. The CRC errors ceased b/c the J* scripts are not hammering the flash media anymore.

That does not mean the flash media is OK. I am so very suspect now that the format really did not complete properly or my flash nand media is failing.

The mistake I think I made (unless my flash media is really failing) is that @L&LD recommends that when formatting the JFFS, you do the steps I outlined in this post. See steps 7, 8, 9, 10 -> https://www.snbforums.com/threads/r...19-is-now-available.65801/page-24#post-615322

You disconnect the USB drive in the GUI, then remove the drive physically, then select to format the flash on next reboot. Hit Apply. Now reboot # 1. Let the router site for 5-10 minutes. Now reboot # 2. Let the router site 5-10 minutes. Now reboot # 3. Let the router sit 5-10 minutes. Now you restore the JFFS backup, reconnect the USB and reboot one more time. Check things out. He said these 3 reboots were required for the router to properly format the flash. He's posted this procedure several times in the past year or so.

I have not tried those repeating steps yet, but I will this weekend. I did not do the 3 reboot pass when I flashed to 384.19. I am also suspect of this file posted earlier. IDK what it does exactly. -> "/jffs/.sys/nc/nt_center.db"

Maybe this is a warning about us hammering the JFFS (cheap flash nand) so maybe moving the J* AMTM scripts to a real USB/SATA/SSD is better long term. IDK. YMMV.

See also -> https://www.snbforums.com/threads/r...19-is-now-available.65801/page-26#post-615897
See also -> https://www.snbforums.com/threads/r...19-is-now-available.65801/page-27#post-616090

The problem with "/jffs/.sys/nc/nt_center.db" is that it is becoming so large after flashing 384.19 that it is taking up the space of jffs. This file is not essential it rewrites it self after being deleted. It is a dumping ground of sorts that can be deleted any time. When flashing to 384.19 this file grows enormous in size causing the jffs error and jffs failures people are experiencing. One flaw with linux is once the storage gets maxed a lock triggers making jffs read only. That is why i tell users to delete this file before they create their backup. Once they restore their backup jffs will be a manageable size again eliminating this error.

 

[New2This]

Here is my other AX88U.... Running smooth..This one is running in AP mode, NOT running AiMesh

 

[cxterm]

Intel released a new driver again with additional improvements and fixes today.

Thanks. I have rolled back to 384.18. May try the new drive this weekend.

 

[gattaca]

The problem with "/jffs/.sys/nc/nt_center.db" is that it is becoming so large after flashing 384.19 that it is taking up the space of jffs. This file is not essential it rewrites it self after being deleted. It is a dumping ground of sorts that can be deleted any time. When flashing to 384.19 this file grows enormous in size causing the jffs error and jffs failures people are experiencing. One flaw with linux is once the storage gets maxed a lock triggers making jffs read only. That is why i tell users to delete this file before they create their backup. Once they restore their backup jffs will be a manageable size again eliminating this error.

TY for the detail! I looked earlier when you listed this as another upgrade gotcha. I noticed this file was created during startup with the (05 May date) before time was set. I will delete this file before backing up the JFFS during my next round of attempts to stabilize the AC86U.

Are you also saying "/jffs/.sys/nc/nt_center.db" will continue growing unbounded on 384.19? If so that's a FUBAR just waiting to happen. I do not recall the size from yesterday. It is captured below. The size of my JFFS has been steady now that J* tooling is on the USB SSD.

>: /jffs/.sys/nc# ls -lrt
-rw-r--r--    1 admin root        947200 May  5  2018 nt_center.db
...
/dev/mtdblock9           47.0M      7.9M     39.1M  17% /jffs

So many gotchas, so little time!

 

[SomeWhereOverTheRainBow]

TY for the detail! I looked earlier when you listed this as a problem. I noticed this file was created during startup with the (05 May date) before time was set. I will delete this file before backing up the JFFS during my next round of attempts to stabilize the AC86U.

Are you also saying "/jffs/.sys/nc/nt_center.db" will continue growing unbounded on 384.19? If so that's a FUBAR just waiting to happen. I do not recall the size from yesterday. It is captured below.

>: /jffs/.sys/nc# ls -lrt
-rw-r--r--    1 admin root        947200 May  5  2018 nt_center.db

So many gotchas, so little time!

no it just grows from the initial flash. it turns out that this is what causes the JFFS problems. once you correct the issue by restoring from your backup from the previous steps then the issue goes away. I have been completely stable since flashing and uploading the working backup.

 

[SomeWhereOverTheRainBow]

TY for the detail! I looked earlier when you listed this as a problem. I noticed this file was created during startup with the (05 May date) before time was set. I will delete this file before backing up the JFFS during my next round of attempts to stabilize the AC86U.

Are you also saying "/jffs/.sys/nc/nt_center.db" will continue growing unbounded on 384.19? If so that's a FUBAR just waiting to happen. I do not recall the size from yesterday. It is captured below.

>: /jffs/.sys/nc# ls -lrt
-rw-r--r--    1 admin root        947200 May  5  2018 nt_center.db

So many gotchas, so little time!

here is mine

root        490496 Sep  5 03:01 nt_center.db

notice the proper date stamp

 

[gattaca]

no it just grows from the initial flash. it turns out that this is what causes the JFFS problems. once you correct the issue by restoring from your backup from the previous steps then the issue goes away. I have been completely stable since flashing and uploading the working backup.

K, so we should add "rm /jffs/.sys/nc/nt_center.db" to the 384.19 upgrade instructions and maybe as a best practice moving forward since this may be a recurring gotcha?

 

[Xentrk]

no it just grows from the initial flash. it turns out that this is what causes the JFFS problems. once you correct the issue by restoring from your backup from the previous steps then the issue goes away. I have been completely stable since flashing and uploading the working backup.

Based on this, I think I will first disable jffs after taking the backup and before flashing the firmware. jffs won't be enabled so it is unable to write to "rm /jffs/.sys/nc/nt_center.db" and fill up jffs. After the upgrade, enable and format jffs, reboot, then restore jffs.

 

[MaziahBebop]

So far so good, running this firmware

Here is my other AX88U.... Running smooth..This one is running in AP mode, NOT running AiMesh

Another happy customer here:

VPN clients and servers up and running.
Dual Band Smart Connect, no issues.

 

[MarkyPancake]

Good point of caution for anybody getting itchy to take the leap to .19 with a 86U. Patiently awaiting your next installment.

Dirty upgraded my AC86U from 384.18 to .19 last week without any issues and didn't have to do anything with my JFFS afterwards either. Although, I don't use any custom scripts and only use features within the Merlin UI.

 

[shabbs]

Hit three weeks of uptime on my main router running .19. So far so good... @RMerlin

 

[gattaca]

^^^^ Yeap.. my conclusion is if you do not use JFFS for the AMTM J* scripts etc...or you do not have JFFS enabled, then you may not ever see this issue or it may bite in a different way we've not seen down the road. Let's hope very much for the former.! Peace.

 

[gattaca]

Based on this, I think I will first disable jffs after taking the backup and before flashing the firmware. jffs won't be enabled so it is unable to write to "rm /jffs/.sys/nc/nt_center.db" and fill up jffs. After the upgrade, enable and format jffs, reboot, then restore jffs.

Yes, that is what I was thinking too. Then I thought well if the JFFS is not enabled, would the router bother to format it? Rock/us/Rock
I was thinking backup, disable, check format, then do @L&LD procedure with 3 reboots, then enable, check format, 3 more reboots... and run from there. At this point, the "nuclear reset" is looking better and better.

 

[dave14305]

Yes, that is what I was thinking too. Then I thought well if the JFFS is not enabled, would the router bother to format it? Rock/us/Rock
I was thinking backup, disable, check format, then do @L&LD procedure with 3 reboots, then enable, check format, 3 more reboots... and run from there. At this point, the "nuclear reset" is looking better and better.

The JFFS partition is always enabled. The only thing you can disable via the GUI is the execution of custom scripts/configs. Otherwise the base firmware still uses /jffs for storage of all these non-script things.

 

[gattaca]

^^^^ TY. I was sort of leaning that way with the hidden dirs -> /jffs/.sys/...
So I think this is what I'm going to do... and if I still see issues, then it's @L&LD nuclear reset and start over.

Since already upgraded to 384.19, below. This is similar to what I have in the 384.19 upgrade procedure but not exactly.

a) rm /jffs/.sys/nc/nt_center.db (hidden file in .sys)
b) Backup JFFS
c) Remove USB in GUI, then remove USB physically
c) In GUI, Disable JFFS for scripts
d) Check format, Check Apply
e) Reboot
f) Sit 10 mins
g) Reboot
h) Sit 10 mins
i) Reboot
j) Sit 10 mins
k) Reconnect USB, re-enable JFFS
l) Reboot
m) Let sit 10 minutes
n) Restore JFFS
o) Reboot,...
p) Monitor closely all J* scripts when moved back to JFFS (1 by 1) to see if CRC errors return.
q) No work right? -> Deploy 2nd router as front-door backup for AC86U
r) Implement @L&LD "nuclear reset" option on this AC86U for ultra clean start.
n) ...