Release Asuswrt-Merlin 384.19 is now available

[doczenith1]

^^^ That's very similar to what I did. The CRC errors surface about 4-5 hours after I restarted the router the last time with the JFFS restored... but not at first. Once all the J* amtm scripts kick in, that seems to be hammering the JFFS. I also checked a restore of the JFFS is just a simple tar file the router untars (as files) back to the JFFS.. so the CRC errors are likely already existing on the JFFS from the upgrade.

I have also reformatted the JFFS now maybe 5x and restored two different copies of the JFFS as it was before .19... nope.. CRC errors will eventually return.

What I'm doing now is I uninstalled all the J* scripts and am slowly adding them back 1 by 1.. and telling the scripts to use the USB drive (which is a real SSD)... I want to see if getting all the I/O off the JFFS behaves. So far, it has with ntpMerlin installed and pointing to USB.

This is uber frustrating as 384.18 has worked perfectly for months with no CRC errros. IDK WTF Asus did with resizing the JFFS, but it's breaking a lot of our tooling. (That peeve is NOT directed at RMerlin's excellent work!!)

At this point, while others have had no issues with the upgrade - hindsight/2020, I'd stayed at 384.18 on my AC86U for (*@(@* sure.

My detail steps -> https://www.snbforums.com/threads/r...19-is-now-available.65801/page-24#post-615322

I am half way through some trouble shooting. I had a vpn client running and also spdmerlin doing an hourly speed test over the vpn. I stopped the vpn client and rebooted. Since then no log entries like the one you and I were seeing for the last few hours. I just started the vpn client and we'll see if the log entries pop back up at 12 past the hour when the speed test runs. I was also getting a new log entry that I never got with .18 regarding the vpn.

Sep 1 13:12:55 ovpn-client1[5123]: AEAD Decrypt error: bad packet ID (may be a replay): [ #3423020 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

If the log entries return I'm going to reset the vpn client to defaults, reboot and then reload the ovpn file.

 

[Centrifuge]

no AMTM apps. If you want to do filtering use a Pi-Hole.

I like my scripts, I think I'll avoid this hot mess and stick to .18. until someone can come up with a stable 386. If something breaks and I switch to something like your pi hole setup, it won't be with an Asus router TFDS.

 

[John Adler]

Dirty flash...
as usual, everything works fine so far !
thanks Merlin !

 

[Valid Name]

Wondering if anyone can help troubleshoot why the update to 384.19 would keep failing.

1. Currently on 384.17, RT-AC68U
2. Downloaded the 384.19 firmware, verified sha256 file hash
3. Connected via ethernet (powerline adapter) to the router web portal (192.168.1.1)
4. Restarted the Router then safely removed the USB disks (not physically disconnected though)
5. Uploaded the 384.19 update file through router web portal, the webpage just gets stuck on waiting and the router seems to get stuck, I’ve tried leaving it for more than 15 minutes.
6. Reconnect to web admin page of router, sluggish, the webpage still displays 384.17 on the top
7. Watchdog in Syslog reports an update firmware 384.19 is available

Any suggestions? Thanks in advance.

 

[gattaca]

^^^^ My sleeping brain thinks that the "format JFFS" may have hozed me up on my AC86U. Why? B/C when I did the first upgrade, I did NOT do the JFFS format. I followed @L&LD instructions and simply restored my JFFS backup as part of that flow. What didn't work then is the AMTM scripts didn't start but after I removed and readded them, things worked great for a week with no JFFS CRC errors.

Then this passed weekend, I decided that well, you didn't do the strongly advised JFFS format, you best "bite that bullet" and get it over with, so I did. I used the detailed process I posted earlier based on @L&LD guide only I added in the "format JFFS" steps. Shortly after that, is when JFFS CRC errors appeared and things rapidly deteriorated. Oh my, "if it ain't broke..."

I may have to "bite the bullet (again)" and resort to a factory reset but then again, CRC errors on flash mean the media (flash) is having I/O problems. This is no different than a HDD/SSD presenting CRC errors to an OS on a PC. There is no reliable way to fix this other then the OS or media to map around the flash bad cells and that does not appear to be happening with the JFFS formats. The CRCed data is essentially corrupted. Funny there's a warning here about hitting the flash too hard -> https://wiki.dd-rt.com/wiki/index.php/Journalling_Flash_File_System#Flash_Wear:_A_Warning

See also " In some cases, however, a lot of CRC errors may appear without power drops (i.e., if the JFFS2 file system is always correctly unmounted before power OFF). In this case, the CRC errors may indicate an I/O problem with the flash. Even though symptoms of the I/O problem appear as JFFS2 errors, there is nothing wrong with JFFS2. To further investigate this, simple tests using “dd” to write and read large data sets to and from the raw MTD device (e.g., /dev/mtd0) are recommended. Any verification errors you see during these tests are from the I/O problem, not from any problem with JFFS2."

From -> https://community.cypress.com/docs/DOC-10438

We also have seen quite a few people using USB keys (cheap nand flash) have reported problems on their keys in these very forums after lots of use. I myself recommended using a USB SATA SSD vs a USB Key for those reasons. Having worked in the computer industry for 40+ years, I'm no stranger to CRC errors and what they mean. If I'd not changed anything and stayed on (384.18) and the CRC errors started, I'd have a lot more confidence the media is failing. But having tossed 384.19 + AC86U "resizing by ASUS" + JFFS Format into the mix - that's another change. A coincidence or not - difficult to discern now - hmmm?

However, OTOH, with multiple AC86U owners now reporting a JFFS CRC issues - is it bad flash, bad upgrade steps (by formatting) or something else ASUS did in 384.19 (outta RMerlin's control) or all?

I can only mess with this router when everyone else is asleep, so I gotta forge another play to re-stabilize this AC86U.

FWIW, gut says 384.19 is fine on anything but the AC86U where ASUS "adjusted" the JFFS size. I have it working perfectly on an AC88U, and multiple AC68U/AC1900U units. No CRC errors but then I'm not running AMTM and beating the crap outta the flash on those either.. they are just WAPs and 1 is a router but without AMTM tooling beating on it.

YMMV. That's all I have to report for now. Oh, ntpMerlin ran all night (but pointed to the USB). no JFFS CRC errors, so I added back in connmon (pointed it to the USB) and will monitor today.

 

[Ramz]

Hello did the upgrade backed up my JFFS. Now how will I know if I have to format and reboot.? Seems everything is running well ATM.

 

[gattaca]

Hello did the upgrade backed up my JFFS. Now how will I know if I have to format and reboot.? Seems everything is running well ATM.

^^^^ LOL - based on several above postings, I'd watch the AC86U router's logs for a week or so. Let us know if your AMTM tooling is all working (assuming you have pointed to the JFFS) and if there are no CRC errors. I also saw my JFFS showed unable to write so I could not even create a simple directory after about 12 hours with AMTM J* scripts hamming on it. The AMTM tooling also stopped working / graphing b/c it could not write. If you see none of that and your AC86U's JFFS is 47M, personally, I would not touch it (ain't broke) using my recently gained 20/20 hindsight! YMMV. Peace. Stay safe, stay alive.

 

[kfahoo]

when vpn client disconnects from server e.g of network failure, it doesn't try to reconnect by itself, is it possible to add somekind of watchdog or reconnect feature?

 

[jak8]

Dirty upgrade from 348.13 to 348.19 on ac68u, everything seems okay even with such a large step. Backed up and restored Jffs and reloaded Diversions usb etc with no problem.

 

[Mutzli]

after update my AX88U to this version, my intel ax200 pcie card no internet connection when connect to 5ghz wifi. 2.4ghz no issues. I tried reinstall wifi driver, reset router, still no lucky. Anyone has the same problem?

Intel released a new driver again with additional improvements and fixes today.

 

[GHammer]

I understand that scripting is a large reason for running the Merlin builds.
Occasionally, the infrastructure changes. When it does, trying/hoping to not start with a factory reset and build up your configuration is foolish.
Keep records, update them. When faced with doing a clean build (no restores of anything), you just tediously go through the pages and enter what you had.
I do 'cheat' and have a backup of my static DHCP entries, but that is a list that is entered, not a file that is placed in jffs. Even those entries don't take long to input manually.
Seeing all the problems from those who don't do this reinforces my old school habits.

I have an 86U and see none of the problems that folks who ignore best practices have.
I also don't use my router as a NAS, an ad blocker, or even use QoS. It routes and provides radio to clients and does so wonderfully.

 

[gattaca]

^^^^ I agree we add the complexity to the router with the AMTM tooling. I fully understand the gotchas. We all know the best practices would be to factory reset and start over with each update is recommended repeatedly. @L&LD has a nice procedure on that route too!

a) I am unsure the FactoryReset/BP will save me at this point b/c the problem appears to be in flash media.
b) I cannot afford to spend 4+ hours every month or so tearing down a router and manually rebuilding it to stay current as I tend to update within 1 week of Merlin releases. Most of use prefer the dirty upgrade route and pray that works (and it does many times.)
c) My family cannot not tolerate such an outage except except from 1AM - 5AM. Everyone is WAH right now.
d) I've shown that the AMTM scripts hammering the JFFS seem to be causing the CRC errors on the AC86U. People not using those, indeed have a simpler setup with less room for gotchas - I agree 100%! When I turn off the J1 scripts hitting JFFS, no more CRC errors. But that does not mean the nand media is OK.. it just means we are not hitting that spot in the nand with I/O right now.

I am posting what I've run into with 384.19 + AMTM tooling b/c so many people look to these forums for guidance. I want folks to know this may be a gotcha. If my case, if it is really nand flash failing which is used by JFFS, then my problem is pHW driven. If it is not, then there's room for more caution with the AC86U and this ASUS JFFS resiszing + AMTM tooling hitting the JFFS hard. I've seen no CRC errors since I turned off the AMTM tooling hitting the JFFS hard. But that does not mean the flash media is not having wear / corruption issues from high usage. Remember, this is "cheap" nand and it typically no better than what's used in USB keys. Thanks!

 

[Smokey613]

I have removed all scripts except those listed in my signature. I have turned off DNS filtering and I pass Google DNS server IPs directly to my clients via DHCP settings, thus removing the router as a DNS resolver for clients. Skynet is the only filtering being used. My network is now much more responsive and faster, decidedly so. I plan to eventually install a dedicated mini pc to handle all routing, filtering, DNS, etc. functions and use my routers in an AP/AiMesh setup.

 

[jsbeddow]

^^^^ I agree we add the complexity to the router with the AMTM tooling. I fully understand the gotchas. We all know the best practices would be to factory reset and start over with each update is recommended repeatedly. @L&LD has a nice procedure on that route too!

a) I am unsure the FactoryReset/BP will save me at this point b/c the problem appears to be in flash media.
b) I cannot afford to spend 4+ hours every month or so tearing down a router and manually rebuilding it to stay current as I tend to update within 1 week of Merlin releases. Most of use prefer the dirty upgrade route and pray that works (and it does many times.)
c) My family cannot not tolerate such an outage except except from 1AM - 5AM. Everyone is WAH right now.
d) I've shown that the AMTM scripts hammering the JFFS seem to be causing the CRC errors on the AC86U. People not using those, indeed have a simpler setup with less room for gotchas - I agree 100%! When I turn off the J1 scripts hitting JFFS, no more CRC errors. But that does not mean the nand media is OK.. it just means we are not hitting that spot in the nand with I/O right now.

I am posting what I've run into with 384.19 + AMTM tooling b/c so many people look to these forums for guidance. I want folks to know this may be a gotcha. If my case, if it is really nand flash failing which is used by JFFS, then my problem is pHW driven. If it is not, then there's room for more caution with the AC86U and this ASUS JFFS resiszing + AMTM tooling hitting the JFFS hard. I've seen no CRC errors since I turned off the AMTM tooling hitting the JFFS hard. But that does not mean the media is not having issues.

Thanks!

I fully understand the difficulty of doing a full reset/rebuild from factory default settings currently, as I am in the same situation with only a handful of ugly/narrow windows of time to get that done with everyone in the house relying on the internet. I too am contemplating doing so though, as I do have some minor instability on 384.19 (RT-AC86U as well): mine will be fine for 5-7 days at a time (and never any CRC errors in logs), but it eventually gets to a point where some select wifi clients get dropped from the network. Sometimes they show as connected, sometimes not, but they get to a point where they lose internet connectivity after these 5-7 days. Never a problem before on previous releases, only now on the 384.19 series (alphas, beta and release).
Following closely your situation, and hoping you find time to test out whether a factory reset will cure your issue, it might nudge me in that direction as well. Last time I needed to do the same was around 384.13 or 384.14, but haven't seen the need to since.

Yes, BTW, I am also aware of the dying radio issue with the 86U, and am hoping what I am seeing is not symptomatic of that problem.

Care to go first, so we can learn from your experience?

 

[doczenith1]

I am half way through some trouble shooting. I had a vpn client running and also spdmerlin doing an hourly speed test over the vpn. I stopped the vpn client and rebooted. Since then no log entries like the one you and I were seeing for the last few hours. I just started the vpn client and we'll see if the log entries pop back up at 12 past the hour when the speed test runs. I was also getting a new log entry that I never got with .18 regarding the vpn.

Sep 1 13:12:55 ovpn-client1[5123]: AEAD Decrypt error: bad packet ID (may be a replay): [ #3423020 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

If the log entries return I'm going to reset the vpn client to defaults, reboot and then reload the ovpn file.

Well I have no idea what was causing my issues but they are gone! The vpn client is up and spdmerlin is running with no errors in the log. My 86U is functioning just fine now.

 

[Mario64]

Forgive me if this is a noob question but Merlin FW is currently at 384.19 while Asus is 385.x. Does this mean that Merlin‘s doesn’t contain the security fixes in the newer official FW? Or does he back-port those into his releases? Thanks

 

[SomeWhereOverTheRainBow]

Forgive me if this is a noob question but Merlin FW is currently at 384.19 while Asus is 385.x. Does this mean that Merlin‘s doesn’t contain the security fixes in the newer official FW? Or does he back-port those into his releases? Thanks

some bits of code from 385 were just 384 repackaged with the name 385.... the only code branch that is truely different is 386. 384 and 385 share same fixes with just different names on the package. @RMerlin decided against changing the name on his package since the code bases virtually identical. However he is changing for 386 because the branch code is changing.

"edit"
For those who are curious just follow RMerlins github commits.

Bumped revision to 386.1 · RMerl/asuswrt-merlin.ng@ad3da4f

Third party firmware for Asus routers (newer codebase) - Bumped revision to 386.1 · RMerl/asuswrt-merlin.ng@ad3da4f

github.com

I have "YET" to see @RMerlin not include a security fix once it has become available to him. Naming is just naming.

 

[maghuro]

Hello guys,
Currently I'm having an issue that started from its own.
I'm getting this log message every two seconds:

Sep 3 07:27:37 kernel: grep[10776]: undefined instruction: pc=0000007f9ee0fd50

Sep 3 07:27:37 kernel: Code: 350000a0 91002021 910004c0 f907d020 (565f03c0)

And when I open amtm, for example, it's going like that (attached). I can't run properly any script. Any clues?

 

[kernol]

I fully understand the difficulty of doing a full reset/rebuild from factory default settings currently, as I am in the same situation with only a handful of ugly/narrow windows of time to get that done with everyone in the house relying on the internet. I too am contemplating doing so though, as I do have some minor instability on 384.19 (RT-AC86U as well): mine will be fine for 5-7 days at a time (and never any CRC errors in logs), but it eventually gets to a point where some select wifi clients get dropped from the network. Sometimes they show as connected, sometimes not, but they get to a point where they lose internet connectivity after these 5-7 days. Never a problem before on previous releases, only now on the 384.19 series (alphas, beta and release).
Following closely your situation, and hoping you find time to test out whether a factory reset will cure your issue, it might nudge me in that direction as well. Last time I needed to do the same was around 384.13 or 384.14, but haven't seen the need to since.

Yes, BTW, I am also aware of the dying radio issue with the 86U, and am hoping what I am seeing is not symptomatic of that problem.

Care to go first, so we can learn from your experience?

My RT-AC86U is rock steady and entirely stable with all add-ons and Aimesh as per my signature ... however ... ... before I started the upgrade I had read RMerlin's readme about the JFFS partition having been truncated by Asus [chopping off 1 Mb ] ... so I resigned myself to the idea that a full factory reset [nuclear in fact] was the only fully reliable way to get the job done.

I too have been "spoilt" by successful "Dirty" upgrades for many versions before - so felt I really could not complain about having to do some heavy lifting for a change if I wanted the kind of stability I have enjoyed for so long already. I took precautionary backups of everything [settings / jffs / usb cloned etc] just in case - but had no plans to use them to restore anything ... preferring to rebuild everything from scratch.

I won't go through all the detail of the steps taken - @L&LD has that covered in his excellent guides - but my kick-off point was in fact to eject USB [and remove it], turn off JFFS script usage and format the JFFS partition on restart. Then following the restart I nuked everything by putting the RT-AC86U into Rescue Mode and installing Asus Stock code Version 3.0.0.4.384.81992. Only after doing minimal setup on that stock firmware did I manually upload the RMerlin 384.19 firmware and flash it.

Did pretty much the same with my AC5300 by resetting to factory - then Rescue mode with stock firmware - minimal setup - loaded its 384.19 Merlinware version and added it back as an Aimesh Node [slick and simple].

My entire setup feels "brand new" and runs like a dream. Very noticeable drop in the amount of space used in the JFFS partition after the above process - even though all my old add-ons reinstalled from scratch and settings put back manually - plus a good few more manual ip assignments. Even NVRAM consumption lower than it was. Must assume all those dirty upgrades left bits and bobs behind.

Highly recommend putting aside the few hours of dedicated time to rebuild from scratch - and get the family in line ... after all without YOU there would be ZERO ZILTCH NADA safe internet in the home !

 

[gattaca]

^^^^ Yes, we've been spoiled badly with many successful "dirty upgrades" over many Merlin releases! This resizing JFFS curveball ASUS tossed in for the AC86U may have been the straw, for some of us AC86U owners.

Over the last 3 days, have re-installed all of the J* AMTM tooling, one per day, but pointed each one where it allows in the menu, to use the USB instead of the JFFS. My USB is a SATA SSD on a uGREEN USB2SATA carrier which I've posted about before. The USB is connected to the USB3.0 port on the router. As of this AM, I am seeing no CRC errors on the JFFS at this point - nada, 0 - none since I unistalled and moved those scripts.

I think a router probably needs a full factory reset for sure. IDK about L&LD nuclear option but that may be best.

FWIW, I also found at least 2 posts from @L&LD stating when when formatting the JFFS, you need to reboot the router 3 times in the next 15 minutes (with no USB drive installed)" to allow the router time to properly format the JFFS! I did not reboot 3 times in 15 minutes, only once after I applied the JFFS format. That may be related to the JFFS CRC errors - IDK at this point - makes sense if I started using the JFFS before it was actually fully formatted... but I do recall df -h on the JFFS and seeing files still there - thinking strange... format should have wiped everything. So I am considering formatting the JFFS and rebooting 3x in 15 minutes, then putting the J* utilies back just to see. That would be very, very valuable to know.

Does anyone know if after you "format the JFFS successfully" if there are any files or default directories created by the router?

I'm trying to figure out how to best use the nvram save and restore utility to CMA. I located the thread and have been reading it. Having nvram working would make me less cautious that I'm going to botch a full reset - at least I'd have all the settings which could be restored as I've tweaked many settings over the longer haul. I saw a script that keeps like 14 days of nvram backups, that looks useful. I also need to fully backup the USB drive - just in case.

So there's some prep needed either using nvram or screen capping critical page.
I want to collect L&LD steps so I can be sure I don't miss something like "reboot the router 3 times after...."

Right now the router appears to be humming along with no JFFS CRC errors + all the AMTM J* scripts running as prior - as long as they are on USB and not JFFS.