Release Asuswrt-Merlin 384.19 is now available

[RMerlin]

I agree with most. Either ASUS has something coming with new features or they are fixing something they just found wrong in the AC86U or both.

They just resized it so they can reserve 1 MB of flash space to store crash logs. Nothing more.

 

[wuckawucka]

Hi all, first my newbie credentials:
- First post here
- First installed Merlin on my RT-AC86U last week
- Basic level knowledge of router features and networking

I have installed 384.19 on the 86U. I did a factory reset, and have formatted JFFS.

What I'm hoping to get help with is this. I want to get detailed daily info on my data usage, as I have just moved to a provider that only gives very basic stats. In Traffic Monitor, I seem to be getting incorrect (very low) data showing on "Internet Connection (WAN)" - in the order of MB per day. Reality is that I'm using GB per day. The "Wireless (5GHz)" looks much better - gigabytes - and this is how most of my devices are connected, 5GHz. The traffic is mainly external, not between devices on my network.

Any clues as to why WAN data usage is not showing up? Things I can try?

I've done a bit of searching here and elsewhere. I have tried disabling NAT acceleration - on the 86U this seems to be done indirectly by enabling QoS. I've tried Adaptive and Bandwidth Limiter. One of these has made "runner" disabled under Tools, but "Flow Cache" is still enabled. Not sure what either of these really does.

My provider uses a VLAN ID so I've set that under IPTV if that's relevant.

Not sure if this thread is the right place, but anyway, any help appreciated.

 

[JGrana]

Strange, but you can also download the administration version and manually install it through the device manager.

Sorry, I should have reposted. I tried the next day and the link worked. Must have been a temporary issue.
Update went well, thanks for the link!

 

[Smokey613]

They just resized it so they can reserve 1 MB of flash space to store crash logs. Nothing more.

What does that say about the stability of the RT-AC86U router when you have to increase storage space for crash logs.

 

[Jeffrey Young]

Hi all, first my newbie credentials:
- First post here
- First installed Merlin on my RT-AC86U last week
- Basic level knowledge of router features and networking

I have installed 384.19 on the 86U. I did a factory reset, and have formatted JFFS.

What I'm hoping to get help with is this. I want to get detailed daily info on my data usage, as I have just moved to a provider that only gives very basic stats. In Traffic Monitor, I seem to be getting incorrect (very low) data showing on "Internet Connection (WAN)" - in the order of MB per day. Reality is that I'm using GB per day. The "Wireless (5GHz)" looks much better - gigabytes - and this is how most of my devices are connected, 5GHz. The traffic is mainly external, not between devices on my network.

Any clues as to why WAN data usage is not showing up? Things I can try?

I've done a bit of searching here and elsewhere. I have tried disabling NAT acceleration - on the 86U this seems to be done indirectly by enabling QoS. I've tried Adaptive and Bandwidth Limiter. One of these has made "runner" disabled under Tools, but "Flow Cache" is still enabled. Not sure what either of these really does.

My provider uses a VLAN ID so I've set that under IPTV if that's relevant.

Not sure if this thread is the right place, but anyway, any help appreciated.

The RSTAT binary has trouble dealing with properly counting vlan tagged internet traffic. Another user, not long ago, had the same issue. His traffic stats would not count anything with vlan tagged traffic on the internet side (WAN). In that case, we set up a test setup with a downstream Merlin Router attached to the internet gateway router (another Merlin router). With that set up, once the vlan tagged traffic was removed, RSTAT recorded the traffic properly (on the downstream router).

 

[ATLga]

They just resized it so they can reserve 1 MB of flash space to store crash logs. Nothing more.

Turning off and dialing down the damned logs is what most people want to do, not increase them further

 

[wuckawucka]

The RSTAT binary has trouble dealing with properly counting vlan tagged internet traffic. Another user, not long ago, had the same issue. His traffic stats would not count anything with vlan tagged traffic on the internet side (WAN). In that case, we set up a test setup with a downstream Merlin Router attached to the internet gateway router (another Merlin router). With that set up, once the vlan tagged traffic was removed, RSTAT recorded the traffic properly (on the downstream router).

Interesting, thanks. I assume RSTAT deals with counting for traffic monitor.

I have been in touch with another user with an 86U using the same provider (so also VLAN tagged), and they have this working apparently.

Wonder if I have any options - other than a second router just for counting traffic.

 

[gattaca]

I'm posting the results of my format JFFS + 3 reboots 10+ minutes apart. This is the 4th time the JFFS has been formatted in the past 2 weeks since my 384.19 + ASUS JFFS resize journey nightmare FUBAR started.

1. I've had the J* scripts using the USB for a week +, with no CRC / JFFS errors being reported.
2. This AM I unmount, then remove the USB, the, backup the JFFS (without that large file earlier), then select "Format JFFS" + APPLY, then reboot.
3. Sit 10+ minutes, reboot
4. Sit 10+ minutes, reboot
5. Sit 10+ minutes, reboot
6. Sit 10 minutes, reconnect the USB, restore the JFFS, reboot.
7. Verified no CRC or GC errors being reported in router's logs.
8. At 1007 I moved connmon from USB back to JFFS with connmon AMTM GUI toggle
9. At 1013, the CRC errors / No space for GC errors returned.

Sep  7 10:11:00 spdMerlin: Starting speedtest using auto-selected server for WAN interface
Sep  7 10:11:02 kernel: jffs2: Data CRC dc07b7ed != calculated CRC 34d07a98 for node at 02308c38
..
Sep  7 10:13:53 kernel: jffs2: Data CRC dc07b7ed != calculated CRC 34d07a98 for node at 02308c38
Sep  7 10:13:53 kernel: jffs2: read_cache_page() returned error: -5
Sep  7 10:13:53 kernel: jffs2: Error garbage collecting node at 02308c38!
Sep  7 10:13:53 kernel: jffs2: No space for garbage collection. Aborting GC thread
Sep  7 10:13:55 kernel: jffs2: Data CRC dc07b7ed != calculated CRC 34d07a98 for node at 02308c38
Sep  7 10:13:55 kernel: jffs2: read_cache_page() returned error: -5
Sep  7 10:13:55 kernel: jffs2: Error garbage collecting node at 02308c38!
....
Moved code back to USB, rebooted the router. The CRC errors ceased but these 3 bad blocks remain in the NAND.
...
May  5 01:05:11 kernel: nand: Macronix NAND 256MiB 3,3V 8-bit
May  5 01:05:11 kernel: nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
May  5 01:05:11 kernel: bcm63xx_nand ff801800.nand: Adjust timing_1 to 0x65324458 timing_2 to 0x80040e54
May  5 01:05:11 kernel: bcm63xx_nand ff801800.nand: detected 256MiB total, 128KiB blocks, 2KiB pages, 16B OOB, 8-bit, BCH-4
May  5 01:05:11 kernel: Bad block table found at page 131008, version 0x01
May  5 01:05:11 kernel: Bad block table found at page 130944, version 0x01
May  5 01:05:11 kernel: nand_read_bbt: bad block at 0x0000032e0000
May  5 01:05:11 kernel: nand_read_bbt: bad block at 0x000007b00000
May  5 01:05:11 kernel: nand_read_bbt: bad block at 0x00000f880000
May  5 01:05:11 kernel: >>>>> For primary mtd partition rootfs, cferam/vmlinux.lz mounted as JFFS2, vmlinux fs mounted as UBIFS <<<<<
May  5 01:05:11 kernel: Secondary mtd partition rootfs_update detected as JFFS2 for cferam/vmlinux source and UBIFS for vmlinux filesystem
May  5 01:05:11 kernel: setup_mtd_parts: misc indx 2 name misc3 nvram configured size 1
May  5 01:05:11 kernel: setup_mtd_parts: name misc3 configured size 0x00100000 offset 0xF600000
May  5 01:05:11 kernel: setup_mtd_parts: misc indx 1 name misc2 nvram configured size 47
May  5 01:05:11 kernel: setup_mtd_parts: name misc2 configured size 0x02f00000 offset 0xC700000
May  5 01:05:11 kernel: setup_mtd_parts: misc indx 0 name misc1 nvram configured size 8
May  5 01:05:11 kernel: setup_mtd_parts: name misc1 configured size 0x00800000 offset 0xBF00000
May  5 01:05:11 kernel: Creating 11 MTD partitions on "brcmnand.0":
....
No GC or CRC errors reported since.

df -hT
Filesystem           Type            Size      Used Available Use% Mounted on
ubi:rootfs_ubifs     ubifs          77.2M     60.6M     16.6M  79% /
devtmpfs             devtmpfs      214.9M         0    214.9M   0% /dev
tmpfs                tmpfs         215.0M    196.0K    214.8M   0% /var
tmpfs                tmpfs         215.0M    988.0K    214.1M   0% /tmp/mnt
mtd:bootfs           jffs2           4.4M      3.3M      1.1M  75% /bootfs
tmpfs                tmpfs         215.0M    988.0K    214.1M   0% /tmp/mnt
mtd:data             jffs2           8.0M    736.0K      7.3M   9% /data
tmpfs                tmpfs         215.0M    988.0K    214.1M   0% /tmp
/dev/mtdblock9       jffs2          47.0M      6.5M     40.5M  14% /jffs
/dev/sda             ext4           58.4G      3.0G     52.5G   5% /tmp/mnt/1234-ASUS-AC86U
tmpfs                tmpfs         215.0M    988.0K    214.1M   0% /www/index_style.css
tmpfs                tmpfs         215.0M    988.0K    214.1M   0% /www/require/modules/menuTree.js

Team A86U/Merlin/384.19 - my final testing results are repeatable. AC86U owners, if you are getting this CRC error, then moving your J* scripts to USB may solve the gotcha for now by using the AMTM menu option in each script. Again, to be clear, all J* scripts on this AC86U were using the JFFS BEFORE ASUS decided it was a great idea to "resize from 48M to 47M"in 384.19. The setup had been running for 1+ years with no CRC or JFFS GC errors reported b/c I check the logs to see if there are issues before any upgrade. My gut says, my AC86U needs a @L&LD nuclear reset and then still, I'm doubting it will fix/repair/ or bypass the bad NAND blocks. I think this router is going to end up being a WAP - somewhere safe where it's not beating up the JFFS.

Game over on 384.19 + AC86U. I am sorry I do not have better news to report. For AC86U owners who have had zero issues with 384.19 + JFFS + AMTM J* scripts on JFFS - be happy!

My gut also says hitting the JFFS hard with J* scripts may not be in the best long term health of the AC86U's NAND even if it is SLC and even if it should "outlive me by design." I'm going to permanently move the J* scripts to the USB SSD and see what happens there going forward with a replacement router - AC86U or not.

When I go nuclear on this AC86U, I'll post if it has any impact. Any other suggestions, feel free to share! I might even wipe it down to 384.18 just to see..

I'll also be implementing the blue/green update approach I mentioned earlier. I've burned way too many hours on this issue which I feel ASUS's little JFFS resizing may have been a root cause of or for sure a contributing factor - sure looks that way from my experiences.

Stay safe, stay alive. Peace.

 

[Jeffrey Young]

Interesting, thanks. I assume RSTAT deals with counting for traffic monitor.

I have been in touch with another user with an 86U using the same provider (so also VLAN tagged), and they have this working apparently.

Wonder if I have any options - other than a second router just for counting traffic.

Considering you have a friend with the same setup on the same IP and their RSTAT is working and counting properly - or at least realisticlly, I would first look at some basic troubleshooting;

Have you tried resetting the file (Tools -> Other Settings -> Create or Reset Data Files)?

Where are you storing the data file now? The file location should be on a USB drive or your jffs partition (custom location). I have mine set to /jffs/ as I have had issues in the past with the USB drive becoming unmounted. I use custom location as opposed to NVRAM only so I know where to find the file. I have a cron job that backs up the file to my NAS twice a day and and take a daily snap shot each midnight for history purposes (RSTAT keeps the last 62 days of daily of stats and 24 months of monthly stats).

This feature is why I switch to Merlin - since then, it has kinda exploded in what I have my router do (particularly in using SoftEther VPN).


As this is a bit off topic, if you want to trouble shoot further, we can create a separate thread.

 

[DHLarson]

OK - so I'm not seeing CRC errors but I am seeing repeating GC errors on JFFS in the log. Running VPNMGR and Skynet on 384.19 on AC-86U with 2 AI Mesh AC-68U nodes running same. No previous issues on x.17. Tried backing up JFFS, formatting, then restoring with the issue returning. Did not do 3 10 minute apart resets after any of these steps. Suggestions? Do I downgrade to x.18 or x.17? There seems to be some real JFFS code issues out there.

Sep  7 08:20:37 kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
Sep  7 08:20:37 kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28
Sep  7 08:20:37 kernel: jffs2: Error garbage collecting node at 010c0000!

 

[Cranium]

Running 384.19 for FOUR days with about a dozen devices and no issues that I notice.

 

[anotherengineer]

Another thing I have noticed with this firmware is some random LAN disconnects. Well possibly, I also updated the old motorola surboard to a Hitron Coda 4582 (bridge mode) about the same time.

 

[gattaca]

Nope.. you are not seeing wrong.. I've been seeing these since 384.19... but I didn't want to add more "issues" to this already hot thread. I've seen these happening at very odd times during the day. I know the modem is not dropping this much....

Sep 7 02:35:26 kernel: eth0 (Int switch port: 3) (Logical Port: 3) Link DOWN.
Sep 7 02:35:30 WAN_Connection: ISP's DHCP did not function properly.
Sep 7 02:35:30 DualWAN: skip single wan wan_led_control - WANRED off
Sep 7 02:35:30 nat: apply redirect rules
Sep 7 02:35:41 WAN_Connection: Ethernet link down.
Sep 7 02:35:43 kernel: eth0 (Int switch port: 3) (Logical Port: 3) Link UP 1000 mbps full duplex
Sep 7 02:35:46 WAN_Connection: Ethernet link up.
Sep 7 02:35:46 rc_service: wanduck 1189:notify_rc restart_wan_if 0
...
Sep 7 02:35:51 wan: finish adding multi routes
Sep 7 02:35:51 WAN_Connection: WAN was restored.

 

[MaziahBebop]

Tried backing up JFFS, formatting, then restoring with the issue returning.

If the problem appears to be on the JFFS, and you're backing up the JFFS when its known to be in a problematic state, would it make sense to not restore from said backup?
Been curious about this for a long time, so decided to finally ask and find out.

 

[gattaca]

^^^ Not quite. When I (and others) take the JFFS backup, the logs are NOT reporting any JFFS/CRC/GC errors. When I restore the JFFS backup and run without the J* scripts hitting the JFFS, no errors are reported and all appears GTG.

Also, the JFFS backup is a simple tar file which is untared to the freshly formatted JFFS... so it's restoring each file individually - it's not like JFFS restore is trying to a sector copy. The problem is these supposedly bad NAND sectors which appeared on my AC86U only after the "reformat" by ASUS. My first attempt on JFFS was with a 384.18 JFFS backup which was showing no CRC / GC issues at all with all J* scripts hitting JFFS then. At this point, I'm of the mindset this JFFS resizing by ASUS in 384.19 on AC86U hozed something up or surfaced something for the AC86U.... Other's have reported no such issues - so YMMV. I think for the *@(*@* of it, when I nuke this AC86U, I'm going to reinstall 384.18 just to see... I really think this resize left insufficient "free blocks" in the NAND to perform the GC.. and the only way to fix that is @L&LD nukem procedure. Remember flash NAND has to have whole free areas to copy the "still good" data to and it must do this a page at the time to perform GC. Quite a different approach than old HDD. This is also why if you run SSD/NVMes hard and never "Trim" them, the unit's GC ends up juggling gobs of "trash" literally that the OS no longer needs. It's a real performance gotcha that no one want's to discuss - industry-wide - trust me.

 

[RMerlin]

What does that say about the stability of the RT-AC86U router when you have to increase storage space for crash logs.

They didn't increase it, they allocated it. There wasn't any before.

Turning off and dialing down the damned logs is what most people want to do, not increase them further

Crash log have nothing to do with the system log. It's a location to store the content of a kernel panic, so it can survive reboots. They have similar mechanisms on most of their model, just that for some reason it's implemented differently on this particular model.

 

[RMerlin]

No free space left for GC.

There's your problem.

 

[gattaca]

There's your problem.

Yeap - and this burns a lot of folks - remember the computer industry does not like to talk about the SSD / GC issues with TRIM. Linux by default does NOT enable TRIM and that's b/c the Linux TRIM code sux.

This is a Flash NAND Garbage Collection issue a/or bad blocks interfering with GC issue a/or something ASUS did on this reallocation or all of the above.. The OS thinks it has plenty of space - the GC on the NAND controller does not have enough free space to perform its GC chores - 2 different animals.

There's plenty of free space from the linux's PoV.

df -hT
Filesystem Type Size Used Available Use% Mounted on
ubi:rootfs_ubifs ubifs 77.2M 60.6M 16.6M 79% /
devtmpfs devtmpfs 214.9M 0 214.9M 0% /dev
tmpfs tmpfs 215.0M 196.0K 214.8M 0% /var
tmpfs tmpfs 215.0M 988.0K 214.1M 0% /tmp/mnt
mtd:bootfs jffs2 4.4M 3.3M 1.1M 75% /bootfs
tmpfs tmpfs 215.0M 988.0K 214.1M 0% /tmp/mnt
mtd:data jffs2 8.0M 736.0K 7.3M 9% /data
tmpfs tmpfs 215.0M 988.0K 214.1M 0% /tmp
/dev/mtdblock9 jffs2 47.0M 6.5M 40.5M 14% /jffs
/dev/sda ext4 58.4G 3.0G 52.5G 5% /tmp/mnt/1234-ASUS-AC86U
tmpfs tmpfs 215.0M 988.0K 214.1M 0% /www/index_style.css
tmpfs tmpfs 215.0M 988.0K 214.1M 0% /www/require/modules/menuTree.js

 

[MaziahBebop]

No free space left for GC

There's your problem.

@DHLarson I think this could be related to a previous thought I posted:

If the problem appears to be on the JFFS, and you're backing up the JFFS when its known to be in a problematic state, would it make sense to not restore from said backup?

Possible there's too much data on the restored backup, or a script in there that's filling jffs up too quick.

 

[MaziahBebop]

@gattaca can you make a pastebin of

ls -Rla /jffs

On your router's internal JFFS, at the time when its giving you CRC errors (not your external USB JFFS).