Release Asuswrt-Merlin 386.12 is now available for AC models

[heywire]

I also had issue with 386.12 on my AC86U, more than just 2-3 a day for me. Whenever I'm watching Twitch or I think it happens to some other video streaming services too, I got frequent buffers. Tried to rollback to 386.11, didn't face such issue. I only use Cake-QoS Merlin, no other add-on.

Do you have a reproducible example that can be used to verify whether this is a problem for all .12 CakeQOS users? If not, please flash back to .12 and try to find one. Because if we don't get this, it will likely not be fixed in .13 ....+


And to other readers of this post:
Can anybody else please confirm or dismiss whether they've had the same experiences with .12 and CakeQOS enabled with respect to what's being described by Sicario above^? And if you do experience it, please try and produce an example that other users in here can verify.

 

[heywire]

UPDATE:
Came to think of it, I actually have had some problems with streaming services being occationally laggy/stuck after a while lately. Now I'm on a new client, and the problem persists. Did a stress test by scrolling through TikTok for a few minutes, and low and behold, it seemed to reliably stop playing certain videos after a few minutes scrolling back and forth, occationally going in and out of some videos. The videos that wouldn't play, wouldn't even react to me clicking the play button (when I scroll, they autoplay as they come into focus on the screen, but they have a play/pause button).

Flashed back to .11, and the problem went away.

Can some other .12 & CakeQOS users try browsing through TikTok for a few minutes, and see if you experience any frozen videos that won't autoplay?
(So far the best stress test I can think of)
If yes, you can try and flash back to .11 and see if it helps. May be a CakeQOS issue.

Does this reproduce the issue for you, @Sicario ?

 

[heysoundude]

UPDATE:
Came to think of it, I actually have had some problems with streaming services being occationally laggy/stuck after a while lately. Now I'm on a new client, and the problem persists. Did a stress test by scrolling through TikTok for a few minutes, and low and behold, it seemed to reliably stop playing certain videos after a few minutes scrolling back and forth, occationally going in and out of some videos. The videos that wouldn't play, wouldn't even react to me clicking the play button (when I scroll, they autoplay as they come into focus on the screen, but they have a play/pause button).

Flashed back to .11, and the problem went away.

Can some other .12 & CakeQOS users try browsing through TikTok for a few minutes, and see if you experience any frozen videos that won't autoplay?
(So far the best stress test I can think of)
If yes, you can try and flash back to .11 and see if it helps. May be a CakeQOS issue.

Does this reproduce the issue for you, @Sicario ?

I am not experiencing this with my AC86.
And I am using CakeQoS. (for those of you with triple digit download speeds, I'm still making my way on the interwebz with a 50/10Mbps package from my ISP...)
Are all your Entware packages up to date? check amtm and the scripts/entware - there may be some surprises if you don't regularly check (the pixelserv update seems to have been a fairly big deal for my Diversion config...)

 

[heywire]

I am not experiencing this with my AC86.
And I am using CakeQoS. (for those of you with triple digit download speeds, I'm still making my way on the interwebz with a 50/10Mbps package from my ISP...)
Are all your Entware packages up to date? check amtm and the scripts/entware - there may be some surprises if you don't regularly check (the pixelserv update seems to have been a fairly big deal for my Diversion config...)

Did you do the TikTok test?

I'm not using any other scripts than the bundled Merlin-CakeQOS.

I do have a 150/150Mbps fibre optic connection, tho. Maybe it acts differently on copper due to higher latency.

 

[heysoundude]

Did you do the TikTok test?

I'm not using any other scripts than the bundled Merlin-CakeQOS.

I do have a 150/150Mbps fibre optic connection, tho. Maybe it acts differently on copper due to higher latency.

No. I avoid TikTok like the plague.
You should too

 

[heywire]

No. I avoid TikTok like the plague.
You should too

Well, if we keep it technical, what we need is a site that loads data in short intervals.

I guess skipping around a lot, quickly, in any video-streaming site would reproduce the same use-case scenario as Tiktok (I don't think google is a good example because it didn't affect Shorts, even if it's equivalent to Tiktok. And probably then Youtube isn't the best either. Maybe because youtube has very good optimizations and bandwith).

Especially if you have a low latency connection like fibre, it would "zap" the CakeQOS with a lot of request that cause it to stumble in the case of .12 for me and @Sicario .

 

[dave14305]

it would "zap" the CakeQOS with a lot of request that cause it to stumble

Capture the output of these commands immediately before and immediately after your tests on 386.12. If you have time, repeat on 386.11.

source /etc/cake-qos.conf
tc -s qdisc show dev $MIF
tc -s qdisc show dev $ULIF root

 

[heywire]

Capture the output of these commands immediately before and immediately after your tests on 386.12. If you have time, repeat on 386.11.

source /etc/cake-qos.conf
tc -s qdisc show dev $MIF
tc -s qdisc show dev $ULIF root

I ran the test on 386.11.

Then I flashed to 386.12 and ran the test immediately after.
However, I did not experience the buffering issues then. Maybe because it takes time for the problem to accumulate? Or perhaps they have come and gone just like the client list issues. Staying on 386.12 for now to see if the problems pop up again.

Anyways, here is the output of those commands:

EDIT:
Forgot to take the tests before, I only took them after.
By the way, how fast do I have to execute the commands after doing the stress test?

Spoiler: 386.11

admin@RT-AC86U-xxxx:/tmp/home/root# source /etc/cake-qos.conf
admin@RT-AC86U-xxxx:/tmp/home/root# tc -s qdisc show dev $MIF
qdisc cake 8002: root refcnt 2 bandwidth 153600Kbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 0
 Sent 104895884146 bytes 81353954 pkt (dropped 20743, overlimits 66341483 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 3241664b of 7500Kb
 capacity estimate: 153600Kbit
 min/max network layer size:           42 /    1500
 min/max overhead-adjusted size:       42 /    1500
 average network hdr offset:           14

                  Tin 0
  thresh     153600Kbit
  target            5ms
  interval        100ms
  pk_delay       9.15ms
  av_delay        900us
  sp_delay          2us
  backlog            0b
  pkts         81374697
  bytes    104924692627
  way_inds      4270969
  way_miss       639989
  way_cols          734
  drops           20743
  marks           21120
  ack_drop            0
  sp_flows            4
  bk_flows            1
  un_flows            0
  max_len          1514
  quantum          1514

admin@RT-AC86U-xxxx:/tmp/home/root# tc -s qdisc show dev $ULIF root
qdisc cake 8001: root refcnt 2 bandwidth 153600Kbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 0
 Sent 8855793158 bytes 26530697 pkt (dropped 10974, overlimits 5122742 requeues 7)
 backlog 0b 0p requeues 7
 memory used: 2737920b of 7500Kb
 capacity estimate: 153600Kbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       28 /    1500
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh       9600Kbit   153600Kbit    38400Kbit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us       1.55ms        221us
  av_delay          0us         47us         17us
  sp_delay          0us          2us          1us
  backlog            0b           0b           0b
  pkts                0     26508645        33026
  bytes               0   8862688923      4595265
  way_inds            0      1213998            0
  way_miss            0       643053          368
  way_cols            0         2185            0
  drops               0        10973            1
  marks               0          334            0
  ack_drop            0            0            0
  sp_flows            0            4            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0         1514          590
  quantum           300         1514         1171

Spoiler: 386.12

admin@RT-AC86U-xxxx:/tmp/home/root# source /etc/cake-qos.conf
admin@RT-AC86U-xxxx:/tmp/home/root# tc -s qdisc show dev $MIF
qdisc cake 8002: root refcnt 2 bandwidth 153600Kbit besteffort dual-dsthost nat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 0
 Sent 434945874 bytes 312664 pkt (dropped 74, overlimits 163851 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 822Kb of 7500Kb
 capacity estimate: 153600Kbit
 min/max network layer size:           42 /    1500
 min/max overhead-adjusted size:       42 /    1500
 average network hdr offset:           14

                  Tin 0
  thresh     153600Kbit
  target            5ms
  interval        100ms
  pk_delay        8.7ms
  av_delay        504us
  sp_delay          1us
  backlog            0b
  pkts           312738
  bytes       435040139
  way_inds          414
  way_miss         2036
  way_cols            0
  drops              74
  marks               3
  ack_drop            0
  sp_flows            3
  bk_flows            1
  un_flows            0
  max_len          1514
  quantum          1514

admin@RT-AC86U-xxxx:/tmp/home/root# tc -s qdisc show dev $ULIF root
qdisc cake 8001: root refcnt 2 bandwidth 153600Kbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 0
 Sent 15091002 bytes 116323 pkt (dropped 8, overlimits 4802 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 34432b of 7500Kb
 capacity estimate: 153600Kbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       28 /    1500
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh       9600Kbit   153600Kbit    38400Kbit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us       2.48ms       1.38ms
  av_delay          0us         64us         22us
  sp_delay          0us          1us          0us
  backlog            0b           0b           0b
  pkts                0       116034          297
  bytes               0     15053820        43846
  way_inds            0          341            0
  way_miss            0         1710           15
  way_cols            0            0            0
  drops               0            8            0
  marks               0            2            0
  ack_drop            0            0            0
  sp_flows            0            1            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0         1514          590
  quantum           300         1514         1171

 

[dave14305]

Maybe because it takes time for the problem to accumulate?

Or the problem is out in the Internet somewhere.

Forgot to take the tests before, I only took them after.

The “after” results are not useful without the “before” results, so we can see how the counters changed over the duration of your test.

 

[Sicario]

Do you have a reproducible example that can be used to verify whether this is a problem for all .12 CakeQOS users? If not, please flash back to .12 and try to find one. Because if we don't get this, it will likely not be fixed in .13 ....+

Unfortunately I don't have a reproducible example. What I'm thinking now is the problem might be not only with the streaming, it's just because we can really see how affecting to the stream. Currently, streaming is the most intensive usage in my home, no game, no voice or video call over internet, but I rarely seen VoWIFI status on my phone while at home wifi.

Anyway, I don't use TikTok though. Other than Twitch, the other example was when I watch some films from JFF+ Independent Cinema with 576p resolution at max, it buffers so many times, quick rollback to 386.11, buffers problem gone.

 

[igo]

After upgrading from 386.11 to 386.12 on my AC86U main node, I have experienced intermittent connection drops (about 2-3 a day, for a few minutes) on my AC66U_B1 Aimesh node (on stock firmware).

I set the logging level to "notice" in the System Log page, but still could not see any messages around the time when the connection drops happened.

I cannot verify whether it affected the whole network, or only the AC66U_B1 node, because I do not have physical access to it, and the users there may not experience- or report those small intermittent connection drops 2-3 times a day in the short run.

Back on 386.11 now, I will report back if the connection drops persist. But it was not a problem before upgrading to 386.12.

Noticed several connection drops as well. The OpenVPN did drop and was toggling between the two VPNs by VPN Monitor every 10 min or so. Roll back to 386 .11 seems stable again (for now) Anyone else has noticed this?

 

[heywire]

Noticed several connection drops as well. The OpenVPN did drop and was toggling between the two VPNs by VPN Monitor every 10 min or so. Roll back to 386 .11 seems stable again (for now) Anyone else has noticed this?

Have you tried other clients?

I used a macbook pro retina 2012 15" with OCLP (unofficial support) running MacOS Monterey at the time I wrote this post, because my main unit, Imac 2017 was out of order.

The MBPR2012-15 has a wifi card that is not officially supported after MacOS Catalina, so the OCLP developers has retrofitted these drivers to run with newer versions of MacOS. So my guess is that the connection drops was related to this. I am not experiencing it on my Imac now, even on 386.12.

I also ran OpenVPN (tunnelblick), and as it holds a constant connection to the host and pings it regularly, it would show a disconnection notice every time there was a drop.

If you have another client, try running openvpn on it, disable sleep/hibernation so the openvpn connection stays online, and pay notice to the time connected. If you check on in on it and the uptime shows a shorter time than when you started it, you probably had a wifi connection drop in-between.

 

[heywire]

Unfortunately I don't have a reproducible example. What I'm thinking now is the problem might be not only with the streaming, it's just because we can really see how affecting to the stream. Currently, streaming is the most intensive usage in my home, no game, no voice or video call over internet, but I rarely seen VoWIFI status on my phone while at home wifi.

Anyway, I don't use TikTok though. Other than Twitch, the other example was when I watch some films from JFF+ Independent Cinema with 576p resolution at max, it buffers so many times, quick rollback to 386.11, buffers problem gone.

Hmm.. It sounds to me like you just described one (reproducible example)?

If you want to avoid staying on 386.11 infinitely in order to avoid this bug, your best bet is to try the procedure @dave14305 described above, namely:

Capture the output of these commands immediately before and immediately after your tests on 386.12. If you have time, repeat on 386.11.

source /etc/cake-qos.conf
tc -s qdisc show dev $MIF
tc -s qdisc show dev $ULIF root

Post the results using

Spoiler

and

 tags like I did above.

 

[heywire]

All this talk about reproducible testing has me thinking of some ideas.

I'm wondering what tests @RMerlin are running before the release of the firmware releases? (we sure know from empirical evidence that Asus must be running just about none). Surely he/you only have a limited set of actual real-world devices/environments/clients to test on.

Is there any battery of standardized tests that can members can be guided to run in order to stress-test each release?

In that case, these release-threads could be filled with a lot more interesting comments than the more or less useful "dirty-flashed two minutes ago, everything working wonderful, *forgets to post which router*" comments that sadly tend to fill the thread.

A list/spreadsheed could be created, of how many members - and on what equipment (and maybe which clients?) - has successfully or unsuccessfully run these test on the firmware in question.

I'm thinking of something like bufferbloat tests (eg. https://www.waveform.com/tools/bufferbloat), file transfer tests, even usb test etc. Probably some others here have better ideas and knowledge of how we could test the performance(s) reliably and efficiently...

 

[dave14305]

All this talk about reproducible testing has me thinking of some ideas.

This is really drifting off-topic.

 

[heywire]

This is really drifting off-topic.

Not really. I, of whomever wants, may spin it off to another thread if we get a response indicating that it's worth pursuing something that passes the smell test. Just thought since we're talking about reproduing some errors, I'd ask what ways we could test errors for firmwares, now and in general, beginning with the current issues that are now the main topic of this thread.

 

[Tech9]

Very few people using their routers as core network devices can do that, @heywire. I used to do it on available routers not used for anything else, but test devices. In this case whatever happens even router bricked just doesn't matter - you don't disturb anyone in the process. I used to read what the issue is and attempt reproducing it on close platform router. Most people just update and go on with features they use. Something else may be broken for someone else and they'll report it. This is the testing process. There is no individual router testing before release. Not possible.

 

[heywire]

@Tech9 I was thinking more along the lines of a battery of simple test/log analyses that could weed out the most bugs that usually pops up in these threads after a while anyways (disconnection issues, buffering issues etc) - not reliable, controlled-environment rigorous "lab-testing" that could brick a router (like the one Asus should be doing, but surely aren't!).

In other words; instead of all network members learning the hard way through gettin blisters, burns and shovels thrown at your face through a number of days in the initial stages of a newly released firmware in threads like this, at least a subset of the most common problems could be excluded in the first hour or so by some recommended tests like the bufferbloat test, file transfer tests etc. (I remember I used a script on linux and an android app for this a number of years ago to check the max transfer rate of the wifi at different spots).

This would be done by a bunch of members running different setups, one could get at good grip on whether the firmware is stable enough for daily use instead of having to scan comments for weeks before daring to deploy at a semi-critical home setup.

Specifically as it relates to the current topic, I wonder if a bufferbloat test or some other suggested test would reproduce @Sicario and @igo 's issues.

 

[L&LD]

@heywire, with a billion+ different 'unique' combinations of environmental, devices, and device driver versions/settings possible, such a test is like a drop of rain in the ocean. And that doesn't even take into account non-WiFi sources of interference that may further skew the 'expected' testing results.

'Online' is a constantly and continuously evolving medium. No test bed that can be dreamed up can contain all those variables and properly account for them. Particularly, as a testbed needs to be comparable (and therefore 'static') to any previous such tests to have any impact.

The current model, while not perfect, is as close as you can get to a 'free' and superior firmware, and the quality is shocking considering this is a single person (RMerlin) doing it all.

That is why best practices, minimally and manually configuring your router after it is fully reset and other similar suggestions exist. Following them allows anyone to test if their hardware/network responds as expected (when it is in a good/known state). When/if their results differ, then they have a chance to track down any issues and fix them if possible.

To me, what you're suggesting is like starting at the tail and building proof that there is a dog chasing it.

 

[Tech9]

disconnection issues, buffering issues etc

This all comes from Asuswrt base. The reason I switched to testing Asuswrt only... before giving up completely. There no bug-free version. One thing fixed and another broken every single time release after release. All base issues eventually end up in Asuswrt-Merlin. Drivers, all TrendMicro, other proprietary components - all closed source. There was no point for me testing Asuswrt-Merlin first and then finding the same issue in base Asuswrt.