Skynet / dnsmasq / Flowcache / Merlin 388 releases

[aex.perez]

Before anyone says anything I don't care what the Speedtest speed results are, I use them to monitor consistency, quality
At this point my Speedtests are within what I consider my normal: between 900 to 930 Mbps every 30m - both up and down
Been running for weeks all good and even te occasional hiccup (crash / restart) and been unoticed or minor nuissance (dnsmasq restart)

Then, without changing a thing...
Then this happens: See attached "router reboot time.txt"

After all this my new Speedtest normal is 450-500Mbps on the download and 653-921Mbps on the upload every 30m
Speedtest test results on AT&T BGW210 during this? Between 950 to 998 Mbps consistently - both up and down
Again, for the narrowly focused, ugh another speedtest perfrmance complaint, let me reiterate
I don't care what the Speedtest speed results are, I use them only to monitor consistency, quality.
After the Skynet/Dnsmasq/flowcache shennnigans something way off on consistency

Restatrting Skynet and dnsmasq has minor impact on consiistency of the speedtest results 550-700Mbps on the download, and unchanged on upload.

Latency and Jitter remain in relatively unchanged, the amounts of data uploaded and downloaded to achieve a result unchanged as well
Picking the "best" server at the instant the speed test is run. Once this error occurs the only recovery is a reboot.
After the reboot, back to my normal: between 900 to 930 Mbps every 30m - both up and down
Again, I'll share that the Speedtest test results on AT&T BGW210 during this were consistenly between 950 to 998 Mbps - both up and down before and after the reboot, and during the wierdness on the router.

No impacts to WiFi or Wired clients initially. But after sometime (have no measure) they also begin to degrade in terms of packet loss and their own speedtests
No TrendMicro features are enabled, no IPv6, no QoS, use DNSSEC (Cloudflare DNS - to block malicious content), do use ASUS DDNS

I thought I had seen the last of this, haven't seen it happen in awhile, no crashes either since I implemented "sync; echo 3 > /proc/sys/vm/drop_caches" in a script I run every AM before things get busy
But did get back to back crashes on 5/3- and 5/31 - no impact during the veray early AM hours, and a dnsmasq restart in the middle of streaming Quatumamia a few days ago but everything recovered without going hands on.

When this occurrs, I run TOP to look at the running processes, BCMSW_RX (Broadcom drivers?) is almost always in the top spot, under normal conditions it never is...
Running HTOP and TOP side by side, (separate monitors) BCMSW_RX does not show on HTOP when on TOP it does

TOP
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
21072 21036 Master S 7928 0.8 2 0.4 htop
231 2 Master SW 0 0.0 3 0.4 [bcmsw_rx]

This didn't happen in 386 family, had the same setup but with IPv6
Which brings it all back to normalcy, so not an ISP as the AT&T BGW210 was consistent during the time the router was not...

I've HW reset, reloaded the firmware, reconfigured by hand, still happens but I can't seem to capture the trigger, just the after effects - see attached "router reboot time.txt"
Any thoughts, that don't involve a variation of "you shouldn't run, trust, count on, expect" anything from running Speedtest on a router as this has 0 to do with "a" speedtest result but with the fact that when this happens all results are off until a reboot.

EDIT/UPDATE: a router reboot does NOT fix the issue. Only a Router power cycle, on/off button, restores the router to it's normal operating state

 

[dave14305]

Bookmark the router’s speedtest page and don’t navigate to it via the Adaptive QoS menu anymore. Loading the Bandwidth Monitor page (which you do when you click on Adaptive QoS) can load some bwdpi modules and result in similar logs like you posted.

 

[Tech9]

After all this my new Speedtest normal is 450-500Mbps on the download and 653-921Mbps on the upload every 30m
Speedtest test results on AT&T BGW210 during this? Between 950 to 998 Mbps consistently - both up and down

It's not about the Speedtest, but what will happen if you run it on a wired device only when you notice something off and leave the router alone? Does anyone else see any inconsistencies and connection quality issues during normal Internet daily activities? Why is this constant monitoring necessary?

 

[aex.perez]

It's not about the Speedtest, but what will happen if you run it on a wired device only when you notice something off and leave the router alone?

Great question, initially nothing. But after a few hours (a day) it will begin to degrade.

On Wifi devices you'll begin to get assymetrical results like what's happening on the router. The download results on thoose devices will sho a 200-350Mbs delta between download and upload, where on the router the upload is higher and the download is lower. Under normal conditions the download/upload results are almost identical. On a wired devices the impacts are seen after a longer while, but there too under normal conditions the U/D results are almost identical to each other not so when this happens.

On the Mesh nodes, while this is going on, any devices on on those AX86u's also get impacted, to be expected. The results are higher Mbps than when on the router for WiFi devices but the symetrical/assymetrical anomaly stays constant. Same for the Wired devices on mesh nodes.

I can say for sure that on the router these results occur AFTER all these restarts as in the log. For WiFi devices, on the Router or Node, I strongly believe I was getting assymetrical results for a while before this log, but after the power off/on - everything is symetrical (+/- 50Mbps on the U/D results) regardless of the result. I need to deep dive/log that more to be 100% sure.

EDIT/UPDATE, going through the phones and tablets (2.4 and 5Ghz) and correlating results with dates. I would get assymetrical results BEFORE these errors on the router. Sometimes lower or higher (to be expected) but a 150-300Mbps delta versus the norm +/- 50MMbps between U/D

 

[aex.perez]

Bookmark the router’s speedtest page and don’t navigate to it via the Adaptive QoS menu anymore. Loading the Bandwidth Monitor page (which you do when you click on Adaptive QoS) can load some bwdpi modules and result in similar logs like you posted.

I use spdMerlin to avoid that, as I gives me a data base with the results making it easier to track and a nice graph to visually see it but the automated tests...
And whether I use the built in or external binary the results are the same, I don't use QoS but I have them set to be excluded from QoS just the same.

Here the graphic with the resuts, that first dip aligns exactly to the time of the events in the log, at the end there a quick blip from the reboot, and when that didn't work a power cycle, then after the power cycle, back to normal...

A little more direct here, which makes it easier to narrow down where I have to look in the logs to a 30m window
Time ↓ Download (Mbps) Upload (Mbps) Latency (ms) Jitter (ms)
2023-06-03 19:00:25 496.92 918.11 4.25 0.21
2023-06-03 18:30:24 455.1 889.85 4.01 0.24
2023-06-03 18:00:27 476.57 647.7 4.43 0.25
2023-06-03 17:30:24 476.41 841.02 3.69 0.32
2023-06-03 17:00:28 500.07 844.98 4.33 0.3
2023-06-03 16:30:23 926.48 935.69 3.88 0.22
2023-06-03 16:00:24 923.02 930.34 4.02 0.85
2023-06-03 15:30:24 923.91 937.25 4.15 0.25
2023-06-03 15:00:26 925.8 935.42 4.37 0.87
2023-06-03 14:30:25 928.86 933.11 4.51 0.14

 

[Tech9]

Have you ever tried running stock Asuswrt on all your routers without any scripts and monitoring and check what's the behavior? If there is no inconsistencies with simpler setup and no much 3rd party involvement the origins of your problem will be obvious. @shabbs has one router (I believe he had the same RT-AX88U main router before) and 3x AX86U APs and he does long uptimes with no issues on Asuswrt-Merlin. Very similar setup.

 

[aex.perez]

That's next, but a major undertaking. I have tried a basic setup with Merlin without a lot of addons, expect for OpenVPN and Speedtest to see if/when it happens. It still did.
I need to find a window of time to get everything on stock ASUS to do this again. The challenge is finding a good window of time to manage through all the dependencies (people/IOT devices). I haven't run stock ASUS for years, got to get back up to speed though for most of what I have and how I have it, it'll be a non issue. Then once I get past that hurdle, I need to figure out how I would monitor/capture the event if it happens to be able to share it...

What I was thinking of doing is switching roles of one of the AX86u nodes and the AX88u router, though I think same GPL, HW is different with ASUS I this the GPL is different on each

 

[Tech9]

This is my last time saying this to you: I have tested all add-ons one by one in details - everything they can possibly do, good or bad. The ones you have installed impact your user experience and even interfere with each other in a negative way depending on the sequence they were installed. I mean - up to completely wiped other custom settings with no questions asked, one of the add-ons you install does that. Five of your add-ons don't know about 388 firmware and are in fact unsupported with stopped or limited development. What you experience is perhaps created by one of two add-ons you install every time or the combination of both. I'm going to leave on you to discover which ones because someone may call me a bad guy again for looking into details too much. The fact is people with very similar to your configuration have no issues because - 1) they have eliminated AiMesh problem; 2) they don't use TrendMicro bloatware; 3) they don't install so many add-ons for no reason; 4) they don't have control and monitoring obsession.

 

[aex.perez]

Well, back to stock. Was much less painful than I imagined.

22525 on the router (AX88u) and 23285 on the nodes (AX86u), pretty painless. Obviously no add ons/scripts. LEDs at the router always on. Mesh nodes (bedrooms) always off. Turned IPV6 back on, simple one GUI panel and came right up. Did not choose to take it to factory default, in the hope that most everything would stay the same from a config standpoint, and it mostly did. If It still burps, that's always an option. Also Firewall had to change a bit without Skynet. Still no TrendMicro though. DDNS registered back with ASUS getting both IPv4 and IPv6 addresses.

Leaving SSH access on, though other than running TOP to get a picture of what the router/nodes are doing I don't think there much use for having it on anymore. Also noticed that the System log is not as noisy, likely as a result of the log message level, need to check that, there's no way to set log levels from the GUI like there used to be. Whether that a ASUS firwamre thing or related to not taking it back to factory, not sure.

The disabling of 802.11b at the nodes NVRAM settings were kept as well, one less thing to worry about.

Nodes to ASUS firmware first, made sure they were both operational and the network stable, then the router.

The one last thing, arguably the most important one is OpenVPN but judging by the GUI it looks like most of it came over, only had to change the cipher back to what it was and set the access to Internet and local again (came over as custom) and at least my phone connected back in, now just have to validate the tablet my son's nurse uses, but I suspect that'll be fine. Wireguard, no changes need, came right up.

So for now, stock. when 388 and the addons get a little more matured...

 

[Tech9]

For cleaner results I would start fresh on all devices and without any not really needed customizations. I would also keep IPv6 default disabled if public IPv4 WAN IP is available and IPv6 is not needed. Asuswrt has some partial IPv6 support and I found IPv6 enabled messes some of the firmware features on almost every 388 firmware release. This is one of the things I target test every time and there is always something broken. I don't know why you even bothered with IPv4 only Skynet. You locked the front door and left the windows open.

 

[aex.perez]

Time constraints forcing my hand a bit, the logistics of getting the OpenVpn profiles out and people to understand how to use them another. Needed to get this done quick with as little disruption as possible. IPv6 is something I use to test as well, works well on the Google Nest devices (oddly more responsive to real time events). The tablets used to monitor the cameras, 2.4Ghz AX are oddly smoother as well. I suspect, as WiFi is closed source, they'll work just as they did under 388. One thing I miss, sort of, is the site survey, not the Broadcom one but the new one that was added in 388. I say sort of because I have a wired PC that has a M2 slot with a WiFi adapter and I run SSID there to get the same thing.

Certainly starting fresh is an option, but the logistics (a few IoT devices, one medical) and dependencies (very not tech savy people/users primarily) forced my hand to take this route as a start. If this is successful as it has been so far, then if I can get everything and everyone coordinated I can wipe it all and start fresh. Just could not get all that done in time, so kinda had to shortcut the process...

I've got the main WIFI (SmartConnect) with two Guest (1) WIFI, one for 2.4Ghz one for 5Ghz to separate out the IoT devices a couple of those had nuissance issues when they could conect to both radios.

 

[vlord]

Time constraints forcing my hand a bit, the logistics of getting the OpenVpn profiles out and people to understand how to use them another. Needed to get this done quick with as little disruption as possible. IPv6 is something I use to test as well, works well on the Google Nest devices (oddly more responsive to real time events). The tablets used to monitor the cameras, 2.4Ghz AX are oddly smoother as well. I suspect, as WiFi is closed source, they'll work just as they did under 388. One thing I miss, sort of, is the site survey, not the Broadcom one but the new one that was added in 388. I say sort of because I have a wired PC that has a M2 slot with a WiFi adapter and I run SSID there to get the same thing.

Certainly starting fresh is an option, but the logistics (a few IoT devices, one medical) and dependencies (very not tech savy people/users primarily) forced my hand to take this route as a start. If this is successful as it has been so far, then if I can get everything and everyone coordinated I can wipe it all and start fresh. Just could not get all that done in time, so kinda had to shortcut the process...

I've got the main WIFI (SmartConnect) with two Guest (1) WIFI, one for 2.4Ghz one for 5Ghz to separate out the IoT devices a couple of those had nuissance issues when they could conect to both radios.

Have you performed any performance tests? I’ve been toying with this idea as well and moving services like AGH to a Raspberry Pi.

 

[aex.perez]

Have you performed any performance tests? I’ve been toying with this idea as well and moving services like AGH to a Raspberry Pi.

Really can’t see a difference from a performance aspect. It’s to early on the stability piece. Even with scripts I was using under 386/388 it didn’t pound the CPU enough to impact performance. But while under 386 my setup was stable, under 388 I had challenges I could not resolve.

That said, and I have no comparable metrics to measure, WiFi seems to be more responsive and consistent as the tablets that have the video feeds from the Nest cams. As I say this, the AX88 is running the same GPL base as in 388.2-2, so it may just be the placebo effect kicking in…

 

[Tech9]

If you reset your system and start fresh it will be perhaps even more responsive. For performance on home routers with RPi-like hardware and RAM constraints the simpler you go the better. You still want to optimize things you don't need to optimize. You are not very far from the fact that latest 386 series firmware works indeed better and are more mature and bug-free. When testing 388 I often compare to last 386 release (on my AX86U) and very often find 386 works right and 388 is broken. On one of the firmware releases the firewall was broken for a month exposing IPv6 enabled devices to Internet (with specific settings). It got into Asuswrt-Merlin and was discovered by accident by @Treadler if I remember correctly. I was the only one to post a warning in Asuswrt release thread. In current new 388 firmware the firewall is broken again - this time not retaining settings in Network Services Filter tab. Bandwidth Limiter and Traditional QoS are broken with IPv6 enabled for few firmware releases already. I'm not even talking about copy/paste GUI errors present for many months. This is the level of firmware quality control you are getting on a device with $50 hardware inside. The more you poke settings the more issues you are going to have. When I say ToysRUs routers I mean it. Very disappointing getting this from Asus, but this is the reality.

Don't push for 160MHz in DFS. You are making your Wi-Fi less reliable. If you cut off SmartConnect you may see some instant improvements as well. If you cut off AiMesh altogether you may get up to 2x increase in aggregate throughput to clients connected to the APs and more control for better roaming.

 

[aex.perez]

Actually my WiFi/roaming has been the one constant in terms of reliabilty. The big change was moving from Auto to a fixed channel, that really helped stabilize it in the early 388 days and I've kept it that way. I'm blessed to live in an area where I have little to no neighborly interference, and no airport/military radar to contend with. But the drive by traffic going to and from the nearby elementary & secondary schools, it's amazing to see how many vehicles have AP's these days (and the creative names people use for them), that a momentarily blip at a very low dBm and not enough to create any issues...

I've got 1Gb U/D with AT&T fiber and get nearly that at the router, and devices on the wire. On the iPhone 12Max doing from 550 to 650Mbps, which is more than enough for any of my WiFi devices. Nodes wired 1GbE backhaul as the router only does 1GbE even though the nodes can do 2.5GbE.

I'll have three 4K UHD streams going, plus all the IoT, Nest Cams and the tablets that have their video feeds streaming and other stuff and still get great results. At this level nothing hiccups, now if I ever manage to push it harder and I can get AT&T to deliver 2Gb or 5Gb service here, a different story. But with my son moving out soon, they'll be much less of a demand (unless there's such a thing as WiFi collars for the cats and dogs, I don't want to know if there is )

But for now, the balance between simplicity and performance using the AX88/AX86 nodes makes things easy, and the whole house and driveway is covered.

You'll not like this part (40Mhz) but it's working great and the 2.4Ghz AX tablets and the wife's Volvo (can connect to WiFi network, either radio) utilize the bandwidth, note 802.11b disabled at the router and nodes.

DFS enabled, rarely drops except for the rare Police flyover, but then comes back, 5Ghz doesn't travel as well through my walls, so you can likely guess this computher is closer to the router vs the nodes based on signal strength. Basic Rates used to be 6, 12, 24 for the nodes under 386/388 but I'll need to dig into NVram on them to look for differences, or maybe it's them on 23285 vs the router on 22525. I'll look at that over the next few days.

Could I get more LAN performance, sure but it's pretty good right now, and most of the I/O is LAN/Internet not LAN/LAN (If I ever fully get IoT/Matter enabled that'll change).
Will it make a difference with what I have, maybe - I regularly get 900-950Mbps with under 5ms from a wired client and the DirecTV streaming boxes and Samsung TV's max out at 98Mbps even with all three streaming 4k UHD Dolby Atmos (and so on) still get great performance. Worth it, likey not enough of a gain to be made to make it worth the effort, but you tell me.

That said, 5GB Service and that ROG-BE98 backed by MultiGB switches for the wired devices, a whole other story.
But seeing as I'm not pushing it hard enough now to have it break a sweat, and soon we'll be empty nesters, I suspect we'll be pushing it even less...

Ugh, got to change my equipment/software list now...

 

[vlord]

If you reset your system and start fresh it will be perhaps even more responsive. For performance on home routers with RPi-like hardware and RAM constraints the simpler you go the better. You still want to optimize things you don't need to optimize. You are not very far from the fact that latest 386 series firmware works indeed better and are more mature and bug-free. When testing 388 I often compare to last 386 release (on my AX86U) and very often find 386 works right and 388 is broken. On one of the firmware releases the firewall was broken for a month exposing IPv6 enabled devices to Internet (with specific settings). It got into Asuswrt-Merlin and was discovered by accident by @Treadler if I remember correctly. I was the only one to post a warning in Asuswrt release thread. In current new 388 firmware the firewall is broken again - this time not retaining settings in Network Services Filter tab. Bandwidth Limiter and Traditional QoS are broken with IPv6 enabled for few firmware releases already. I'm not even talking about copy/paste GUI errors present for many months. This is the level of firmware quality control you are getting on a device with $50 hardware inside. The more you poke settings the more issues you are going to have. When I say ToysRUs routers I mean it. Very disappointing getting this from Asus, but this is the reality.

Are the 388 issues you’ve encountered specific to v6 or are they with v4 as well? Which 386 release are you running and is it Asus WRT or Merlin?

 

[aex.perez]

This specific issue, 388. The same exact setup was great under 386.8 / 386.7.2. Never had anything like what you see in the log on the first post in any release even going back to when I had the 5300’s.

IPV6, turning that off got some CPU and memory stability, even though the dnsmasq restart would occur. Also eliminated some of the variability in Speedtest results, the CPU utilization over time average was higher with IPV6 which obviously was seen in the Speedtest results.

But with IPV6 off I thought maybe I had overcome it (truth be told under 386 simple to turn on, under 388 not so much). But the log in the first post proves that theory wrong.

When that set of circumstances occurs, it’s power cycle time, not even a reboot recovers it. Which is why I anticipate still having problems with ASUSWRT 388 and will likely end up back on the last of the Merlín 386 for my gear, and all the scrips, which was working flawlessly…

Seeing as I got some emails from my WD NAS last night about losing connectivity, that move back to 386 might be sooner than later…. Got some work to do to see the what and why of it

 

[aex.perez]

Back to 386 I go, same crash under ASUSWRT 388 but with a little more detail... See attached...
Firgured it would happen, just not this soon

Same oddball before and after power cycle

 

[Tech9]

Are the 388 issues you’ve encountered specific to v6 or are they with v4 as well?

Mostly around IPv6 and one is really bad on 388.22525 bringing the 2.5GbE controller to some coma state and only hard reset recovers the router. For obvious reasons I did not post anywhere the settings sequence. I can also reproduce it 2 out of 3 attempts and AX86U has undocumented hardware variants with Broadcom and Realtek 2.5GbE controllers. Reported to Asus, let's see what happens. On 386 there is no issue. I still believe 386_49599 is the best Asuswrt for this router and 386.5_2 is the best Asuswrt-Merlin. Unfortunately, both are getting old and with missing security patches.

Which 386 release are you running and is it Asus WRT or Merlin?

I mostly follow stock Asuswrt development because new versions come first and I can test and report bugs before they get to Asuswrt-Merlin. The router is used for experiments only and I often jump between Asuswrt and Asuswrt-Merlin to compare. It's running 388_23285 at the moment, but this can change any moment. I'll get one RT-AX86U Pro when it becomes available around here to compare the Pro to non-Pro hardware and software.

 

[vlord]

Mostly around IPv6 and one is really bad on 388.22525 bringing the 2.5GbE controller to some coma state and only hard reset recovers the router. For obvious reasons I did not post anywhere the settings sequence. I can also reproduce it 2 out of 3 attempts and AX86U has undocumented hardware variants with Broadcom and Realtek 2.5GbE controllers. Reported to Asus, let's see what happens. On 386 there is no issue. I still believe 386_49599 is the best Asuswrt for this router and 386.5_2 is the best Asuswrt-Merlin. Unfortunately, both are getting old and with missing security patches.

Does Asus have variants outside of the Broadcom BCM54991ELB0KFEBG and Realtek RTL8221B-VB-CG? Those are the only two documented 2.5G controllers for the AX86U in the US at least.