GT-BE98 Pro & 3 BE30000 - AiMesh Wireless Backhaul Dropping Packets Every 15 Minutes

[NotTheHerbie]

I have a GT-BE98 Pro as my primary router with three BE30000 AiMesh nodes. They are using the Wi-Fi 7 wireless backhaul. The GT-BE98 Pro has firmware version 3.0.0.6.102_32882 installed. (I had tried version 3.0.0.6.102_34491 three plus months ago, but it crashed my router and took me multiple firmware reloads and factory resets to get back to the original firmware. Unfortunately, it has been long enough I don't remember the details of all the problem, just all the pain.) The BE30000 nodes have firmware version 3.0.0.6.102_35476 installed.

The BE30000 nodes replaced three of four RT-AX92Us that were set up in media bridge mode. Prior to bringing in the GT-BE98 Pro, the four RT-AX92Us were operating as AiMesh nodes for a GT-AX11000 without any problems.

Since swapping in the BE30000s, I have been noticing intermittent dropouts and connectivity issues. My son and I work from home where this is most noticeable during video calls.

Over the past 24 hours, I've identified a way to reliably capture these events, which turned out to be packet loss over the wireless backhaul. The test environment involves running EMCO Software's Ping Monitor on a PC that is hardwired to the GT-BE98 Pro. The free version of this software can ping up to 5 hosts simultaneously every 1 second, collecting and displaying all of the relevant metrics. If 3 consecutive packets are lost, a connection is flagged as down. This results in a minimum reported down duration of 2 seconds. If only 1 or 2 consecutive packets are lost, the connection is not flagged as down, but the count of lost packets is still incremented. Once Ping Monitor begins receiving packets from a down connection, its state is changed to Up. This is a great piece of software that I once used to prove to Comcast that they had a time-of-day dependent network stability issue. They were able to take that information and identify the failing hardware.

To eliminate other wireless factors in testing the AiMesh backhaul connections, the hosts being pinged were all hardwired to their respective nodes. As controls, I used an RT-AX92U (in media bridge mode) connected to the 5 GHz channel with a host hardwired to it, and a host hardwired to the GT-BE98 Pro. The hosts were a collection of Windows PCs, Raspberry Pi 4s and a WiiM Pro Plus streamer.

Over a 5 hour time frame all hosts connected to a BE30000 showed 2-9 second disconnects every 14-15 minutes. The hosts connected to the RT-AX92U and directly to the GT-BE98 Pro showed no packet loss.

Has anyone else experienced Wi-Fi 7 wireless backhaul packet loss that is predictable with either of these routers? Does anyone have any suggestions on how to fix this? I know changing to a wired backhaul would probably eliminate the problem, but that just isn't practical at this time.

Does anyone know if Asuswrt-Merlin 3006.102.1 might fix this problem? The change log mentions "Merged with GPL 3.0.0.6.102_34369". Does this mean it's based on a stock Asuswrt version between what I currently have installed (3.0.0.6.102_32882) and what I tried to install (3.0.0.6.102_34491)?

Any help will be greatly appreciated! Thanks!

 

[jzchen]

I have the latest firmware installed, but I used 2 betas provided to me in between the two firmwares (on the BE98 Pro). I have it in AP mode with AiMesh but with wired backhaul.

My OnePlus Open phone does not get good speed from connection to the BE98 Pro, sadly...

 

[NotTheHerbie]

I took a closer look at the packet loss data from yesterday. Over a 12 hour period, using the Ping Monitor definition of connection down being 3 consecutive dropped packets and connection back up being 1 received packet, I calculated the mean and standard deviation for the number of seconds from the start of one connection down period to the next and for the duration of the down period.

Mean Time Between Wireless Backhaul Going Down: 900.043 seconds
Standard Deviation: 0.743 seconds
Mean Time Connection Down: 4.375 seconds
Standard Deviation: 0.781 seconds

The raw data for the above measurements was taken using two Windows 11 PCs, one hardwired to the GT-BE98 Pro and the other hardwired to a BE30000.

Since the determination of up or down is based on a 1 second ping interval, the above times need to be taken with a grain of salt. The actual start and end times for these connection loss periods fall somewhere between two consecutive pings. That being said, something is obviously occurring at a very predictable ~900 second interval that is taking the Wi-Fi 7 AiMesh Wireless Backhaul channel down for roughly 4-3/8 seconds.

Any ideas?

 

[NotTheHerbie]

The packet loss on the wireless backhaul connection for each of the three BE30000 AiMesh nodes begins simultaneously with the others. This would seem to point to the GT-BE98 Pro as the source of the problem. Every 900 seconds, the wireless backhaul goes down for all connected nodes for a little over 4 seconds.

 

[Lax]

What's the hardware version of the GT-BE98 Pro router? If it's V1.0 then you need to get it replaced with V2.0

 

[NotTheHerbie]

What's the hardware version of the GT-BE98 Pro router? If it's V1.0 then you need to get it replaced with V2.0

Hi Lax,

Thanks for the reminder. I meant to post this earlier. The hardware is V2.0.

I assume that if this is a problem with the AiMesh wireless backhaul code, it's maintained by ASUS and contained in one of the pre-compiled binary blobs provided to RMerlin.

If this is the case, does anyone happen to know if the latest ASUS firmware version (3.0.0.6.102_34491) or RMerlin's 3006.102.1 has a newer AiMesh code base?

As mentioned earlier, my attempt more than three months ago to upgrade to 34491 went horribly wrong. After spending some additional time in this forum, I think I know a little bit more about this creature called Wi-Fi 7 and ASUS' implementation.

Given my current predicament with the AiMesh backhaul dropouts, I'm going to make another attempt at upgrading the firmware. I just need to decide which firmware to start with.

Anyone have any suggestions?

 

[Lax]

I'm using ASUS firmware for now. Went back from RMerlin's 3006.102.1 firmware to ASUS firmware version (3.0.0.6.102_34491) for a reason that ASUS firmware allowed me to mix WPA2/WPA3 for MLO whereas RMerlin's firmware forces to WPA3 only. See the attached snapshot. And I'm using wired backhaul only.

 

[NotTheHerbie]

I'm using ASUS firmware for now. Went back from RMerlin's 3006.102.1 firmware to ASUS firmware version (3.0.0.6.102_34491) for a reason that ASUS firmware allowed me to mix WPA2/WPA3 for MLO whereas RMerlin's firmware forces to WPA3 only. See the attached snapshot. And I'm using wired backhaul only.

Thanks for the info. I'm not running MLO (or Smart Connect) since I don't yet have any Wi-Fi 7 clients. I'm currently giving 3006.102.2_alpha1 a try. Getting it installed and configured on my network was fairly painless.

Getting the BE30000 nodes set up is another story. Both the factory reset and the add AiMesh node take a long time and fail often. I'll post an update once I've got everything up and running and have collected enough data to know if the periodic pack loss problem is gone.

 

[jzchen]

Have you considered trying the ZenWiFi BQ16 Pro as router and BE98 Pro as node?

 

[doc octopus]

What's the hardware version of the GT-BE98 Pro router? If it's V1.0 then you need to get it replaced with V2.0

what problems does v1.0 have?

 

[NotTheHerbie]

Unfortunately, installing 3006.102.2_alpha1 in the GT-BE98 Pro did not eliminate the periodic packet loss on the wireless backhaul. This may be an indication that the problem lies in the BE30000 firmware or it just hasn't been found and corrected yet.

I have made an additional observation related to the packet loss that will hopefully help in locating the source of the problem. With both versions of the GT-BE98 Pro firmware tested and numerous reboots of everything, the packet loss ALWAYS occurs at 10, 25, 40 and 55 minutes past the hour!

@RMerlin are you aware of anything that could be causing this very predictable packet loss? I do understand that the problem could lie in one of the pre-compiled binary blobs that you don't have visibility into.

Would it make sense to try the latest ASUS firmware? I saw in the release notes for Asuswrt-Merlin 3006.102.1 that it was "Merged with GPL 3.0.0.6.102_34369." I also read your post explaining that the packages ASUS provides to you are custom made for you and cannot be directly compared to their public releases. Am I wrong to assume that version 3.0.0.6.102_34491 might contain some newer components?

Thanks!
- Mike

 

[RMerlin]

Would it make sense to try the latest ASUS firmware? I saw in the release notes for Asuswrt-Merlin 3006.102.1 that it was "Merged with GPL 3.0.0.6.102_34369." I also read your post explaining that the packages ASUS provides to you are custom made for you and cannot be directly compared to their public releases. Am I wrong to assume that version 3.0.0.6.102_34491 might contain some newer components?

Same code branch and same model, so 34491 will contain a few additionnal changes over 34369.

If the packet issues are on a schedule, look for any source of interference that would also be on a schedule. Also look at the system log for reports of the channel being preempted due to radar interference, for example.

 

[NotTheHerbie]

Same code branch and same model, so 34491 will contain a few additionnal changes over 34369.

If the packet issues are on a schedule, look for any source of interference that would also be on a schedule. Also look at the system log for reports of the channel being preempted due to radar interference, for example.

Thanks for the reply!

I downloaded the log and could find no references to the channel being preempted or even the word radar. I did find many instances of the "potentially unexpected fatal signal 11" / "dcd Tainted" error. I disabled AiProtection and those errors went away, but the packet loss at 10, 25, 40 and 55 minutes past the hour were still occurring.

A couple of things that I did notice after updating the firmware to 3006.102.2_alpha1:

• In addition to the "scheduled" packet loss, I'm also seeing occasional random dropouts. With 3.0.0.6.102_32882 installed, the only packet loss I was seeing were at the predicted times.
• Previously, the 3 nodes all used the 6 GHz-1 dedicated wireless backhaul, since updating to 3006.102.2_alpha1, two of the nodes will change to the 5 GHz band and stay there.
• For some reason, the 6 GHz-1 dedicated wireless backhaul was given the SSID ChaosCentral_5G_dwb. Before adding the mesh nodes, I had set the 6 GHz-1 band SSID to ChaosCentral_6G-Backhaul.

When I get a chance later today, I'm going to try the latest ASUS GT-BE98 Pro firmware, 3.0.0.6.102_34491. Before attempting this, I want to make sure everything is reset to factory defaults.

What is the most reliable way to reset the BE30000 to factory settings? Is it through the UI? There is no user manual (UM) specifically for the BE30000. The FAQ says to use either the UI or hard reset method 2 (hold WPS button while applying power) for the BE30000 (and the BQ16 Pro). ASUS recently added a user manual for the BQ16 Pro, which I understand should be the same hardware and firmware. The BQ16 Pro UM says to use the UI or press the Reset button. There is no additional information regarding short press or long press or hold while applying power. The BQ16 Pro UM does not reference the WPS button. If you try method 2, you are supposed to release the WPS button once the power light goes off. It never does. Any suggestions here will be greatly appreciated. I think both hard reset methods may work (method 2 and Reset button), but I'm not certain, because it takes a very long time to complete and sometimes it never completes. Maybe it's a combination of using both .

One other thing I noticed after installing 3006.102.2_alpha1 was performing a hard factory reset (method 2) on the GT-BE98 Pro appeared to clear everything except for removing the AiMesh nodes. Although they were all turned off, the GT-BE98 Pro still showed them (disconnected) after the factory reset was complete. I manually removed the nodes before setting everything back up, but I'm now wondering if this was sufficient. Should method 2 have removed the nodes? Do I need to remove them before hard resetting or even installing new firmware?

Thanks,
Mike

 

[NotTheHerbie]

After Installing 3.0.0.6.102_34491 in the GT-BE98 Pro, I'm still seeing periodic packet loss in the wireless backhaul, but now it's every 20 minutes instead of the previously seen 15 minutes.

For this test, I've left Smart Connect enabled and enabled MLO. I used Guest Network Pro to set up three SDNs to mirror my previous setup of one SSID per band. I used the Customized Network template as the starting point for each. I have the default IoT Network that ASUS creates disabled because it does not support all of the options the Customized Network template supports.

The 2.4GHz and 5GHz bands are set to WPA2/WPA3-Personal and the 6GHz band to WPA3-Personal authentication. All three profiles are displaying the WiFi7 icon, but must not be using the updated GCMP-256 encryption since all of my legacy devices (40+) appear to be connecting without issue.

I did still notice the mesh nodes switching between using a 5GHz and 6GHz backhaul. I also saw a lot of stationary clients jumping between nodes. I contemplated disabling Roaming Assistant, but instead lowered the threshold to -80dBm. The backhauls now appear to be fixed at 6GHz and I am seeing significantly fewer clients bouncing between nodes. What is the current recommendation for Roaming Assistant? Do most people leave it enabled or disabled? Are there any other non-default settings currently recommended for these new routers?

If the problem with the wireless backhaul dropping every 20 minutes could be solved, this feels like a configuration that could be the best of all worlds. The remaining caveat is that I don't currently have a WiFi 7 client that supports MLO so that capability has not been tested yet.

Please ASUS, fix the wireless backhaul!!!

 

[jzchen]

I know of one person running two GT-BE98 Pros with wireless backhaul and is happy with the performance. He's on the ROG forums...

As background my BE98 Pro is an old HW version, I don't even want to look, but I'm sure highest it could be is Ver 1.0...

I notice you tried the latest firmware, which enables MLO. Be careful, because once you touch a setting that it auto configures, you may have disabled MLO without even realizing it. Case in point if you changed the WPA to WPA2/WPA3 Personal, this is not an MLO setting, because MLO requires WPA 3 Personal AES AND GCMP 256. So I'm afraid you'll need to not touch that, to truly try the MLO setting.

To confirm this suspicion after setting MLO on toggle the MLO toggle switch OFF, then back ON. It should warn you that your settings are not MLO compatible, and it will continue to correct all incompatible settings. Also, even though Legacy devices do connect to the MLO SSID, they eventually fall off...

The concern I have with the latest firmware is that once you create MLO, it takes all four channels, AND helps create a Legacy/IoT SSID, BUT that SSID is WPA2, and I want to switch it to WPA2/WPA3. They have not enabled that. Without WPA3 my wife's work provided laptop will not work, (had to figure out why it would not work with EEROs and it was because they enforce WPA3)...

A couple things until I get my single BQ16 Pro delivered by FedEx:

Either completely remove or hardwire the AX router node. As I noted the ROG forum member is running two BE98 Pros with wireless backhaul AND is happy. No AX(E) routers in the mix...

Once you toggle MLO ON do not touch settings that could disable it, because it will without warning. You can toggle OFF then back ON to confirm that the setting affected it with the warning that shows up....

 

[jzchen]

Oops, I should clarify something. WiFi 7 requires AES + GCMP 256. What WiFi 7 enables that I didn't realize is 320 MHz bandwidth. Without it you're limited to 160 MHz on the 6 GHz bands. So that means be careful what you set the WPA setting, because I don't know how to check whether you disable WiFi 7 without knowing it...

 

[jzchen]

Adding the single BQ16 Pro to the room I spend the most time, wired backhaul, seems to have solved my poor speed issue. Stranger yet, when I go and connect my OnePlus Open phone to the BE98 Pro, it's Speed Test in the app (Ookla) is good now too.

It may be worth a try to set up settings as ASUS defaults and/or recommends. It's very likely that ASUS tried to make sure everything is functioning well with the settings they advise you to set. Here's the MLO suggested settings:

[Wi-Fi 7 Wireless Router] What is MLO (multi-link Operation) and how it works? | Official Support | ASUS USA

Like you I have chosen the BE98 Pro as main Access Point with AiMesh. (But I only did so because the BE98 Pro has a higher power consumption, 19.5 V at 3.33 A vs 12 V at 5 A).

I'm at peace now although ping inconsistency exists...

 

[jzchen]

BTW- You don't happen to have a clock or something similar that happens to chime, etc every quarter hour?

(These things seem sensitive to noise of the electric nature)...

 

[NotTheHerbie]

After numerous factory resets with different firmware versions, I have come to the conclusion that the wireless backhaul between the GT-BE98 Pro and ZenWiFi BE30000 (BQ16 Pro) is just not stable. Depending on the configuration, I get very predictable disconnects at either 15 or 20 minute intervals. I should point out that these "disconnects" are actually very long lag spikes (>1500 milliseconds) that result in clients declaring no internet connection.

These periodic disconnects are unacceptable during working hours or while gaming in the evening. During business meetings, chat clients will disconnect for 5-10 seconds and GeForce Now often disconnects and shuts down the virtual machine your connected to. When online gaming on your local machine, the lag death is real.

I have resigned myself to converting my 3 BE30000 mesh nodes to $1700 worth of paperweights until ASUS fixes their firmware. (p.s., I despise the marketing guys that decided these routers would not support Media Bridge Mode.) Everything must now connect to my GT-BE98 Pro, which means I don't have complete coverage of my house and speeds are much worse in certain rooms. At least I am not getting the every 15 or 20 minute disconnects.

I do want to point out that there still is something "wrong" with the GT-BE98 Pro. I still see periodic lag spikes in the 100s of milliseconds. They just aren't long enough to cause clients to disconnect.

ASUS, please fix your firmware!

Note: The last configuration I tested with used the latest stock firmware for each model.
GT-BE98 Pro: 3.0.0.6.102_34491
BE30000: 3.0.0.6.102_36736