AX86U LAN Link Aggregation: doesn't seem to be working?

[i0ntempest]

Hi all,
Since I have a 1.5G internet package I figured I should try link aggregation so my devices can take advantage of that. My main router is an AX86U with the latest Merlin firmware, and I have setup my network to be like this:

Modem <== 2.5G link ==> ASUS RT-AX86U <== 2*1G LAG ==> Netgear MS510TX switch <== 1/2.5/5G links ==> Some other devices
                             ^^                          ^^          ^^
                             ||                          ||          ||
                          1G links                    10G link    10G link
                             ||                          ||          ||
                             vv                          vv          vv
                      Some other devices                PC 1        PC 2

I get it that each one MAC address will only get a max of 1 Gbps, but even when I run speed tests on PC 1 and 2 at the same time, their combined speed will not exceed 1 Gbps. Using the router's built in speed test I am able to get 1.5 Gbps. Am I missing something, or am I still not understanding LAG properly?

 

[Tech Junky]

LAG / L:ACP has different modes to choose from and sometimes the default sucks for performance. I used mode 4 on my server / modem setup as the modem doesn't like other options.

LACP bonding and Linux configuration -

docs.rackspace.com

The behavior of the bonded interfaces depends on the mode that is selected.

• Mode 0 (balance-rr): This mode is also known as round-robin mode. Packets are sequentially transmitted and received through each interface one by one. This mode provides load balancing functionality.
• Mode 1 (active-backup): This mode has only one interface set to active, while all other interfaces are in the backup state. If the active interface fails, a backup interface replaces it as the only active interface in the bond. The media access control (MAC) address of the bond interface in mode 1 is visible on only one port (the network adapter), which prevents confusion for the switch. Mode 1 provides fault tolerance.
• Mode 2 (balance-xor): The source MAC address uses exclusive or (XOR) logic with the destination MAC address. This calculation ensures that the same slave interface is selected for each destination MAC address. Mode 2 provides fault tolerance and load balancing.
• Mode 3 (broadcast): All transmissions are sent to all the slaves. This mode provides fault tolerance.
• Mode 4 (802.3ad): This mode creates aggregation groups that share the same speed and duplex settings, and it requires a switch that supports an IEEE 802.3ad dynamic link. Mode 4 uses all interfaces in the active aggregation group. For example, you can aggregate three 1 GB per second (GBPS) ports into a 3 GBPS trunk port. This is equivalent to having one interface with 3 GBPS speed. It provides fault tolerance and load balancing.
• Mode 5 (balance-tlb): This mode ensures that the outgoing traffic distribution is set according to the load on each interface and that the current interface receives all the incoming traffic. If the assigned interface fails to receive traffic, another interface is assigned to the receiving role. It provides fault tolerance and load balancing.
• Mode 6 (balance-alb): This mode is supported only in x86 environments. The receiving packets are load balanced through Address Resolution Protocol (ARP) negotiation. This mode provides fault tolerance and load balancing.

 

[Tech Junky]

Also, move these to the switch instead to see if you can get more throughput from the router as sometimes things only go as fast as your slowest port.

 

[i0ntempest]

LAG / L:ACP has different modes to choose from and sometimes the default sucks for performance. I used mode 4 on my server / modem setup as the modem doesn't like other options.

LACP bonding and Linux configuration -

docs.rackspace.com

The behavior of the bonded interfaces depends on the mode that is selected.

• Mode 0 (balance-rr): This mode is also known as round-robin mode. Packets are sequentially transmitted and received through each interface one by one. This mode provides load balancing functionality.
• Mode 1 (active-backup): This mode has only one interface set to active, while all other interfaces are in the backup state. If the active interface fails, a backup interface replaces it as the only active interface in the bond. The media access control (MAC) address of the bond interface in mode 1 is visible on only one port (the network adapter), which prevents confusion for the switch. Mode 1 provides fault tolerance.
• Mode 2 (balance-xor): The source MAC address uses exclusive or (XOR) logic with the destination MAC address. This calculation ensures that the same slave interface is selected for each destination MAC address. Mode 2 provides fault tolerance and load balancing.
• Mode 3 (broadcast): All transmissions are sent to all the slaves. This mode provides fault tolerance.
• Mode 4 (802.3ad): This mode creates aggregation groups that share the same speed and duplex settings, and it requires a switch that supports an IEEE 802.3ad dynamic link. Mode 4 uses all interfaces in the active aggregation group. For example, you can aggregate three 1 GB per second (GBPS) ports into a 3 GBPS trunk port. This is equivalent to having one interface with 3 GBPS speed. It provides fault tolerance and load balancing.
• Mode 5 (balance-tlb): This mode ensures that the outgoing traffic distribution is set according to the load on each interface and that the current interface receives all the incoming traffic. If the assigned interface fails to receive traffic, another interface is assigned to the receiving role. It provides fault tolerance and load balancing.
• Mode 6 (balance-alb): This mode is supported only in x86 environments. The receiving packets are load balanced through Address Resolution Protocol (ARP) negotiation. This mode provides fault tolerance and load balancing.

Thanks for the info. I believe 802.3ad is the only supported protocol in Asus firmware so I used that on the switch as well. In the switch there's also "LACP System Priority" and "LACP Priority" settings, but I'm not sure what those do.

 

[Tech Junky]

I wouldn't bother with the LACP settings beyond pairing 2 ports for 2GE. Moving the mentioned devices to the switch should make the router push 2GE through the switch and result in higher throughput though.

 

[i0ntempest]

I wouldn't bother with the LACP settings beyond pairing 2 ports for 2GE. Moving the mentioned devices to the switch should make the router push 2GE through the switch and result in higher throughput though.

So sorry, my bad... I'm stupid enough to have forgot the shaping rate was limited on one of the bonded interfaces. Removing that allowed a combined speed of about 1.4 Gbps.

 

[Tech Junky]

It happens!

 

[i0ntempest]

It happens!

I just noticed while speedtest.net app shows a max speed of more than 1200mbps, it also shows a whopping 30% loss. Any idea how could that be? Maybe some device can't handle the speed and is dropping packets?

 

[Tech Junky]

Speed testing should be for speed not the rest of the gamut of things they produce results for.

Some sites don't have the bandwidth for proper tests / results or their testing isn't configured correctly. If you're not having an issue then ignore the rest of the numbers being displayed. I can get a tight grouping of test numbers using different methods of which one is a CLI based test I can run directly attached modem <> server (router) which gives me the wire speed results vs going through the network and then out. Internally I speed test by moving files around and gauge things that way.

There's a bunch of different variables that come into play with these platforms for testing. If you're hitting 1.2gbps I wouldn't worry but, if you were on DSL it would matter more.

 

[i0ntempest]

Okay thanks, I am indeed not having any issues so I'll just ignore it.

 

[MattM]

@Tech Junky , @i0ntempest

Are you aware of a method to change AX86U-WAN-LinkAggregation Mode from #4 (802.3ad) to Mode#0 (balance-rr) ?
For my modem Mode#0 is required to achive > (940Mbps) and WAN Link AGG fails if I keep the 802.3ad mode#4.

Orignal post asking about help no the matter:

AX88U WAN aggregation using Merlin firmware

Well I added a AX88U as my main router yesterday, and I have a SB8200 with the new firmware that supports LAG. I also have gigabit internet via my ISP as well. After enabling LAG on the modem, and "WAN Aggregation" on the router. It seems to work, as the router main dashboard showed WAN...

www.snbforums.com

 

[Tech Junky]

@MattM

Sounds like a limitation of the firmware being used on the device. Most consumer routers don't even use this sort of things and the ones that do can be hobbled as to the options that are embedded in them. If your router + FW only has mode 4 enabled or coded as the only option then you either need to make a feature request to the dev or hack it yourself into the FW file to enable it.

I run a DIY setup not bound by any FW for this and many other reasons. There are 2-3 different FW options though if this one doesn't work then try openwrt / freshtomato / etc. until you find one that does.

Home | Asuswrt-Merlin

www.asuswrt-merlin.net

https://openwrt.org/toh/start?dataflt[Brand*~]=asus - doesn't appear to be supported
https://wiki.freshtomato.org/doku.php/hardware_compatibility - doesn't appear to be supported

I would recommend opening your own thread w/ AX86U + balance-rr in the title and someone should bite to assist on making it work. Or reach out to @RMerlin and see if this is a hard coded limitation or something that can be added as an option.

 

[MattM]

@MattM

Sounds like a limitation of the firmware being used on the device. Most consumer routers don't even use this sort of things and the ones that do can be hobbled as to the options that are embedded in them. If your router + FW only has mode 4 enabled or coded as the only option then you either need to make a feature request to the dev or hack it yourself into the FW file to enable it.

I run a DIY setup not bound by any FW for this and many other reasons. There are 2-3 different FW options though if this one doesn't work then try openwrt / freshtomato / etc. until you find one that does.

Home | Asuswrt-Merlin

www.asuswrt-merlin.net

https://openwrt.org/toh/start?dataflt[Brand*~]=asus - doesn't appear to be supported
https://wiki.freshtomato.org/doku.php/hardware_compatibility - doesn't appear to be supported

I would recommend opening your own thread w/ AX86U + balance-rr in the title and someone should bite to assist on making it work. Or reach out to @RMerlin and see if this is a hard coded limitation or something that can be added as an option.

Thanks. I will certainly try your suggestion of opening a new thread.

My understanding of the bonding driver in the asuswrt-merlin firmware is that is it based on <bonding.ko> which has all of the modes implemented but only 802.3ad=Mode=4 was verified...
RMerlin notes on the matter: https://github.com/RMerl/asuswrt-merlin.ng/wiki/Link-Aggregation

 

[Tech Junky]

@MattM

Well, I basically use the same modules in Linux and specify the mode in the /etc/network/interfaces file.

#you need an interface / bind to bo0
auto enp10s0
iface enp10s0 inet manual
bond-master bo0

allow-hotplug  enp10s0

auto bo0
iface bo0 inet dhcp
        bond-mode 4
        bond-miimon 100
        bond-lacp-rate 1
        bond-slaves enp10s0 enp11s0

These are the basics / context of the bonding / ifenslave modules in the file.

 

[MattM]

@MattM

Well, I basically use the same modules in Linux and specify the mode in the /etc/network/interfaces file.

#you need an interface / bind to bo0
auto enp10s0
iface enp10s0 inet manual
bond-master bo0

allow-hotplug  enp10s0

auto bo0
iface bo0 inet dhcp
        bond-mode 4
        bond-miimon 100
        bond-lacp-rate 1
        bond-slaves enp10s0 enp11s0

These are the basics / context of the bonding / ifenslave modules in the file.

Thanks.

On your recommendations I have now created a dedicated thread on the subject of <WAN-LinkAggregation-Mode#0> and hopefully someone might have an idea as to whether it is feasible or not and how much effort it would take to allow this mode to run.

If it is feasible I can certainly see a benefit for all Virgin-UK users with the Asus-AX family of routers , since it would allows to max-out the available ISP bandwidth without incurring additional cost -

https://www.snbforums.com/threads/r...-for-mode-0-balance-rr-feature-request.78928/

 

[Tech Junky]

@MattM

When it comes to 1g+ speeds most jump to the SMB level or DIY something together as it's cheaper and you have more control over the traffic.

DIY is best if you want to add a 2.5GE copper or an SFP card it's cheaper than buying a router to do it typically if you have a PC laying around. Even repurposing a SFF PC into a router running Linux doesn't cost much @ $150 + NIC's.

The problem with most consumer devices is they're usually under powered and spec'd for home use under 1GE. Most of them even spec'd for 1GE can't actually hit those speeds when really pushed to get there. I had an old Asus AC router that fell on its face when I upgraded to gig and it could only hit 300-400mbps for some odd reason. I swapped it for a R7800 that had no problem hitting 950-970mbps. I got sick of the FW issues though and went DIY and combined a few devices into a single PC case in the process.

It all depends on how much control you want and how much effort you can put into it. Convenience = $$$$ / effort = le$$

Durability of a PC is a whole lot more than a plastic box with networking bits in it. Consumer products are designed to fail these days and usually just after the warranty period expires.

 

[Threska49]

Durability of a PC is a whole lot more than a plastic box with networking bits in it. Consumer products are designed to fail these days and usually just after the warranty period expires.

Usually. Got eleven years out of my N66U. Still works.

 

[Tech Junky]

Usually. Got eleven years out of my N66U. Still works.

11 years ago is a different story.

 

[Threska49]

11 years ago is a different story.

Maybe, but as long as the evidence either way is anecdotal no one will ever know.

 

[Spc]

I just tested RT-AX86U with LACP and it works, but kinda strange.

MAINPC <== 2.5G link ==>  ASUS RT-AX86U <== 2*1G LACP ==> HPE A5120-48G SI Switch <== 4*1G LACP ==> HPE A5120-48G SI Switch <== 2*1G LACP ==> HPE A5120-48G SI Switch

PC's are connected to HPE A5120-48G SI switches with LACP 2x1Gbit/s (2Gbit/s dual port intel nics).
Ping in local network while copying files is very low (CAT6 UTP with shielding):

PING Server (192.168.0.1) 56(84) bytes of data.
64 bytes from Server (192.168.0.1): icmp_seq=1 ttl=255 time=0.058 ms
64 bytes from Server (192.168.0.1): icmp_seq=2 ttl=255 time=0.060 ms
64 bytes from Server (192.168.0.1): icmp_seq=3 ttl=255 time=0.065 ms
64 bytes from Server (192.168.0.1): icmp_seq=4 ttl=255 time=0.067 ms
64 bytes from Server (192.168.0.1): icmp_seq=5 ttl=255 time=0.061 ms

If i copy file from 2 PC's on my local network to my MAIN PC that is using 2.5GbE port on RT-AX86U, transfer is correct, EXACTLY 2Gbit/s.

If i copy files from my MAIN PC to other PC's in my local network transfer is stuck at 1.0Gbit/s.

This only happens if i use ASUS routers.

If i use dual port NIC (and unplug 2.5Gbit/s NIC to 2.5GbE port to RT-AX86U) in my MAIN PC with LACP directly to switch 2.0Gbit/s it works in both ways.

I am using latest merlin, also tested on latest official asus firmware, same thing happens.


Dual port Intel NIC, LACP 2Gbit/s to switch (1.9Gbit/s send):

So the asus firmware or merlin can only receive 2Gbit/s downstream through 2.5GbE port if LACP is used but can't send at 2.5GbE out to LACP ?

What was used:
OS: Windows Server 2022, Datacenter
1.0 GbE NICs: Intel i350-T2 v2 - Dual Port
2.5 GbE NIC: ASUS 2.5G Ethernet USB Adapter (Plugged into 10Gbit/s USB 3.0)
Switch: 3x HPE A5120-48G SI
Jumbo Frames: 10240 was used, HPE A5120-48G SI supports 10K enabled on all ports.

Specs:
https://andovercg.com/datasheets/hpe-5120-si-series-switch.pdf
https://www.mouser.com/datasheet/2/612/ethernet-i350-server-adapter-brief-472432.pdf
https://www.asus.com/Networking-IoT-Servers/Wired-Networking/Wired-Adapters/USB-C2500/


2.5GbE driver is current from realtek, not old one from ASUS:

REALTEK DRIVER: 10.54.20

2022/08/24

Windows Server has all latest updates installed on all PC's
Firmware on all devices is latest.