AX88U WAN aggregation using Merlin firmware

[lilstone87]

Well I added a AX88U as my main router yesterday, and I have a SB8200 with the new firmware that supports LAG. I also have gigabit internet via my ISP as well. After enabling LAG on the modem, and "WAN Aggregation" on the router. It seems to work, as the router main dashboard showed WAN Aggregation 2 Gbps. However I ran into an issue, and it's very annoying to say the least.

The router system log is being spammed with "kernel: net_ratelimit: *** callbacks suppressed" errors. I checked many different things, tried enabling/disabling LAG modem side, which every time you change it on the modem, the modem reboots. I checked the connection rates of both cable's from the router to modem, both had gigabit connection rates. The reason I know it's related to the "WAN Aggregation" feature, I decided to disconnect one of the cable's, with the feature still being enabled both modem, and router side. The router log shows one of the cable's being disconnected, after that, the spam of this error stops.

However as we all know, this feature requires two cables being connected for the feature to work. Currently I will likely have to disable this feature, until ASUS fixes this problem. Because there's no way I can handle seeing the router system log spammed filled with these errors. I will attach a screenshot I took of the system log last night, includes a 5-8 minute time period. Starting off with both cables being connected, you will see the error repeating itself often. When the rate of the callbacks were higher, I was actually running a speedtest. So the more data I push, the more callbacks being suppressed. You will see a log message when I disconnected one of the cables. I actually waited 3-4 minutes after disconnecting one of the cables, before I took the screenshot. As you will see, the errors completely stopped.

So there's zero doubt the cause of the issue. Now it's trying to figure out how to make ASUS aware of this issue, and get it fixed in a future firmware release. Because currently this is to annoying to keep enabled, because of it flooding the system log.

 

[lilstone87]

Well I decided to make the router system log, log everything, and I plugged LAN port 4 back into the modem, to re-enable this feature.
This is what I found besides the "kernel: net_ratelimit: *** callbacks suppressed" error.

May 5 12:18:37 kernel: eth1 (Ext switch port: 0) (Logical Port: 8) (phyId: 8) Link UP at 1000 mbps full duplex
May 5 12:19:35 syslogd started: BusyBox v1.25.1
May 5 12:19:35 kernel: klogd started: BusyBox v1.25.1 (2019-05-02 16:02:57 EDT)
May 5 12:19:39 kernel: net_ratelimit: 481 callbacks suppressed
May 5 12:19:39 kernel: protocol 0800 is buggy, dev eth1 <-- 10 times, every time the callbacks error is logged.

@RMerlin Any ideal the cause of this last error? Do you think it's just the "WAN Aggregation" feature bugged right now?

 

[miggystyle]

I am thinking of running a similar setup as you @lilstone87
Exept that my modem is the motorola Motorola MB8600. Have not yet purchased my AX88U, but am going to anyways whether or not this works. However I am curious to see if I would be future proofed if my ISP would offer me higher than 1Gbps speeds. I already get 948Mb/s downloads over wired connection, so once I have the AX88U and just adding an additional cat6 cable would boost my speed closer to 1Gbps then I will do it IF its a stable connection.

Another bit of information is I am planning on running the manufacturer provided firmware on the AX88U, as well.

 

[mbze430]

I have a AX88U and about to get a DOCSIS 3.1 modem (SB8200) and was looking at the LAG/802.3ad. Has these issues been resolved?

I wonder if the SB8200 uses IEEE 802.1AX or IEEE 802.3ad? because in the AX88U documentation it only mention of IEEE 802.3ad which is the older spec of LAG

 

[coolbeans2016]

After seeing this post, looking for help on OpenDNS problem, checked my logs & it only showed one incident of a 2 line item listing this "buggy"..... when going back though there was a large block of them... still monitoring as I post this.

 

[coolbeans2016]

Usually rolls along with only a single or double listing for quite a bit. Of course until seeing this I have not been looking.
I caught a post on Perfect-Privacy.com community looking for why I cannot get OpenVPN working, saying IPv6 doesn't work (I know it does) in Merlin's firmware....
Does WAN aggregation prevent OpenVPN from working also?
Thanks, ahead of time....

 

[mbze430]

I ended up getting a Motorola MB8600. and I am also having the "kernel: protocol 0800 is buggy, dev eth1" spam from enabling WAN Aggregation.

What is more. If you enabled WAN and LAN Aggregation, all my wired devices will not show up in the Network Map list.

 

[Stevie]

Interesting thread, I myself was looking at bonding 2x wan fibre connections. I can only get 40/10 speeds as was thinking of doubling them up.

If it’s too costly, then it won’t be worth it. I’ve got an ac86u, which means I’ll have to get a ax88u Plus a LAG switch and to pay for 2nd fibre line.

what’s the cheapest LAG Switch to get nowadays.

seeing you still struggling to get it actually working, I will wait on this.

have anyone tried pfsense and will it not be a cheaper option though and what other cheaper methods are out there to WAN aggregate.

 

[mbze430]

@RMerlin Is this something we the user have to get ASUS to look into or something you can help getting ASUS's attention on fixing??
with the whole "kernel: protocol 0800 is buggy, dev eth1" spam

 

[RMerlin]

with the whole "kernel: protocol 0800 is buggy, dev eth1" spam

It typically means you had unexpected traffic. For instance, could be IPv6 traffic being sent on a network that isn't supposed to receive IPv6 traffic. Or, receiving jumbo packets on an interface only configured to handle 1500 bytes.

So, it's unlikely to be a bug and more likely to be a configuration issue.

 

[mbze430]

None of my switches or end points have jumbo packets enabled. Nothing in the MB8600 that enables Jumbo frame either. I do have IPv6 enabled on the AX88U. I will try disabling IPv6 and see I still get the error message.

Unless there is a problem with the firmware from Spectrum for Ipv6 I don't see how anything is out of the ordinary when WAN Aggregation is enabled

 

[mbze430]

It typically means you had unexpected traffic. For instance, could be IPv6 traffic being sent on a network that isn't supposed to receive IPv6 traffic. Or, receiving jumbo packets on an interface only configured to handle 1500 bytes.

So, it's unlikely to be a bug and more likely to be a configuration issue.

Just tried it and disabled IPv6. Same results.

Also I don't know if you came across my other post. But when I have WAN Aggregation and LAN Aggregation is on. Wired machines are completely missing from the network map and list

 

[RMerlin]

On paper, I should be able to plug 2 WAN cables from the AX88U into the ECS2110-26T and each interface would get it's own public IP and 1G of WAN bandwidth.

That part of your reasoning is flawed BTW. That is not how WAN aggregation works... Link aggregation means that both interfaces are tied together as a single one, and gets only one single IP. Having a modem with two ports does not imply that you get two separate connections for the price of one. Your ISP must be providing you a service faster than 1 GBps, in which case the modem will need to have both interfaces bridged together, and only then you can also have your router use WAN aggregation.

 

[princi]

10 GBps fibre from the ISP straight into the ECS2110-26T

It’s the ECS2110-26T that handles the link aggregation in my case, not the ISP. You’re correct about the single IP, though.

Anyway, as soon as the AX89U is available, I will just plug the SFP+ straight into the router and do it that way.

Alas, no Merlin support.

 

[walter carboni]

My ISP is upgrading the internet connection from 1Gbps to 2.5 Gbps, fro free! They say customer don't have to do anything, but I changed the provided modem as it is really bad....

To be able to use another router they provide a media converter, called ONT, which I connected to the WAN port of my AX88U and everything is working fine.
As the ONT only has 1 ethernet port, I was thinking to use a switch LACP capable to connect WAN and port 4 from AX88U, then the only port provided by ONT, but I am not sure this configuration will work and give me any benefit, any suggestion on this topic?
ISP authentication is mac address based, static IP assigned.
thanks for any support, I also use a second AX88U in my office obviously with merlin FW, really great work!

 

[MattM]

@RMerlin : Is there a possibility to change the <WAN-Link Aggregation> mode from #4(802.3ad) to #0(balance-rr) on an RT-AX86U (f/w : Merlin 386.5) ?

My ISP (Virgin UK) provides 1.2-Gbps but I cannot achieve the maximum speed without also introducing WAN-Link Aggregation Mode#0 <balance-rr> as the modem has no 2.5Gbe port and even if it had (new Virgin SuperHub 5) would not help me without sacrificing my setup.

The modem itself is a Virgin SuperHub 4 (SH4) , locked fw , and based on <ARRIS TG3492LG-VMB>

There are useful information describing how for example the introduction of a Mikrotik linux-derived RouterOS products (RB4011 , CRS3005) can create a bridge where the WAN side is set to <balance-rr> mode#0 and the speed can then reach the full delivered bandwidth of 1.2Gbps:

From: https://www.cableforum.uk/board/sho...5f0af64fbd342da11fe19&p=36058944&postcount=84
"The superhub has an undocumented feature, in that the LAN ports in modem mode are in a balance-rr group. This means that with a switch or router that supports load balancing, you can connect as many of the ports as you like in one group. As this is balance-rr, it is per packet, and even a single download stream is split across the nics."


Since the Merlin firmware allows tinkering with many things I had imagined that with the AX86U offical WAN-Link Aggregation being possible between WAN(eth0)+LAN4(eth1) ports, it likely be possible to change the Mode of operation from #4 to #0 to as a minimum try it out....

However, my crude attempts to simply take down the <bond1>, followed by a change of Mode to #0 has not been a success as the change is not accepted at the step where one requests Mode#0 - I think it might relate to some sort of permisisons issues ?

I was trying to follow more or less the steps decribed in this port:
https://www.thegeekdiary.com/how-to-change-the-network-bonding-mode-in-centos-rhel-6/

Let me know if you possibly would have some ideas as to how to test WAN-Link_aggregation in <balance-rr>Mode#0 .

The following post suggests that one could look into nvram editing to change bonding parameters : nvram bonding changes
There is indeed in my nvram settings a parameter called <bondst=4> which could indicate which of the <bonding.ko> mode to use (ref. https://github.com/RMerl/asuswrt-merlin.ng/wiki/Link-Aggregation)?

Maybe just changing the nvram parameter <bondst> to "bondst=0" would be enough to force WAN-LinkAgg to operate in Mode#0 ??

Just tried it but unfortunately it seems that the "nvram commit" instruction following the "nvram set bondst=0" is always forcing back <bonst=4> as can be checked with "nvram get bonst"...
Without the "nvram commit" the parameter changes '0' and can be checked with "nvram get bondst"

Any ideas ?

Thank you.

 

[RMerlin]

@RMerlin : Is there a possibility to change the <WAN-Link Aggregation> mode from #4(802.3ad) to #0(balance-rr) on an RT-AX86U (f/w : Merlin 386.5) ?

This is beyond the scope of this project, you'd have to ask Asus.

 

[MattM]

This is beyond the scope of this project, you'd have to ask Asus.

Noted. Thanks.


ASUS product support contacted and got the following reply, (not surprising):

"
Thank you for contacting ASUS Support.

Sadly this cannot be implemented due to product hardware design.

It already has 2.5 Gigabit WAN port therefore the limitation comes from the ISP equipment not really the ASUS unit.

Our apologies for any inconveniences caused to you."

 

[stoli412]

Noted. Thanks.


ASUS product support contacted and got the following reply, (not surprising):

"
Thank you for contacting ASUS Support.

Sadly this cannot be implemented due to product hardware design.

It already has 2.5 Gigabit WAN port therefore the limitation comes from the ISP equipment not really the ASUS unit.

Our apologies for any inconveniences caused to you."

I too am on Virgin Media with a Hub 4 and came searching for this very same answer. Very disappointing to hear Asus won't implement this. I have an AX88U, which doesn't have a 2.5Gb port, so even when the Hub 5 comes available I won't be able to get >1Gb speeds.

Is it really a hardware limitation on the Asus routers that's the issue? It seems like it's only software since link aggregation is already possible.

If only VM didn't hand out such terrible kit....we'd be ace if they at least supported 802.3ad like many other cable modems do.

 

[MattM]

I too am on Virgin Media with a Hub 4 and came searching for this very same answer. Very disappointing to hear Asus won't implement this. I have an AX88U, which doesn't have a 2.5Gb port, so even when the Hub 5 comes available I won't be able to get >1Gb speeds.

Is it really a hardware limitation on the Asus routers that's the issue? It seems like it's only software since link aggregation is already possible.

If only VM didn't hand out such terrible kit....we'd be ace if they at least supported 802.3ad like many other cable modems do.

I looked into the matter further after getting the <asuswrt-merlin.ng> repo git'ed out.
https://github.com/RMerl/asuswrt-merlin.ng/wiki/Compile-Firmware-from-source

I do not believe for a moment there is any real HW limitations -

I seem to remember the section of source code where the WAN bonding default mentionned 802.3ad and it could be changed.

"mode=802.3ad" and "mode=4" set the same mode.

#if defined(RTCONFIG_BONDING_WAN) && defined(RTCONFIG_QCA)
                if (!strncmp(ifr.ifr_name, "bond", 4)) {
                    ether_etoa((unsigned char *) ifr.ifr_hwaddr.sa_data, eabuf);
                    set_bonding_iface_hwaddr(ifr.ifr_name, eabuf);
                } else {
#endif

However , I am not competent enough to build from source with the WAN Agg mode changed from Mode#4 (802.3ad as above) Mode#0=<balance-rr>
Too bad.

At the same time there are certain features that "self-compiled-builds" will not support : all TRENDNet features is not supported under a self-build
You only get those binaries implemented if someone trusted by ASUS like @RMerlin does the compilation for us....

I got over it...but it is a shame.