AX86U PPPoE MTU problem/questions

[i0ntempest]

Hi all,
So I've been trying to optimize my WAN MTU. I'm with Bell Fibe 1.5G and using HH4000 PPPoE passthrough, using merlin 386.7_2. From what I heard Bell should support baby jumbo frames and should allow 1500 MTU. But I noticed that if I set MTU and MRU to 1500 in WAN settings, I get

start_wan_if(): Error setting MTU on eth0 to 1500

in the system log, though eth0 is already at 1500 MTU as I can see from ifconfig. The MTU of ppp0 though, stays at 1492, but I can set 1500 with ifconfig and verify it works with

ping 1.1.1.1 -M do -s 1472

.
I'm not sure if this is the correct behavior, or there's some sort of bug. Would appreciate some explanation.

 

[drinkingbird]

Hi all,
So I've been trying to optimize my WAN MTU. I'm with Bell Fibe 1.5G and using HH4000 PPPoE passthrough, using merlin 386.7_2. From what I heard Bell should support baby jumbo frames and should allow 1500 MTU. But I noticed that if I set MTU and MRU to 1500 in WAN settings, I get

start_wan_if(): Error setting MTU on eth0 to 1500

in the system log, though eth0 is already at 1500 MTU as I can see from ifconfig. The MTU of ppp0 though, stays at 1492, but I can set 1500 with ifconfig and verify it works with

ping 1.1.1.1 -M do -s 1472

.
I'm not sure if this is the correct behavior, or there's some sort of bug. Would appreciate some explanation.

Edited to organize my jumble of thoughts.

When you set the Asus to PPPoE, the WAN is set to 1500 MTU, the MTU and MRU it lets you change are for the PPPoE tunnel virtual interface. PPPoE requires 8 bytes overhead so the most you can set these to is 1492. My guess is the Asus is not RFC 4638 aware ("baby jumbo") so when you set it to 1500 it gets confused and throws an error, it can't set the WAN interface to 1500 and expect to handle 1500 on the PPPoE interface.

If your ISP supports baby jumbo / 1508 byte, you can go into the CLI on the router and use ifconfig to set the eth0 to 1508 bytes then try setting your MTU and MRU to 1500 in the GUI, see what happens. To make that permanent (if it works) you probably will need a script, or may be able to save it to NVRAM somewhere.

In some cases you might need 1512 byte to avoid fragmentation but that's usually only if the ISP has DSL or ATM in the path, that is unlikely with a 1.5G setup.

To test if that is really your optimal setup, you need to send pings of 1500 bytes with "DF" (Do not Fragment) bit set to see if it is actually working.

If that doesn't work, set the WAN back to 1500, PPPoE MTU/MRU to 1492 and do a ping test with 1492 and DF.

There are some cases where setting MTU/MRU to 1452 might yield slightly better performance, as it doesn't require padding to be added to every packet. Unlikely to make much difference these days though.

So I guess I'd see if you can get 1500 byte MTU working (via 1508 physical MTU) without fragmentation and see how your speeds look, that should be the best performance.

If not, then try 1500/1492, assuming no fragmentation, test your speed
Then try 1500/1452, test fragmentation to be sure but definitely should not be any there, then test your speed.

 

[i0ntempest]

Edited to organize my jumble of thoughts.

When you set the Asus to PPPoE, the WAN is set to 1500 MTU, the MTU and MRU it lets you change are for the PPPoE tunnel virtual interface. PPPoE requires 8 bytes overhead so the most you can set these to is 1492. My guess is the Asus is not RFC 4638 aware ("baby jumbo") so when you set it to 1500 it gets confused and throws an error, it can't set the WAN interface to 1500 and expect to handle 1500 on the PPPoE interface.

If your ISP supports baby jumbo / 1508 byte, you can go into the CLI on the router and use ifconfig to set the eth0 to 1508 bytes then try setting your MTU and MRU to 1500 in the GUI, see what happens. To make that permanent (if it works) you probably will need a script, or may be able to save it to NVRAM somewhere.

In some cases you might need 1512 byte to avoid fragmentation but that's usually only if the ISP has DSL or ATM in the path, that is unlikely with a 1.5G setup.

To test if that is really your optimal setup, you need to send pings of 1500 bytes with "DF" (Do not Fragment) bit set to see if it is actually working.

If that doesn't work, set the WAN back to 1500, PPPoE MTU/MRU to 1492 and do a ping test with 1492 and DF.

There are some cases where setting MTU/MRU to 1452 might yield slightly better performance, as it doesn't require padding to be added to every packet. Unlikely to make much difference these days though.

So I guess I'd see if you can get 1500 byte MTU working (via 1508 physical MTU) without fragmentation and see how your speeds look, that should be the best performance.

If not, then try 1500/1492, assuming no fragmentation, test your speed
Then try 1500/1452, test fragmentation to be sure but definitely should not be any there, then test your speed.

Thanks for the detailed explaination. If I set 1508 MTU on eth0 I get "SIOCSIFMTU: Invalid argument". Probably a hardware limitation? I do have jumbo frame enabled under switch control tab of the router webui, but I doubt it's doing anything since all the eth* interfaces still have 1500 MTU and can't be increased any further.

 

[drinkingbird]

Thanks for the detailed explaination. If I set 1508 MTU on eth0 I get "SIOCSIFMTU: Invalid argument". Probably a hardware limitation? I do have jumbo frame enabled under switch control tab of the router webui, but I doubt it's doing anything since all the eth* interfaces still have 1500 MTU and can't be increased any further.

One interesting thing from your original post, when you set the MTU and MRU on PPPoE to 1500, you were able to send 1472 byte packets to 1.1.1.1 with no fragment successfully? And ETH0 was reporting 1500 MTU? That would seem to imply it is working and the WAN is sending 1508 bytes, which would be odd if the eth0 interface is showing 1500. Can you confirm you got a successful ping response when you had it set that way? You should not be able to send 1472 bytes over PPPoE without baby jumbo support and DF set. If you can, either the ETH interface is setting itself to 1508 and not reflecting in ifconfig (very unlikely) or something in the HND code is intercepting your packets and reducing the size. I'm not aware of any standard function that would do that with ICMP packets though. Either that or something was disregarding/stripping your DF bit and fragmenting the packets. Would have to install tcpdump and look at the actual packets to confirm.

From looking through a few threads and sites, people have gotten it working (1508 byte MTU with 1500 byte on PPPoE) on the 6x series like mine, but the 8x apparently does not support jumbo frame on the WAN, so that explains why you can't change it to 1508. But doesn't explain why your pings are successful when set to 1500 on the PPPoE interface.

Normal setup on the Asus (or most any router that doesn't natively support 1508) for PPPoE over fiber is 1492 with 1500 on the physical interface. The Asus router as far as I know automatically enables MSS clamping when you use PPPoE so every TCP SYN you send, it inserts MSS of 1452 (the actual payload of a 1492 byte frame). That way the remote host knows to limit its MTU/MSS on the response. Originally this was supposed to be handled by PMTUD but that never works end to end across the internet, most firewalls block that ICMP packet and it fails, so MSS clamping was developed and is common now. Since both the SYN and SYN ACK get clamped and the MSS inserted, both machines know the maximum payload and never attempt to send a packet that is too large, so PMTUD will never even come into the picture if everything works right.

In reality, if 1492 in the GUI gets you full throughput and there are no issues with some sites not working etc, you should be all set. The extra 8 bytes isn't a big deal as long as MSS clamping is working (if it wasn't, everyone on PPPoE would be having problems). TCP packets should never have to be fragmented due to MSS clamping, ICMP and UDP may fragment, but not a big deal as not much uses those. The majority of TCP packets have DF set on them so can't be fragmented, while ICMP and UDP usually allow it by default from my experience.

 

[i0ntempest]

One interesting thing from your original post, when you set the MTU and MRU on PPPoE to 1500, you were able to send 1472 byte packets to 1.1.1.1 with no fragment successfully? And ETH0 was reporting 1500 MTU?

About that, actually... I did successfully ping 1.1.1.1 with DF but some websites are broken under that configuration. For example ipv6-test.com reports no IPv4 connectivity, but IPv6 is fine because I'm using route48 6in4. So I don't know, it seems to be not fully working. I'll test further with 6in4 tunnel disabled.

 

[drinkingbird]

About that, actually... I did successfully ping 1.1.1.1 with DF but some websites are broken under that configuration. For example ipv6-test.com reports no IPv4 connectivity, but IPv6 is fine because I'm using route48 6in4. So I don't know, it seems to be not fully working. I'll test further with 6in4 tunnel disabled.

Yeah it really doesn't make sense that it would work.