Latency spikes on AC3100

[unrealdude24]

I've been using the latest stable Merlin firmware on my AC3100. I keep getting severe latency every few seconds and have traced it to the router itself. The latency spikes make low latency tasks almost impossible.

Its happening consistently every few seconds or so. I've tried with both QoS enabled with the device doing pings with highest possible priority in 'game mode' and without and I still get these latency spikes. The same spikes occur not just to the router but the first hop for my ISP as well. Monitoring the CPU usage via the webui shows not much usage so I don't think the bottleneck is the CPU. Reloading router doesn't seem to make a difference. I have also tested with other devices to make sure its not my PC NIC with the issue and was able to reproduce it on my cellphones over wifi. Pinging first hop of my ISP has similar level of latency spike as pinging the router.

Syslogs show nothing related to the sudden spikes. Any suggestions on how to get to the bottom of this? Anything I can look at via SSH to root cause the spikes?

I do get the following logs:
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:25 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:30 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:30 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:32 dnsmasq[382]: failed to send packet: Operation not permitted
Jan 10 19:24:32 dnsmasq[382]: failed to send packet: Operation not permitted

But I believe it might be unrelated to the latency spikes on the router as the timing of the syslogs do not match the spikes at all.
Any help is appreciated

 

[L&LD]

How are you 'reloading the router'?

 

[unrealdude24]

How are you 'reloading the router'?

Reboot through the webui. Factory reset and then configuring the same settings results in the same issue. Spikes of delayed packets makes VoIP cut out and games borderline unplayable when jitter is so high.

 

[L&LD]

You'll need to be a bit more specific.

How are you Factory resetting it? What settings are you changing past defaults?

 

[unrealdude24]

You'll need to be a bit more specific.

How are you Factory resetting it? What settings are you changing past defaults?

Just QoS as mentioned in the original post both in 'game mode' and without. I use traditional QoS with SFQ (also tried fq_codel) with the bandwidth set to 85% of my ISP advertised speed (I usually get 940 mbps).

 

[unrealdude24]

After a lot of investigation, I've found the issue I believe. I used PingPlotter to generate a graph and waited for the sudden high latency spikes and correlate it to what's happening on the router. Posting my investigation in case someone else has this issue.

Using PingPlotter, I managed to generate this little graph:

Simultaneously, I monitor throughput over the WAN using the webui:

This was all done after a factory reset and simply enabled QoS SFQ with 850 mbps bandwidth (85% of my ISP such that I never saturate it).
While the scale of the 2 graphs aren't exactly matching, from real-time observation, I noticed each spike of latency over 100ms correlates with each spike in traffic. The spikes in traffic actually come from streaming Amazon Prime/Netflix. It seems each time the streaming buffer is emptied on the streaming app, a spike of data consumption occurs before dropping down to 0 again once buffer is filled and repeats. Every time the traffic goes close to 0 and then spikes high again, the jitter is observed, sometimes going to over 600ms ping.

While the whole reason for enabling QoS was to prioritize one device over the others to reduce jitter for those devices, its having an opposite effect. Using QoS or any of the AiProtection features disables hardware acceleration which significantly reduces switching capacity on this router. The CPU on the router is simply not powerful enough to deal with these spikes in traffic.

Now the other thing is once I disable QoS, hardware acceleration does not turn on automatically. Even after a reboot, it did not turn on for me. I ended up doing a factory reset again to get hardware acceleration enabled (Cut through forwarding).

Hope this helps somebody out there that might be getting these spikes. Alternatively, you can reduce your bandwidth QoS to around 250 mbps (this is what I've tested the maximum the CPU on the AC3100 can handle without hardware acceleration) or bandwidth limit devices that spike like that to smooth out the usage since its only the sudden spikes that cause the jitter.

With hardware acceleration enabled, I no longer get the ping spikes unless I saturate my full ISP bandwidth and even then it only goes up around 8ms unlike the 100-800 ms I was receiving with QoS

 

[L&LD]

I don't know 'game mode'? But using QoS on the RT-AC3100's underpowered processors with a 1Gbps connection would be your issue.

Glad you solved it too.

 

[unrealdude24]

I don't know 'game mode'? But using QoS on the RT-AC3100's underpowered processors with a 1Gbps connection would be your issue.

Glad you solved it too.

Yeah got to the bottom of it. If you're curious about this game accelerator mode, there's not much documentation regarding this mode in terms of what configuration is being applied but you essentially pick devices you want prioritized which I don't see any difference whether its on or off compared to just setting high priority QoS. It looks like this:

Seems very gimmicky and I don't think this 'feature' really adds any value to a router, especially considering it has the opposite effect for high bandwidth cases. There aren't many routers out there that can handle QoS at gigabit speeds but if anyone has any suggestions, I'd appreciate it.

 

[shoman94]

A lot of these features are gimicky.... but at least Asus gives the options.

 

[MrSmithers0705]

After a lot of investigation, I've found the issue I believe. I used PingPlotter to generate a graph and waited for the sudden high latency spikes and correlate it to what's happening on the router. Posting my investigation in case someone else has this issue.

Using PingPlotter, I managed to generate this little graph:
View attachment 29394
Simultaneously, I monitor throughput over the WAN using the webui:
View attachment 29393
This was all done after a factory reset and simply enabled QoS SFQ with 850 mbps bandwidth (85% of my ISP such that I never saturate it).
While the scale of the 2 graphs aren't exactly matching, from real-time observation, I noticed each spike of latency over 100ms correlates with each spike in traffic. The spikes in traffic actually come from streaming Amazon Prime/Netflix. It seems each time the streaming buffer is emptied on the streaming app, a spike of data consumption occurs before dropping down to 0 again once buffer is filled and repeats. Every time the traffic goes close to 0 and then spikes high again, the jitter is observed, sometimes going to over 600ms ping.

While the whole reason for enabling QoS was to prioritize one device over the others to reduce jitter for those devices, its having an opposite effect. Using QoS or any of the AiProtection features disables hardware acceleration which significantly reduces switching capacity on this router. The CPU on the router is simply not powerful enough to deal with these spikes in traffic.

Now the other thing is once I disable QoS, hardware acceleration does not turn on automatically. Even after a reboot, it did not turn on for me. I ended up doing a factory reset again to get hardware acceleration enabled (Cut through forwarding).

Hope this helps somebody out there that might be getting these spikes. Alternatively, you can reduce your bandwidth QoS to around 250 mbps (this is what I've tested the maximum the CPU on the AC3100 can handle without hardware acceleration) or bandwidth limit devices that spike like that to smooth out the usage since its only the sudden spikes that cause the jitter.

With hardware acceleration enabled, I no longer get the ping spikes unless I saturate my full ISP bandwidth and even then it only goes up around 8ms unlike the 100-800 ms I was receiving with QoS

thank you for the graphs this is exactly the same problem I'm having for the same reason. I'm curious to know what you do have adjusted for your settings now that you have figured out QoS is not real on the 3100s?

 

[unrealdude24]

thank you for the graphs this is exactly the same problem I'm having for the same reason. I'm curious to know what you do have adjusted for your settings now that you have figured out QoS is not real on the 3100s?

Apologies for the incredibly late response. You can disable QoS and make sure hardware acceleration is enabled. If you insist on using QoS on this router, you can set it to maximum download bandwidth to 250 mbps in the QoS configuration and you wont get the spikes.