[RT-AC88U] Looking for help to investigate log messages "net_ratelimit: n callbacks suppressed"

[gburlingame]

Hello,

I noticed my system log was filled with error messages of the form:

net_ratelimit: n callbacks suppressed

Where n is a small number, typically less than 15.

After doing some of my own research, I tracked it down to three WeMo Mini smart plugs that I purchased from Amazon for the holidays. I used them with Apple HomeKit to control 3 different illuminated Christmas decorations. WeMo is a brand name for Belkin.

When I log into the router and type the command dmesg, I see a flood of messages like this:

TCP: Possible SYN flooding on port 58863. Dropping request.

The port number remains constant until I reboot the router, and then in becomes a different port number.

I then found this handy command netstat -nta | egrep "State|58863" which creates a long list of many open connections in the TIME_WAIT state. (Note: I didn't know what any of this meant until I did some research.)

tcp 0 0 192.168.0.1:58863 192.168.0.58:47275 TIME_WAIT

I experimented a bit, and if I only have 1 switch plugged in my system log remains empty. If I add a second switch, it remains empty. After adding the 3rd switch, I start to get the net_ratelimit messages.

I think perhaps the WeMo Mini's aren't quite doing something properly and theyare triggering a mechanism in the RT-AC88U intended to prevent SYN flooding? It doesn't seem to be causing any problems, at least none that I am aware of.

I wanted to share this with the community. Thank you for sharing any thoughts or ideas.

Thank you for reading.

Best regards,

Greg

 

[gburlingame]

Hello,

I noticed my system log was filled with error messages of the form:

net_ratelimit: n callbacks suppressed

Where n is a small number, typically less than 15.

After doing some of my own research, I tracked it down to three WeMo Mini smart plugs that I purchased from Amazon for the holidays. I used them with Apple HomeKit to control 3 different illuminated Christmas decorations. WeMo is a brand name for Belkin.

When I log into the router and type the command dmesg, I see a flood of messages like this:

TCP: Possible SYN flooding on port 58863. Dropping request.

The port number remains constant until I reboot the router, and then in becomes a different port number.

I then found this handy command netstat -nta | egrep "State|58863" which creates a long list of many open connections in the TIME_WAIT state. (Note: I didn't know what any of this meant until I did some research.)

tcp 0 0 192.168.0.1:58863 192.168.0.58:47275 TIME_WAIT

I experimented a bit, and if I only have 1 switch plugged in my system log remains empty. If I add a second switch, it remains empty. After adding the 3rd switch, I start to get the net_ratelimit messages.

I think perhaps the WeMo Mini's aren't quite doing something properly and theyare triggering a mechanism in the RT-AC88U intended to prevent SYN flooding? It doesn't seem to be causing any problems, at least none that I am aware of.

I wanted to share this with the community. Thank you for sharing any thoughts or ideas.

Thank you for reading.

Best regards,

Greg

So I have taken down 2 of the 3 WeMo mini smart switches as we take down our Christmas decorations.

The single WeMo mini smart switch has not caused any system log entries to appear, matching what I had found previously. It seems like the aggregate amount of these 3 switches plus a couple of other devices were overflowing a mechanism in the kernel, leading to the SYN flooding messages and suppressed callbacks. I'm curious to understand if these devices are doing something wrong, and if so what I should report to their manufacturers. Or perhaps there is a configuration problem with my network that I need to correct for?

Thanks for sharing ideas/thoughts.

Greg

 

[GeneM]

Aye folks. I am also seeing this in my RT-AC87U (Asurwrt-Merlin 384.9) today, and am trying to learn what's going on.
I do have 6 Wemo Mini plugs.

Do we have any additional info on this?
I'm suspecting the rate of these should be concerning me, right?

Tnx
Gene

 

[andyman]

On my system I traced this to a Devolo LAN power adapter. Connect the LAN power adapters and the messages start appearing and no proper connection to router or Internet, remove them and it all works fine and the messages stop.

Odd, because I've had it in exactly this config for 6 weeks or more now and it was working fine. If you only have wired connections into the AC86U unplug them one at a time and see what happens, if you have a switch attached ..same thing..just unplug each lead and see which device is causing the (I assume) loopback.

 

[gburlingame]

I'm still encountering this two years later. I recently conversed with someone on Reddit who also ran into this problem 2 years ago. He ended up dumping his Wemo's because of this.

It looks like later Merlin firmware (running 386.1 beta 1) reports the SYN flooding messages into the System Log directly now.

What I just noticed is that the Wemo's and my Verizon Fios cable boxes (made by Arris) seem to be communicating to the router over the same port -- and they open up a lot of ports that end up in the TIME_WAIT state.

Here's a little snippet -- one of those IP addresses is a WEMO, the other is a Verizon Fios cable box.

ASUSWRT-Merlin RT-AC88U 386.1_beta1 Sat Dec  5 07:55:13 UTC 2020
admin@AC88U-Garage:/tmp/home/root# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 AC88U-Garage.:57649     192.168.1.155:60817     TIME_WAIT   
tcp        0      0 AC88U-Garage.:57649     192.168.1.27:47656      TIME_WAIT   
tcp        0      0 AC88U-Garage.:57649     192.168.1.155:60816     TIME_WAIT   
tcp        0      0 AC88U-Garage.:57649     192.168.1.27:47658      TIME_WAIT   
tcp        0      0 AC88U-Garage.:www       192.168.1.17:51073      TIME_WAIT

 

[ColinTaylor]

You can check what process is associated with a port by:

netstat -lnp

 

[gburlingame]

@ColinTaylor Thank you!

The process is "miniupnpd". I'm going to disable UPnP on the RT-AC88U as an experiment.

tcp        0      0 0.0.0.0:57649           0.0.0.0:*               LISTEN      787/miniupnpd

 

[ColinTaylor]

I suspected that's what it might be. Looks like your WeMo's are spamming the UPnP server, probably trying to open up an outside connection.

You can look at System Log > Port Forwarding to see if it's forwarding any ports via UPnP.

 

[gburlingame]

@ColinTaylor You could be right -- same holds true for the Verizon Fios boxes too. I plugged in more Wemo's since turning off UPnP and so far no SYN flooding.

 

[amorak]

thanks everyone - I was having the same problem, and this thread helped me narrow down the issue to my 4 or 5 wemo devices... Any issues with disabling uPnP?

 

[gburlingame]

Hi @amorak, it's always a great feeling when you find others with the same issue!

In my case disabling UPnP did not lead to any issues.

Best of luck!