UPnP Intermittent Failure

[Hyper_Eye]

For the last few years I have been using an RT-N66U running Merlin firmware. I never had any major issues with it. It was on 380.65_2 when last week I upgraded to an RT-AC88U. I immediately put Asuswrt-Merlin 380.65_4 on it. Overall I'm impressed with the router but I'm having UPnP issues that I did not have with the old router. I have not yet determined if downgrading the firmware or upgrading to the beta will fix it.

I first noticed the problem when my son and I tried to play Destiny on PS3 together. It wouldn't let us join the same fireteam. I ran a network test and it reported that UPnP was unavailable. It was late so I didn't mess with it any further that night. The next night I checked it again and it reported UPnP was available and we were able to play. Tonight we were once again unable to play. I set out to determine if the issue is limited to the PS3.

My main desktop is Linux so I attempted to obtain an open port list with upnpc (a test client application included with miniupnpc.) When the router is properly responding the output looks something like this:

$ upnpc -l
upnpc : miniupnpc library test client, version 2.0.
 (c) 2005-2016 Thomas Bernard.
Go to http://miniupnp.free.fr/ or http://miniupnp.tuxfamily.org/
for more information.
List of UPNP devices found on the network :
 desc: http://192.168.0.1:57512/rootDesc.xml
 st: urn:schemas-upnp-org:device:InternetGatewayDevice:1

Found valid IGD : http://192.168.0.1:57512/ctl/IPConn
Local LAN ip address : 192.168.0.69
Connection Type : IP_Routed
Status : Connected, uptime=2218s, LastConnectionError : ERROR_NONE
  Time started : Fri Apr 28 02:11:11 2017
MaxBitRateDown : 10000000 bps (10.0 Mbps)   MaxBitRateUp 10000000 bps (10.0 Mbps)
ExternalIPAddress = ************
 i protocol exPort->inAddr:inPort description remoteHost leaseTime
 0 UDP 51426->192.168.0.53:51426 'Teredo' '' 0
 1 TCP 27161->192.168.0.97:27161 '349CBAA' '' 84711
 2 TCP 27162->192.168.0.97:27162 '349CBAB' '' 84713
 3 UDP  5353->192.168.0.66:5353  'PCP MAP edc67bb90d2e6ac8d96fcd9d' '' 5764
 4 UDP  4500->192.168.0.66:4500  'PCP MAP edc67bb90d2e6ac8d96fcd9d' '' 5764
GetGenericPortMappingEntry() returned 713 (SpecifiedArrayIndexInvalid)

When I get a response like this I am able to utilize UPnP with both PS3 consoles, my Xbox One, etc. When the consoles are not able to use UPnP I am unable to query UPnP status from my desktop. The router port forwarding page displays any ports opened by UPnP and there is no indication of a problem in the logs but it is not working.

I restarted UPnP with the 'service restart_upnp' command. That did not fix the issue. I stopped UPnP on the router with the 'service stop_upnp' command and I started it back up in debug mode by running 'miniupnpd -f /etc/upnp/config -d'. When the service is working I can see the multicast SSDP M-SEARCH query come in and I see the SSDP response:

miniupnpd[1103]: SSDP M-SEARCH from 192.168.0.55:50716 ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1
miniupnpd[1103]: Single search found
miniupnpd[1103]: SendSSDPResponse(): 0 bytes to 192.168.0.55:50716 ST: HTTP/1.1 200 OK
CACHE-CONTROL: max-age=120
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1
USN: uuid:b98aba76-35fb-4898-bf6b-bf4bf7b45505::urn:schemas-upnp-org:device:InternetGatewayDevice:1
EXT:
SERVER: AsusWRT/380.65 UPnP/1.1 MiniUPnPd/2.0
LOCATION: http://192.168.0.1:57512/rootDesc.xml
OPT: "http://schemas.upnp.org/upnp/1/0/"; ns=01
01-NLS: 1493365680
BOOTID.UPNP.ORG: 1493365680
CONFIGID.UPNP.ORG: 1337

During the times where the service is not working I do not see any activity in miniupnpd. It does not indicate that a request has come in. If I query the root description URL directly it responds:

upnpc -u http://192.168.0.1:57512/rootDesc.xml -l

I can open and close ports this way even if the router is not discoverable through an SSDP search on port 1900. It would appear that the issue is in receiving the multicast UDP request. I have no idea what triggers it to stop receiving the multicast traffic or what triggers it to start working again. When it is failing to respond it fails for all devices on the network. Multicast is not failing network wide at those times because other UPnP devices are discoverable.

Netstat shows me that the multicast port 1900 is actually opened twice.

# netstat -anu | grep 1900
udp        0      0 0.0.0.0:1900            0.0.0.0:*                           
udp        0      0 0.0.0.0:1900            0.0.0.0:*

If I stop the upnp service one connection on the port will remain in netstat. I don't know what the other process is. I intend to install lsof and find out. It could be that miniupnpd fails to start during boot and it leaves a hanging descriptor. lsof should shed some light but I am curious if another descriptor is intercepting the search requests.

I will check on those things over the weekend. I will also hook up the old router and do some comparisons. Any guidance would be greatly appreciated.

 

[Hyper_Eye]

After a lot of investigation I believe this problem is the same as the one that has been causing problems with minidlna. The solution is described here and here.

I have added:

echo 0 > /sys/class/net/br0/bridge/multicast_snooping

to JFFS startup-script and problem with randomly not visible DLNA serves has gone

On stock DLNA and Debian minidlna as well.

Running Merlin, try making a
/jffs/scripts/init-start
script with the following lines

#!/bin/sh
# Turn off multicast snooping (required for minidlna advertisement)
echo 0 > /sys/class/net/br0/bridge/multicast_snooping

I am going to continue testing it but I haven't failed to receive UPnP information from the router since I made this change.

 

[matthew johnson]

Im trying to put a script in jffs is there a tutorial somewhere that helps. I have knowledge of linux/android . Im using putty to ssh.

 

[Hyper_Eye]

This should help: https://github.com/RMerl/asuswrt-merlin/wiki/User-scripts

 

[ColinTaylor]

I have knowledge of linux

Just create the file as you normally would using one of the built-in editors (vi or nano).

 

[Hyper_Eye]

Keep in mind that the script will only run after you reboot the router. You can make the change take affect immediately by simply running the command on the shell:

echo 0 > /sys/class/net/br0/bridge/multicast_snooping

The script is necessary for the setting to be applied when the router reboots.

 

[Shonk]

I have posted multiple times that upnp is broken on the rt-ac88u its only visable for a few mins after boot
its been like it since release on both stock and merlin firmware

have just added the script will post back in a few days
if it turns out this is the problem would be nice if merlin added it


edit
looking good so far its been around an hour since reboot
and upnp browser on my phone is picking it up straight away
stock it only shows up for a few mins

 

[matthew johnson]

Does the file have to be named init-start or is that a folder inside the /jffs/scripts/

 

[Vexira]

ive a had similar upnp issues, example if i change the values in adaptive qos as in update the speed, upnp wont forward the ports from my call of duty games on pc, i have to change the port range from 1024 to 1 inorder for upnp to display the port number in its list of forwared ports, so i half think there is a conflict with adaptivve qos and miniupnp, mabe a diffrent upnp daemon could be better or the mini upnp dev needs to fix it.

 

[Vexira]

al in all upnp on the 88u is inconsistant at best and seems to have issues with forwading port requests i hope it gets fixed soon.

 

[sfx2000]

I have posted multiple times that upnp is broken on the rt-ac88u its only visable for a few mins after boot
its been like it since release on both stock and merlin firmware

have just added the script will post back in a few days
if it turns out this is the problem would be nice if merlin added it

There's been some issues with miniupnpd - mem leaks, doesn't just affect AsusWRT (or other platforms)... it's pretty much an upstream problem

 

[RMerlin]

Instead of disabling multicast snooping, can you guys try this instead?

iptables -t nat -I POSTROUTING -d 224.0.0.0/4 -j RETURN

Let me know if that works.

 

[Shonk]

Instead of disabling multicast snooping, can you guys try this instead?

iptables -t nat -I POSTROUTING -d 224.0.0.0/4 -j RETURN

Let me know if that works.

Have just added it to firewall-start will keep an eye out

edit
nope 5mins 30 after boot

 

[RMerlin]

Thanks. That confirms at least the issue is different from what is affecting Red Hat servers, where they were NATing even multicast traffic. Didn't think it would be the case for Asuswrt since it only NAT traffics related to the WAN interface.

 

[Hyper_Eye]

Does the file have to be named init-start or is that a folder inside the /jffs/scripts/

Yes the file should be named init-start and it should be in the /jffs/scripts path. If it isn't there create it and add the content as quoted above. Then make it executable with the following command

chmod a+rx /jffs/scripts/init-start

The script will be executed when you reboot the router. You can verify it worked by running the following command

cat /sys/class/net/br0/bridge/multicast_snooping

It should return a 0. If it returns a 1 the script did not work.

 

[Shonk]

np.

just glad this ones found now its been driving me mad since day one of release of the rt-ac88u
i just gave up on it in the end and either rebooted the router and device needing a port forward or punched out a port manually on my pc

its working fine sofar with
echo 0 > /sys/class/net/br0/bridge/multicast_snooping

though im not sure of the exact reason behind the issue
fingers crossed you find it

 

[RMerlin]

though im not sure of the exact reason behind the issue
fingers crossed you find it

Don't hold your breath. Chances are is tied to something within Broadcom's code. And since I can't reproduce the problem here, it's not something I can investigate either. UPnP has always worked flawlessly for me.

 

[Shonk]

Don't hold your breath. Chances are is tied to something within Broadcom's code. And since I can't reproduce the problem here, it's not something I can investigate either. UPnP has always worked flawlessly for me.

Try this on your rt-ac88u maybe you just havnt noticed it
https://play.google.com/store/apps/details?id=com.dgmltn.upnpbrowser

 

[RMerlin]

3 Days uptime on my router, and it still shows up in that browser.

 

[Hyper_Eye]

Don't hold your breath. Chances are is tied to something within Broadcom's code. And since I can't reproduce the problem here, it's not something I can investigate either. UPnP has always worked flawlessly for me.

Have you tried letting upnpc run a query against it every 30 seconds or so for 20 or 30 minutes? For me it easily stopped responding in any 30 minute span of time and it lasted for a number of minutes before it starts working again. If I re-enable IGMP snooping it will happen quite quickly. Poke me on IRC if you want to remote in. You can spam it with upnpc from my server and ssh to the router from there as well.

I have optware installed with tcpdump and lsof which I used to debug the issue. In tcpdump I could see that the multicast traffic is getting to the router but it doesn't get to the listening socket.