recent and sudden issue with loopback

[GSpock]

Hi all,
I have one OpenVPN client working OK on my RT-AX86U and protecting one of my NAS system. Until recently, I could access that NAS system without any issue via its DDNS name when connected in my LAN.

This morning, it does not work anymore ... and I cannot see what happened in between. I have tested with an external mobile connection and there it is OK.

any ideas where to look at ?

Thx,
GS

 

[octopus]

Hi all,
I have one OpenVPN client working OK on my RT-AX86U and protecting one of my NAS system. Until recently, I could access that NAS system without any issue via its DDNS name when connected in my LAN.

This morning, it does not work anymore ... and I cannot see what happened in between. I have tested with an external mobile connection and there it is OK.

any ideas where to look at ?

Thx,
GS

Did you get a CGNAT address? start with 100. look at network map.

 

[GSpock]

Did you get a CGNAT address? start with 100. look at network map.

Thanks.
No, I have a "normal" public IP. (and the NAS system gets the one of my VPN provider thru openvpn client)

 

[octopus]

Thanks.
No, I have a "normal" public IP. (and the NAS system gets the one of my VPN provider thru openvpn client)

Make sure port forwarding from vpn-provider is still there for client. (aren't you mean vpn-server ?)

 

[GSpock]

Make sure port forwarding from vpn-provider is still there for client. (aren't you mean vpn-server ?)

port fwd at VPN provider is OK otherwise external test would fail ... I am not talking about the router VPN Server here at all .... which btw is stopped.

EDIT: I start wondering if last time I did connect successfully to NAS via DDNS on LAN was before last recent upgrade to 386.7_2 .... but not keen to downgrade to test

 

[octopus]

port fwd at VPN provider is OK otherwise external test would fail ... I am not talking about the router VPN Server here at all .... which btw is stopped.

EDIT: I start wondering if last time I did connect successfully to NAS via DDNS on LAN was before last recent upgrade to 386.7_2 .... but not keen to downgrade to test

ok now you are talking about DDNS...... try to do a nslookup and see if it corresponding to right ipnummer

 

[GSpock]

I have been talking about DDNS since the OP ; Obviously the NAS DDNS (not to be confused with the router DDNS, which is something else) is pointing to the correct public IP of the VPN provider, otherwise once again the external test would fail, and that is NOT the case.
thx

NB: there is a hick-up somewhere in the routing ... because it only happens to device "protected" by openvpn client ; maybe @eibgrad could help here ... ;-)

 

[eibgrad]

Sounds as if the NAT loopback rule isn't there anymore, or perhaps has been preempted by some other matching rule. Given enough enabled features, AddOn, and perhaps even a dirty upgrade, I suppose anything is possible.

Dump the NAT table (where all the action takes place in this regard) and let's see what's there.

iptables -t nat -vnL --line-numbers

 

[GSpock]

Many thanks for your answer @eibgrad
What I just noticed is that if I stop OpenVPN client (2 in this case), I can access the NAS via its DDNS without any issues from the LAN .... therefore I have also added below the content of the route table for this client.

Please note that:
62.xxx.xx.62 is my public IP given by ISP while
185.xxx.xx.242 is the public IP of my VPN provider assigned to openvpn client 2
192.168.1.186 is the internal IP of NAS causing issue with loopback

iptables -t nat -vnL --line-numbers

Chain PREROUTING (policy ACCEPT 31857 packets, 3334K bytes)
num   pkts bytes target     prot opt in     out     source               destination        
1       30  1332 DNSVPN1    tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
2    12110  836K DNSVPN1    udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
3       30  1332 DNSVPN2    tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
4     3849  259K DNSVPN2    udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
5       30  1332 DNSVPN4    tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
6     3029  205K DNSVPN4    udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
7       30  1332 DNSVPN5    tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
8     2892  196K DNSVPN5    udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
9        0     0 DNAT       udp  --  tun12  *       0.0.0.0/0            0.0.0.0/0            udp dpt:56882 to:192.168.1.186
10     181  9564 DNAT       tcp  --  tun12  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9989 to:192.168.1.186
11       0     0 DNAT       udp  --  tun12  *       0.0.0.0/0            0.0.0.0/0            udp dpt:56881 to:192.168.1.190
12      31  1612 DNAT       tcp  --  tun12  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8879 to:192.168.1.190
13    2988  152K GAME_VSERVER  all  --  *      *       0.0.0.0/0            62.xxx.xx.62      
14    2988  152K VSERVER    all  --  *      *       0.0.0.0/0            62.xxx.xx.62      

Chain INPUT (policy ACCEPT 18520 packets, 2363K bytes)
num   pkts bytes target     prot opt in     out     source               destination        

Chain OUTPUT (policy ACCEPT 13324 packets, 1571K bytes)
num   pkts bytes target     prot opt in     out     source               destination        

Chain POSTROUTING (policy ACCEPT 11680 packets, 1218K bytes)
num   pkts bytes target     prot opt in     out     source               destination        
1     1393 91096 MASQUERADE  all  --  *      tun12   0.0.0.0/0            0.0.0.0/0          
2       40  2896 MASQUERADE  all  --  *      tun11   0.0.0.0/0            0.0.0.0/0          
3      292 19029 MASQUERADE  all  --  *      tun14   0.0.0.0/0            0.0.0.0/0          
4     3751  165K MASQUERADE  all  --  *      tun15   0.0.0.0/0            0.0.0.0/0          
5     4280  587K PUPNP      all  --  *      eth0    0.0.0.0/0            0.0.0.0/0          
6     2997  498K MASQUERADE  all  --  *      eth0   !62.xxx.xx.62         0.0.0.0/0          
7     1827  452K MASQUERADE  all  --  *      br0     192.168.1.0/24       192.168.1.0/24    

Chain DNSFILTER (0 references)
num   pkts bytes target     prot opt in     out     source               destination        

Chain DNSVPN1 (2 references)
num   pkts bytes target     prot opt in     out     source               destination        
1        0     0 DNAT       all  --  *      *       192.168.1.12         0.0.0.0/0            to:10.9.0.1
2        0     0 DNAT       all  --  *      *       192.168.1.20         0.0.0.0/0            to:10.9.0.1
3     9808  685K DNAT       all  --  *      *       192.168.1.3          0.0.0.0/0            to:10.9.0.1
4        0     0 DNAT       all  --  *      *       192.168.1.4          0.0.0.0/0            to:10.9.0.1
5        8   464 DNAT       all  --  *      *       192.168.1.188        0.0.0.0/0            to:10.9.0.1

Chain DNSVPN2 (2 references)
num   pkts bytes target     prot opt in     out     source               destination        
1      409 27262 DNAT       all  --  *      *       192.168.1.190        0.0.0.0/0            to:10.9.0.1
2      411 27398 DNAT       all  --  *      *       192.168.1.186        0.0.0.0/0            to:10.9.0.1

Chain DNSVPN4 (2 references)
num   pkts bytes target     prot opt in     out     source               destination        
1       24  1438 DNAT       all  --  *      *       192.168.1.30         0.0.0.0/0            to:10.9.0.1
2       19  1121 DNAT       all  --  *      *       192.168.1.31         0.0.0.0/0            to:10.9.0.1
3       19  1102 DNAT       all  --  *      *       192.168.1.32         0.0.0.0/0            to:10.9.0.1
4       26  1596 DNAT       all  --  *      *       192.168.1.33         0.0.0.0/0            to:10.9.0.1
5       29  1775 DNAT       all  --  *      *       192.168.1.34         0.0.0.0/0            to:10.9.0.1
6       26  1597 DNAT       all  --  *      *       192.168.1.35         0.0.0.0/0            to:10.9.0.1

Chain DNSVPN5 (2 references)
num   pkts bytes target     prot opt in     out     source               destination        
1        1    79 DNAT       all  --  *      *       192.168.1.152        0.0.0.0/0            to:10.9.0.1
2        1    79 DNAT       all  --  *      *       192.168.1.151        0.0.0.0/0            to:10.9.0.1
3        1    79 DNAT       all  --  *      *       192.168.1.150        0.0.0.0/0            to:10.9.0.1

Chain GAME_VSERVER (1 references)
num   pkts bytes target     prot opt in     out     source               destination        

Chain LOCALSRV (0 references)
num   pkts bytes target     prot opt in     out     source               destination        

Chain MAPE (0 references)
num   pkts bytes target     prot opt in     out     source               destination        

Chain PCREDIRECT (0 references)
num   pkts bytes target     prot opt in     out     source               destination        

Chain PUPNP (1 references)
num   pkts bytes target     prot opt in     out     source               destination        

Chain VSERVER (1 references)
num   pkts bytes target     prot opt in     out     source               destination        
1        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9989 to:192.168.1.186:9989
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8879 to:192.168.1.190:8879
3     2983  152K VUPNP      all  --  *      *       0.0.0.0/0            0.0.0.0/0          

Chain VUPNP (1 references)
num   pkts bytes target     prot opt in     out     source               destination

ip route show table ovpnc2

default via 10.35.0.82 dev tun12
1.0.0.1 via 62.xxx.xx.1 dev eth0 metric 1
1.1.1.1 via 62.xxx.xx.1 dev eth0 metric 1
10.31.0.5 dev tun15 proto kernel scope link src 10.31.0.6
10.33.0.82 dev tun11 proto kernel scope link src 10.33.0.81
10.35.0.1 via 10.35.0.82 dev tun12
10.35.0.82 dev tun12 proto kernel scope link src 10.35.0.81
10.37.0.5 dev tun14 proto kernel scope link src 10.37.0.6
62.xxx.xx.0/24 dev eth0 proto kernel scope link src 62.xxx.xx.62
62.xxx.xx.1 dev eth0 proto kernel scope link
127.0.0.0/8 dev lo scope link
185.xxx.xx.242 via 62.xxx.xx.1 dev eth0
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.1

 

[eibgrad]

I think I see the problem here.

The small detail you didn't mention is that these are VPN port forwards, NOT WAN port forwards.

Normally, NAT loopback is based upon the assumption that you are port forwarding over the WAN. Then when you reference the public IP of the WAN while inside the LAN for one of those port forwards, the DNAT rule redirects from the public IP and into the target's LAN ip.

Here's an example I created using the GUI.

Chain PREROUTING (policy ACCEPT 37 packets, 7404 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    9   540 GAME_VSERVER  all  --  *      *       0.0.0.0/0            192.168.63.102      
    9   540 VSERVER    all  --  *      *       0.0.0.0/0            192.168.63.102      

  Chain VSERVER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    8   480 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8443 to:192.168.1.1:8443
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9999 to:192.168.1.100
    1    60 VUPNP      all  --  *      *       0.0.0.0/0            0.0.0.0/0

Notice what triggers the redirect is the match on the public IP of the WAN (192.168.63.102) (ignore the fact it's private; it's an internal router in the lab, only meant for testing purposes). However, that's NOT the case with your VPN port forwards. Those ports forwards specifically limit access over the VPN's network interface (tun12 in this case).

Chain PREROUTING (policy ACCEPT 31857 packets, 3334K bytes)
num   pkts bytes target     prot opt in     out     source               destination        
...
9        0     0 DNAT       udp  --  tun12  *       0.0.0.0/0            0.0.0.0/0            udp dpt:56882 to:192.168.1.186
10     181  9564 DNAT       tcp  --  tun12  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9989 to:192.168.1.186
11       0     0 DNAT       udp  --  tun12  *       0.0.0.0/0            0.0.0.0/0            udp dpt:56881 to:192.168.1.190
12      31  1612 DNAT       tcp  --  tun12  *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8879 to:192.168.1.190
13    2988  152K GAME_VSERVER  all  --  *      *       0.0.0.0/0            62.xxx.xx.62      
14    2988  152K VSERVER    all  --  *      *       0.0.0.0/0            62.xxx.xx.62

That's why it's NOT working. As defined, the port forwards can NOT be triggered while inside the LAN. If they were, then presumably NAT loopback would work as expected.

If you want this to work w/ NAT loopback, you would either need to also enable port forwarding over the WAN using the GUI, or else your VPN port forwarding has to be less specific as to the input network interface, so it includes the WAN (if only implicitly), and NOT specifically the relevant VPN network interface (e.g., tun12).

Long story short, unless and until the DNAT can be triggered locally, it's just NOT going to work.

 

[eibgrad]

P.S. I suppose you could also create your VPN port forwards based on the DDNS associated w/ the NAS, but the problem there is that presumably that public IP could change over time (unless you're fortunate enough to have a static IP w/ the VPN provider). But the DNAT rules are static when it comes to the input network interface once the firewall is established, so they wouldn't recognize any change in the DDNS until the firewall was reinitialized. Perhaps if you wanted something more sophisticated, you could use ipset to handle any such dynamic changes. But that may be overkill for this particular problem.

 

[GSpock]

Thanks, now you completely lost me.
What I do not understand is that:
it used to work before and now it does not while I did not change anything. The fact that it works as soon as I stop OpenVPN Client 2 is another thing I do not understand, except that it is linked.
The fact that there is port forward from VPN provider was not important to me since it works fine when I am using an external connection, proof that it everything is OK at VPN provider side ; furthermore there are ports redirection as well in VSERVER in the case openvpn client would be stopped. So these are WAN port fwd as well !

With regard to VPN provider IP, no it is not dynamic, it is a static IP in this case. (it is the 185.xxx. IP in the ip route of openvpn client 2...)

I have read your posts twice, I will take time to read again but I am not sure what the final solution is .... BTW you talked initial about NAT rules for loopback: what do they look like ?

 

[eibgrad]

The fact that there is port forward from VPN provider was not important to me since it works fine when I am using an external connection, proof that it everything is OK at VPN provider side

Which external connection? You need to be specific. Over the WAN, VPN, both? And which DDNS are you referencing when you access it over the WAN or VPN?

The point I'm making is if you attempt to reference the DNAT that's port forwarding over the VPN, and you do so using the DDNS being updated by the NAS (I assume that's the point you were making when you made the distinction between the DDNS of the router vs. the DDNS of the NAS), that won't work. Of course, if you also have a port forward over the WAN, *and* you reference the DDNS of the WAN, it will work. But again, I'm assuming here it's the former, NOT the latter you're attempting to accomplish.

That's the problem here. I don't know *exactly* how you're trying to access these port forwards wrt NAT loopback. It's only going to work under specific circumstances. And unless you're absolutely precise in your descriptions, I have to make certain assumptions that could explain why it's NOT working. Assumptions that may in fact be wrong. That's why I made a point of the "external connection". You weren't specific!

Details matter in a case like this because we're dealing w/ multiple sources for the port forwarding, multiple VPNs, remote access possible from either the WAN or VPN, multiple DDNS, etc. We even have multiple VPNs w/ the same IP network on the tunnel. Fortunately Merlin addressed that issue w/ a fix on a prior upgrade. But it just emphasizes there's a lot going on here, and a lot of opportunity for things to go wrong.

Why did it work before? I haven't a clue. I have no information on which to draw any conclusions. All I can do at the moment is explain why it doesn't NOW.

 

[GSpock]

OK, let re-start from the begining:
1. I have a VPN provider allowing port fwd on a static IP. I have defined there port fwd on port 9989
2. I have one openvpn client (2 in this case) connecting to this VPN Provider
3. I have assigned my NAS with 192.168.1.186 IP to use that openvpn client 2
4. I have defined a port fwd on the GUI for 9989 in case I would stop openvpn client
5. My NAS has a DDNS defined at NAS provider so that it is accessible by name (let's assume it would be something like myNAS.synology.me) so that it is accessible via https://myNAS.synology.me:9989
6. so far so good, everything used to work, I mean I could access my NAS using the above link independantly of being either outside or inside my LAN
7. external test to me means: I am using my cell phone (and tablet to be complete) with 4G connection to access my NAS. This has been working without issue till the begining
8. Recently, I noticed that when using my devices (PC, tablet, cell phone) while connected in my LAN, accessing the NAS via its DDNS name i.e. https://myNAS.synology.me:9989 was not working anymore as it used to be !
9. It is not really important to have this NAt loopback issue for PC's, but it is very anoying for the cell phone/tablet because as you know then you have to change your link depending on where you are: inside or outside the LAN

Hope it clarifies and thanks for your patience
GS

PS: where do you see multiple VPN with the same IP network on the tunnel in my dumps ??

 

[eibgrad]

PS: where do you see multiple VPN with the same IP network on the tunnel in my dumps ??

I may have mispoken there. It's hard to be absolutely sure, but when I see all of the tunnels referring to 10.9.0.1 for DNS, I have to wonder. Granted, the tunnel IP networks show as 10.31.x.x., 10.33.x.x, 10.35.x.x., and 10.37.x.x., and might be unique provided they are using a netmask of 255.255.0.0, or if they are just PTP (point to point) connections. There just isn't enough information to be 100% sure.

 

[GSpock]

I may have mispoken there. It's hard to be absolutely sure, but when I see all of the tunnels referring to 10.9.0.1 for DNS, I have to wonder. Granted, the tunnel IP networks show as 10.31.x.x., 10.33.x.x, 10.35.x.x., and 10.37.x.x., and might be unique provided they are using a netmask of 255.255.0.0, or if they are just PTP (point to point) connections. There just isn't even information to be 100% sure.

No problem. (and thanks for your time !)
10.9.0.1 is indeed the "forced" DNS for all openvpn clients connecting to the same VPN provider at different localisations, and they (VPN provider) confirmed there is no conflict at all over there between the 10.3x's.

 

[eibgrad]

5. My NAS has a DDNS defined at NAS provider so that it is accessible by name (let's assume it would be something like myNAS.synology.me) so that it is accessible via https://myNAS.synology.me:9989

What public IP does myNAS.synology.me resolve to? The public IP on the WAN, or the public (static) IP of the VPN provider? That's the issue for me. I don't even really care whether you use a DDNS (domain name) in terms of resolving the problem. All I really care about is the public IP that's being referenced, regardless whether directly, or indirectly via DDNS.

 

[GSpock]

What public IP does myNAS.synology.me resolve to? The public IP on the WAN, or the public (static) IP of the VPN provider? That's the issue for me. I don't even really care whether you use a DDNS (domain name) in terms of resolving the problem. All I really care about is the public IP that's being referenced, regardless whether directly, or indirectly via DDNS.

it refers to the public static IP of the VPN provider, that is the purpose in fact !

 

[eibgrad]

it refers to the public static IP of the VPN provider, that is the purpose in fact !

Ok, that's what I had assumed. But the reference to the DDNS name rather than the public IP left some doubt.

As I said before, the redirect required by the DNAT rule for the VPN port forward does NOT reference the VPN's static IP. It only references the inbound network interface on the tunnel (tun12). Therefore, any attempt to have that redirect be triggered from inside the LAN will *fail* because it will never match the rule! Those VPN port forwards need to match on the public IP of the VPN (specifically the destination IP), NOT the tunnel's network interface, just like the WAN port forwarding does w/ the public IP on the WAN. Then the reference to the VPN's static IP from inside the LAN will match the rule, the redirect will take place, and NAT loopback will be invoked.

 

[GSpock]

If you want this to work w/ NAT loopback, you would either need to also enable port forwarding over the WAN using the GUI, or else your VPN port forwarding has to be less specific as to the input network interface, so it includes the WAN (if only implicitly), and NOT specifically the relevant VPN network interface (e.g., tun12).

Long story short, unless and until the DNAT can be triggered locally, it's just NOT going to work.

Reading back the above, I "translated" it to the following:

iptables -t nat -I PREROUTING -i br0 -p tcp --dport 9989 -j DNAT --to-destination 192.168.1.186

... and it worked !

But I am not sure this the correct way of achieving what I need. When you said the port fwd has to be less specific wrt network interface, would not it be better to have one iptable statement applied for the 2 interfaces (br0, tun12) at the same time ?

thx,
GS