Split Tunnel Weirdness

[jim99]

I set up ddns, and it registered correctly. However, I could nslookup from the external nme and get the right external address, correctly even before I did the ddns thing, I'm using registered domains and IP addresses all correctly configured, and this works on a different router.

 

[jim99]

Yet more info: I increased log sensitivity, and got this:
Oct 30 20:14:27 dnsmasq[2127]: using nameserver 149.154.159.92#53
Oct 30 20:14:27 dnsmasq[2127]: using nameserver 162.252.172.57#53
Oct 30 20:14:27 dnsmasq[2127]: using nameserver 212.23.6.100#53
Oct 30 20:14:27 dnsmasq[2127]: using nameserver 212.23.3.100#53

The last two are from my ISP and they don't resolve my doamin name correctly, just timeout. The first two resolve correctly but I don't recognise them.
Not strictly relevant, but these DNS resolvers are different from the DHCP set up, which gives clients nameservers of 8.8.8.8 and 1.1.1.1, and when I nslookup from the client dos window, I get 8.8.8.8 showing as the server its using.

 

[ColinTaylor]

You haven't explicitly answered the question we asked. Does the domain name you're using resolve to an IP address (on the PC with the problem) and is that IP address the same as the WAN IP address shown in the router's GUI?

EDIT: I had assumed you were using DDNS to resolve your router's domain name, as that is what most people here do. But looking back it appears you had already registered your name with a domain registrar. So just replace DDNS with DNS in my previous posts.

 

[jim99]

OK, I thought I'd answered your question, but in case I hadn't, I did some further testing:
If I run through my old router, and nslookup the three websites the NethServer hosts, I get the correct public IP address.
If I run through the Asus router, and nslookup the three websites the NethServer hosts, I get the correct public IP address.

If I browse to the Websites by name through my old router, I get the three sites OK, because although they are operating with the same IP address, Apache separates the requests by site name, and it all works.
If I browse to the Websites by name through my Asus router, I get timeouts on all three sites (used a new clean browser to be sure of no cache-ing) but I can, on that same browser, get to other public sites (e.g. bbc.co.uk ).

So, it is looking like the router is stripping information from the packet headers (or probably higher in the OSI stack) that allows Apache to differentiate one site from another on a particualr IP address. Does that help?

 

[ColinTaylor]

Does your public IP address returned by DNS match the WAN IP address shown in the router's GUI (Network Map > Internet status > WAN IP)?

When trying to access the sites do you get anything appearing in the Apache access log (access.log, possibly /var/log/apache2/)?

 

[jim99]

yes, Public IP = WAN IP.
Some things I've been thionking about and playing with:

1. A Cisco SPA112 VoIP controller and a HikVision NVR, on alternate ports on the router, with different internal IP addresses, work fine;
2. The default web page on the NethServer host works when addressed by IP address through the Asus router;
3. External clients, when attempting to access the virtual servers through the Asus, do so correctly, it's only internal clients, going through the Asus router that fail to access virtual hosts;
4. Internal clients, when attempting to access internal virtual hosts through an alternative router (Archer D2) do so correctly;
5. The internal clients fail to access the internal virtual servers whether or not the VPN is established (tried it both ways, checked whether I was VPN'ed or not using whatsmyip.org) so its nothing to do with the VPN or the VPN Director (is it?);
6. On examining the server logs (and indeed "follow"ing them as I made queries, which is "challenging on a busy server!) with the VPN up or down , I don't see ANY requests from the external address of the VPN, which is where they must come from if the return traffic is to go the right way;
7. On the basis of the above, it appears that the traffic from the internal client to the external address of the internal servers is being blocked....is this possible?
8. Looks like a specific block, because all the other external addresses are accessible from the internal client;
9. Put the old router back in and the incoming requests are serviced and recorded in the logs, which further strengthens my impression that the router is routing traffic from internal clients to internal hosts wrongly....bug or feature

 

[jim99]

And just to add to the fun, the above results tie in with the inability to ping from the router to the public address of the server (Network tools>Network Analysis>Ping)

 

[ColinTaylor]

And just to add to the fun, the above results tie in with the inability to ping from the router to the public address of the server (Network tools>Network Analysis>Ping)

Bear in mind that you are not pinging the web server itself (as you had alluded to previously) but just the WAN interface on the router. Responding to external pings can be enabled in the Firewall > General settings. Although that should not effect pings from the LAN side, which should always work unless you're using a VPN.

Perhaps the Asus' AiProtection thinks there's some kind of attack happening and is blocking it. Check the AiProtecion logs and try disabling it.

• On examining the server logs (and indeed "follow"ing them as I made queries, which is "challenging on a busy server!) with the VPN up or down , I don't see ANY requests from the external address of the VPN, which is where they must come from if the return traffic is to go the right way;

With all VPN clients turned completely off try accessing your sites again. If NAT loopback is working correctly you should see traffic arriving in the access.log with a source address of the router (e.g. 192.168.1.1).

 

[jim99]

1. Pings were already enabled (I use Uptime Robot to alert me if things fail, and it uses ping);
2. There was nothing in AIProtection that was switched on, and there was nothing in the logs;
3. With all VPN clients off, I can see traffic from my internal client's address arriving on the server's internal address if I target the server by IP adddress not name. If I target the server by FQDN, I see no traffic arriving at the server;
4. If I ping the server's internal address, from the router to which it is connected, I get no response. If I ping the single external address, I get a response but that is coming (I guess) from the router, not the server
5. If I ping the server's internal address from my client (the one the router couldn't see!) I get a ping response.

That last point seems to be showing the way if I could just get under the hood of the router: why is the router unable to see the server when the client can? I'm much more used to using the CLI on Cisco kit...I've enables ssh and logged in on the CLI, it looks linux but I don't know where to look

 

[ColinTaylor]

This is all very strange.

What is your router's LAN IP address and subnet mask?

Is there anything unusual about your internal network? e.g. VLANs, static routes or multiple subnets.

What type of device is connected to the Asus' WAN port? e.g. cable modem, ONT, etc.

Verify your port forwarding rules are as you expect by looking at System Log - Port Forwarding.

 

[jim99]

This is all very strange.

What is your router's LAN IP address and subnet mask?

Is there anything unusual about your internal network? e.g. VLANs or multiple subnets.

What type of device is connected to the Asus' WAN port? e.g. cable modem, ONT, etc.

Verify your port forwarding rules are as you expect by looking at System Log - Port Forwarding.

Your mastery of understatement is impressive! "Strange" is the most mild adjective I've used!

The router's LAN IP is 192.168.200.98/24, so a standard RFC1918 private address.
There is nothing particularly odd about the internal network, it's only my home network...single flat LAN, all clients (PCs, Phones, tablets) access it via Wi-Fi and I have three server-type devices, a Dell T3500 running Nethserver (Centos base), a Hik NVR, and a Cisco VoIP SPA, all wired. The external connection is PPPoE to a standard BT Openreach Fibre (FTTP) modem....I think its Huawei. The ISP is Zen.

I attached a screenshot of the Port Forwarding: the only unusual thing is the port translation on the192.168.200.210 device which is the Hik NVR: the port manipulation means I can access it externally even though I only have 1 IP address. The VoIP SPA has no issues because it phones home and establishes an outbound session through the firewall; there is no need to allow inbound session initiation.

There seems to be two things to answer:

1. "Why can't the router ping the server?" The server clearly can respond to pings, as I can ping it from the client, but when pinging from the "Network Tools" to 192.168.200.98 I get 100% loss;
2. Why is traffic from the client to the server not being seen in apache logs? When using the other non-Asus router, I see the traffic coming in as expected, but not when the Asus is in place.

I'm beginning to think the problem is something happening outbound from the router: that would explain the lack of log entries in Apache logs, and, if the ping requests never arrived, or were garbaged on arrival at the server, why the pings fail 100%. What is doesn't explain is why I can log on to the servers browser-based dashboard and look at logs as long as I use the server's IP address, not its name, though that may be something to do with it being the default server

 

[ColinTaylor]

I think your post might have a typo (or just be slightly ambiguous).

What is the router's LAN IP address - I'm guessing it's not 192.168.200.98 but something like 192.168.200.1?

So your web server is at 192.168.200.98, correct? As well as all the individual port forwards this IP address also appears to be in the DMZ. The other rules probably take precedence over the DMZ but I'd remove that just in case there's a conflict somewhere.

I'd agree that "Why can't the router ping the server?" seems to be the crucial problem.

 

[jim99]

Yes, a typo, the router is at 192.168.200.2, the server is at 192.168.200.98.
I put the Server in the DMZ as "something to try", but it made no difference either way. Its back out now.
I've attached a couple more screen shots of the port forwarding table and log, which show the server is now out of the DMZ
I'm beginning to try fairly extreme tricks now. I wondered about the NIC on the server: was it up to running 1Gbps? (the NVR wasn't, I had to drop that to 100Mbps, which is pretty poor for a newish device on a short link), so I connectd the NIC to a spare 100Mbps switch, then the switch to the router, forcing it all to 100Mbps, but that chaged nothing....came up OK at 100Mbps, but still didn't behave.

I'm beginning to wonder what to try next....do you know the developer's contact details?

 

[jim99]

further stuff: I checked my external routing: without a VPN running, the my address is xx.yy.143.80, and the default route out of the router is xx.yy.72.21, and from the router I can ping the DG, but not the internal address of the server

 

[ColinTaylor]

I'm beginning to wonder what to try next....do you know the developer's contact details?

We can tag the developer like so... @RMerlin, but it's his discretion whether he responds as he doesn't offer a personal support service, preferring to leave that to the community. Actual bug reports are another matter though. But as I can't remember anyone else ever having reported such a problem it doesn't seem like a bug.

From the router you can't ping the server's LAN IP address - Can you ping any other IP addresses on your LAN?
SSH into the router and try ping from the command prompt, i.e. ping 192.168.200.98

EDIT: Does the server have its own firewall?

 

[john9527]

From the router you can't ping the server's LAN IP address

Isn't there a Tools option to use/not use dnsmasq for the router DNS server?

 

[ColinTaylor]

Isn't there a Tools option to use/not use dnsmasq for the router DNS server?

Yes it's at Tools > Other Settings > Wan: Use local caching DNS server as system resolver (default: No)

But we're now using the IP address rather than the DNS name to eliminate that as a potential problem.... unless the GUI is interpreting 192.168.200.98 as a domain name and sending it upstream! EDIT: I've just checked this and it does not appear to be trying the resolve this as a name.

Doing the ping from the router's command prompt will hopefully avoid that as a possibility.

 

[jim99]

Some answers...and even more puzzles:
Ping from the server to the router:

"#ping 192.168.200.2
PING 192.168.200.2 (192.168.200.2) 56(84) bytes of data.
^C
--- 192.168.200.2 ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 8000ms
#"

Hence ping from server to router FAILS
and

"Phoenix@RT-AC86U-6150:/tmp/home/root# ping 192.168.200.210 - this is another server on the same network
PING 192.168.200.210 (192.168.200.210): 56 data bytes
64 bytes from 192.168.200.210: seq=0 ttl=64 time=0.727 ms
64 bytes from 192.168.200.210: seq=1 ttl=64 time=0.770 ms
64 bytes from 192.168.200.210: seq=2 ttl=64 time=0.895 ms
64 bytes from 192.168.200.210: seq=3 ttl=64 time=0.632 ms
64 bytes from 192.168.200.210: seq=4 ttl=64 time=0.707 ms
^C
--- 192.168.200.210 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.632/0.746/0.895 ms

Phoenix@RT-AC86U-6150:/tmp/home/root# ping 192.168.200.98
PING 192.168.200.98 (192.168.200.98): 56 data bytes
^C
--- 192.168.200.98 ping statistics ---
8 packets transmitted, 0 packets received, 100% packet loss

Phoenix@RT-AC86U-6150:/tmp/home/root#"

Hence ping from router to server FAILS

So the server can't ping the router, and the router can't ping the server, though the client can ping the router, and its not its not a server firewall issue because ping from the server is establishing outgoing sessions, not trying to break through the server firewall.....yes, the server does have a firewall. I need to check this out, but changing the router either fixes or breaks the connection, so its something I should be able to fix inside the router.

"From the router you can't ping the server's LAN IP address - Can you ping any other IP addresses on your LAN?" Yes, the router pings both other servers OK, see the output above, ping to ....210

and "Wan: Use local caching DNS server as system resolver (default: No)" is set to "no"


The weirdness centres around this inability of the router to ping the server or the server to ping the router, though pings from clients go straight through the router to the server, so its something about the router's inability to handle pings to/from it though it handles pings as transit traffic....once that is fixed, the router should be able to see the server for traffic other than pings and send it corrrectly, something it refuses to do now.

 

[ColinTaylor]

I'm assuming you're using the same LAN IP address range and subnet size on the Asus as you did on your previous router?

Does the server have multiple network interfaces?

Can you post the output of this command when issued from the router please:

ifconfig br0

 

[jim99]

I'm assuming you're using the same LAN IP address range and subnet size on the Asus as you did on your previous router?

Does the server have multiple network interfaces?

Can you post the output of this command when issued from the router please:

ifconfig br0

1. Yes, the new router is as near a copy of the old one as I can make it, so I can just alternate between the two routers by swapping cables and the systems just come up as before the swap.
2. Yes, the server has multiple interfaces: an outer interface 192.168.200.98 that connects to the Internet and has the ability to support clients. These clients are running insecure, on the outside of the firewall in the server. For secure ("corporate" ) clients, I run a separate WLAN, on a second interface, address 192.168.123.1/24. There is a shorewall firewall in the server that allows outgoing session establisment only from the "corporate" clients.

and here is the output of the interface command:

ASUSWRT-Merlin RT-AC86U 386.3_2 Fri Aug  6 21:48:26 UTC 2021
Phoenix@RT-AC86U-6150:/tmp/home/root# ifconfig br0
br0       Link encap:Ethernet  HWaddr 7C:10:C9:29:61:50
          inet addr:192.168.200.2  Bcast:192.168.200.255  Mask:255.255.255.0
          UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
          RX packets:687863 errors:0 dropped:104 overruns:0 frame:0
          TX packets:1008683 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:131848868 (125.7 MiB)  TX bytes:1218278622 (1.1 GiB)

Phoenix@RT-AC86U-6150:/tmp/home/root#

and here's a netstat from the server:

bastion ~]$ netstat -a|grep icmp
raw        0      0 0.0.0.0:icmp            0.0.0.0:*               7
bastion ~]$

and a netstat from the router:

Phoenix@RT-AC86U-6150:/tmp/home/root# netstat -a| grep icmp
Phoenix@RT-AC86U-6150:/tmp/home/root#

Just a random thought....the router uses 22 as its ssh port. My server uses 2222 as its ssh port. Worth considering?