[RESOLVED] Asuswrt-Merlin 384.19 AiMesh Ethernet Backhaul Connectivity Issues

[garycnew]

All:

***I know AiMesh 2.0 was recently released. I'd like to find a short-term solution to my AiMesh 1.0 issue, while evaluating AiMesh 2.0's viability and potential upgrade path.***

I've had a ton of problems with Asuswrt-Merlin 384.19 AiMesh and nodes reporting that they're intermittently Offline in the GUI. I have 4 x Asus RT-AC66U B1 routers. The main RT-AC66U B1 is in Wireless router mode / AiMesh Router mode (Default) with the remaining 3 x RT-AC66U B1 routers in AiMesh Node mode connected via ethernet. To troubleshoot, I disabled the 3 x RT-AC66U B1 routers' 2.4ghz & 5ghz radios in the nvram settings and rebooted them. I successfully confirmed that only ethernet connections were able to connect to the 3 x RT-AC66U B1 routers. Then, I setup cronjobs on each of the 3 x RT-AC66U B1 routers in AiMesh Node mode to preform a ping test to 8.8.8.8 every 5 minutes. I continued to observe the 3 x RT-AC66U B1 routers in AiMesh Node mode intermittently go into Offline status.

To test, I manually ran a ping test to 8.8.8.8 from the offending AiMesh Node to find packet loss:

admin@RT-AC66U_B1-C293:/tmp/home/root# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
--- 8.8.8.8 ping statistics ---
22 packets transmitted, 0 packets received, 100% packet loss

I am able to successfully ping the LAN interface of the main RT-AC66U B1 in Wireless router mode from the offending AiMesh Node:

admin@RT-AC66U_B1-C293:/tmp/home/root# ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: seq=0 ttl=64 time=0.556 ms
64 bytes from 192.168.0.1: seq=1 ttl=64 time=0.376 ms
64 bytes from 192.168.0.1: seq=2 ttl=64 time=21.053 ms
--- 192.168.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.376/7.328/21.053 ms

I can even successfully ping the WAN interface of the main RT-AC66U B1 in Wireless router mode from the offending AiMesh Node:

admin@RT-AC66U_B1-C293:/tmp/home/root# ping ###.91.60.78
PING ###.91.60.78 (###.91.60.78): 56 data bytes
64 bytes from ###.91.60.78: seq=0 ttl=64 time=0.698 ms
64 bytes from ###.91.60.78: seq=1 ttl=64 time=0.375 ms
--- ###.91.60.78 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.375/0.536/0.698 ms

However, I am not able to ping the egress next hop from the WAN interface of the main RT-AC66U B1 in Wireless router mode from the offending AiMesh Node:

admin@RT-AC66U_B1-C293:/tmp/home/root# ping ###.91.60.1
PING ###.91.60.1 (###.91.60.1): 56 data bytes
--- ###.91.60.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

At the same time, I am able to successfully ping test to 8.8.8.8 from the main RT-AC66U B1 in Wireless router mode:

admin@main-wap01:/tmp/home/root/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=118 time=32.075 ms
64 bytes from 8.8.8.8: seq=1 ttl=118 time=32.153 ms
64 bytes from 8.8.8.8: seq=2 ttl=118 time=32.005 ms
64 bytes from 8.8.8.8: seq=3 ttl=118 time=32.000 ms
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 32.000/32.058/32.153 ms

I am also able to successfully ping test to the offending AiMesh Node from the main RT-AC66U B1 in Wireless router mode:

admin@main-wap01:/tmp/home/root/# ping 192.168.0.11
PING 192.168.0.11 (192.168.0.11): 56 data bytes
64 bytes from 192.168.0.11: seq=0 ttl=64 time=0.541 ms
64 bytes from 192.168.0.11: seq=1 ttl=64 time=0.363 ms
64 bytes from 192.168.0.11: seq=2 ttl=64 time=0.376 ms
64 bytes from 192.168.0.11: seq=3 ttl=64 time=0.441 ms
64 bytes from 192.168.0.11: seq=4 ttl=64 time=0.430 ms
--- 192.168.0.11 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.363/0.430/0.541 ms

When the Offline event occurs, I have tried to service restart_wan on the offending AiMesh Node without success. I would have thought that the restart_wan would at least temporarily resolve the issue.

I have noticed that if I run a traceroute in UDP or ICMP mode, it will occasionally bring the offending AiMesh Node back online.

Any thoughts as to what might be causing the issue and whether there is a short-term solution, while evaluating AiMesh 2.0's viability and potential upgrade path?

Thank you for your assistance.

Kind Regards,


Gary

 

[bbunge]

I have no issues with AC66U_B1 main router with AC68U nodes on Asus 386 Beta firmware. One node wired the other wireless.

 

[garycnew]

All:

I've preform a few manual traceroutes and tcpdumps from the offending AiMesh Node and the main RT-AC66U B1 in Wireless router mode.

Unsuccessful ICMP traceroute from the offending AiMesh Node:

admin@RT-AC66U_B1-C293:/tmp/home/root# traceroute -I 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 38 byte packets
1 192.168.0.1 (192.168.0.1) 2.465 ms 0.356 ms 0.284 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *

Unsuccessful tcpdump of the ICMP traceroute (above) from the main RT-AC66U B1 in Wireless router mode:

admin@main-wap01:/tmp/home/root/# tcpdump -en -i any 'host 8.8.8.8 and not host ###.91.60.78 and icmp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
02:13:55.352749 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 1, length 18
02:14:10.364008 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 2, length 18
02:14:10.364638 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 3, length 18
02:14:10.365484 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 4, length 18
02:14:15.367814 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 5, length 18
02:14:20.372960 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 6, length 18
02:14:25.376312 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 7, length 18
02:14:30.377696 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 8, length 18
02:14:35.378597 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 9, length 18
02:14:40.379737 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 47803, seq 10, length 18
10 packets captured
154 packets received by filter
49 packets dropped by kernel

Successful ICMP traceroute from the main RT-AC66U B1 in Wireless router mode at the same time the offending AiMesh Node fails:

admin@main-wap01:/tmp/home/root/# traceroute -I 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 38 byte packets
1 * * *
2 * * *
3 162.252.180.54 (162.252.180.54) 1.828 ms 1.962 ms 1.761 ms
4 pr02-ae12.sea09.net.google.com (206.81.80.69) 31.965 ms 36.013 ms 31.689 ms
5 74.125.243.177 (74.125.243.177) 32.572 ms 32.837 ms 32.588 ms
6 142.251.50.175 (142.251.50.175) 31.882 ms 31.845 ms 31.863 ms
7 dns.google (8.8.8.8) 34.151 ms 31.851 ms 31.744 ms

Successful tcpdump of the ICMP traceroute (above) from the main RT-AC66U B1 in Wireless router mode at the same time the offending AiMesh Node fails:

admin@main-wap01:/tmp/home/root/# tcpdump -en -i any 'host 8.8.8.8 and not host 192.168.0.11 and icmp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
02:21:06.223937 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: ###.91.60.78 > 8.8.8.8: ICMP echo request, id 62756, seq 0, length 40
02:21:06.255904 In 3c:8c:93:77:5e:60 ethertype IPv4 (0x0800), length 76: 8.8.8.8 > ###.91.60.78: ICMP echo reply, id 62756, seq 0, length 40
02:21:16.536010 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: ###.91.60.78 > 8.8.8.8: ICMP echo request, id 7205, seq 0, length 40
02:21:16.567806 In 3c:8c:93:77:5e:60 ethertype IPv4 (0x0800), length 76: 8.8.8.8 > ###.91.60.78: ICMP echo reply, id 7205, seq 0, length 40
02:21:26.900850 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: ###.91.60.78 > 8.8.8.8: ICMP echo request, id 11045, seq 0, length 40
02:21:26.932604 In 3c:8c:93:77:5e:60 ethertype IPv4 (0x0800), length 76: 8.8.8.8 > ###.91.60.78: ICMP echo reply, id 11045, seq 0, length 40
02:21:42.480411 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: ###.91.60.78 > 8.8.8.8: ICMP echo request, id 21029, seq 0, length 40
02:21:42.512183 In 3c:8c:93:77:5e:60 ethertype IPv4 (0x0800), length 76: 8.8.8.8 > ###.91.60.78: ICMP echo reply, id 21029, seq 0, length 40
02:21:47.633693 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: ###.91.60.78 > 8.8.8.8: ICMP echo request, id 22821, seq 0, length 40
02:21:47.665469 In 3c:8c:93:77:5e:60 ethertype IPv4 (0x0800), length 76: 8.8.8.8 > ###.91.60.78: ICMP echo reply, id 22821, seq 0, length 40
10 packets captured
16 packets received by filter
0 packets dropped by kernel

Echo Replies finally are successful in ICMP traceroute from the offending AiMesh Node:

admin@RT-AC66U_B1-C293:/tmp/home/root# traceroute -I 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 38 byte packets
1 192.168.0.1 (192.168.0.1) 0.364 ms 0.356 ms 0.274 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * 142.251.50.175 (142.251.50.175) 32.558 ms 32.462 ms
8 dns.google (8.8.8.8) 32.383 ms 32.371 ms 32.364 ms

Successful tcpdump of the ICMP traceroute (above) from the main RT-AC66U B1 in Wireless router mode:

admin@main-wap01:/tmp/home/root/# tcpdump -en -i any 'host 8.8.8.8 and not host ###.91.60.78 and icmp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
02:25:32.021904 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 1, length 18
02:25:47.035805 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 2, length 18
02:25:47.036279 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 3, length 18
02:25:47.036663 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 4, length 18
02:25:52.040627 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 5, length 18
02:25:57.045647 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 6, length 18
02:26:02.050899 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 7, length 18
02:26:07.055672 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 8, length 18
02:26:12.060720 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 9, length 18
02:26:17.065723 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 10, length 18
...
02:27:07.098669 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 20, length 18
02:27:07.098669 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 20, length 18
02:27:07.504761 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 21, length 18
02:27:07.504761 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 21, length 18
02:27:07.537357 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 22, length 18
02:27:07.537357 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 22, length 18
02:27:07.569476 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: 8.8.8.8 > 192.168.0.11: ICMP echo reply, id 62702, seq 22, length 18
02:27:07.569536 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: 8.8.8.8 > 192.168.0.11: ICMP echo reply, id 62702, seq 22, length 18
02:27:07.571377 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 23, length 18
02:27:07.571377 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 23, length 18
02:27:07.603468 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: 8.8.8.8 > 192.168.0.11: ICMP echo reply, id 62702, seq 23, length 18
02:27:07.603517 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: 8.8.8.8 > 192.168.0.11: ICMP echo reply, id 62702, seq 23, length 18
02:27:07.603883 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 24, length 18
02:27:07.603883 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11 > 8.8.8.8: ICMP echo request, id 62702, seq 24, length 18
02:27:07.635975 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: 8.8.8.8 > 192.168.0.11: ICMP echo reply, id 62702, seq 24, length 18
02:27:07.636016 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: 8.8.8.8 > 192.168.0.11: ICMP echo reply, id 62702, seq 24, length 18
35 packets captured
35 packets received by filter
0 packets dropped by kernel

I've reviewed the iptables on the main RT-AC66U B1 in Wireless router mode and don't see any DROPS associated with ICMP.

@RMerlin How does the Asuswrt code monitor/detect an AiMesh Node is Offline? Any idea why ICMP Replies would intermittently be lost on the Backhaul and not the main RT-AC66U B1 in Wireless router mode? Suggestions for a short-term workaround? Thank you for your GREAT work!

Respectfully,


Gary

 

[RMerlin]

@RMerlin How does the Asuswrt code monitor/detect an AiMesh Node is Offline?

No idea. Closed source code.

 

[garycnew]

No idea. Closed source code.

@RMerlin I was afraid of that. This feels like a NAT over Backhaul issue to me. The issue does not resolve by restarting the WAN interface on the offending AiMesh Node. However, the issue is resolved if the offending AiMesh Node is rebooted. Any suggestions on how to further diagnose/resolve? Thank you for your reply.

 

[garycnew]

@RMerlin I've confirm that it is indeed a NAT issue. To test, I rebooted the offending AiMesh Node verifying it was back in an Online state.

I SUCCESSFULLY ran a UDP traceroute test to 1.1.1.1 from the intermittently offending AiMesh Node:

[Note: I changed the traceroute test from ICMP to UDP to ensure it wasn't protocol related.]

admin@RT-AC66U_B1-C293:/tmp/home/root# /usr/bin/time /usr/bin/traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 10 hops max, 38 byte packets
1 gnutech-wap01 (192.168.0.1) 0.391 ms 0.341 ms 0.280 ms
2 * * *
3 * * *
4 162.252.180.54 (162.252.180.54) 3.215 ms 2.799 ms 2.646 ms
5 six.as13335.com (206.81.81.10) 23.753 ms 25.200 ms 22.589 ms
6 one.one.one.one (1.1.1.1) 22.281 ms 21.770 ms 21.875 ms
real 0m 30.24s
user 0m 0.01s
sys 0m 0.00s

The following is a SUCCESSFUL tcpdump of the traceroute (above) from the main RT-AC66U B1 in Wireless router mode:

[Note: The related UDP packets can be seen connecting from the offending AiMesh Node and the main RT-AC66U B1 in Wireless router's WAN address.]

[SUCCESSFUL NAT]

admin@main-wap01:/tmp/home/root/# tcpdump -en -i any 'host 1.1.1.1 and udp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
04:22:39.886156 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33435: UDP, length 10
04:22:39.886156 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33435: UDP, length 10
04:22:39.888154 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33436: UDP, length 10
04:22:39.888154 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33436: UDP, length 10
04:22:39.888596 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33437: UDP, length 10
04:22:39.888596 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33437: UDP, length 10
04:22:39.888996 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33438: UDP, length 10
04:22:39.888996 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33438: UDP, length 10
04:22:39.889309 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33438: UDP, length 10
04:22:44.894182 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33439: UDP, length 10
04:22:44.894182 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33439: UDP, length 10
04:22:44.894533 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33439: UDP, length 10
04:22:49.894638 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33440: UDP, length 10
04:22:49.894638 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33440: UDP, length 10
04:22:49.894972 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33440: UDP, length 10
04:22:54.899801 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33441: UDP, length 10
04:22:54.899801 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33441: UDP, length 10
04:22:54.900131 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33441: UDP, length 10
04:22:59.904654 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33442: UDP, length 10
04:22:59.904654 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33442: UDP, length 10
04:22:59.904961 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33442: UDP, length 10
04:23:04.906162 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33443: UDP, length 10
04:23:04.906162 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33443: UDP, length 10
04:23:04.906614 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33443: UDP, length 10
04:23:09.910748 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33444: UDP, length 10
04:23:09.910748 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33444: UDP, length 10
04:23:09.911078 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33444: UDP, length 10
04:23:09.971203 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33445: UDP, length 10
04:23:09.971203 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33445: UDP, length 10
04:23:09.971597 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33445: UDP, length 10
04:23:09.974183 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33446: UDP, length 10
04:23:09.974183 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33446: UDP, length 10
04:23:09.974503 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33446: UDP, length 10
04:23:09.977067 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33447: UDP, length 10
04:23:09.977067 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33447: UDP, length 10
04:23:09.977381 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33447: UDP, length 10
04:23:10.004148 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33448: UDP, length 10
04:23:10.004148 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.36492 > 1.1.1.1.33448: UDP, length 10
04:23:10.004487 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 54: ###.91.60.78.36492 > 1.1.1.1.33448: UDP, length 10
51 packets captured
51 packets received by filter
0 packets dropped by kernel

The following is a SUCCESSFUL netstat-nat of the traceroute (above) from the main RT-AC66U B1 in Wireless router mode:

[SUCCESSFUL NAT]

admin@main-wap01:/tmp/home/root/# netstat-nat -n | grep 1.1.1.1
udp 192.168.0.11:36492 1.1.1.1:33451 UNREPLIED
udp 192.168.0.11:36492 1.1.1.1:33451 UNREPLIED
udp 192.168.0.11:36492 1.1.1.1:33452 UNREPLIED
udp 192.168.0.11:36492 1.1.1.1:33448 UNREPLIED
udp 192.168.0.11:36492 1.1.1.1:33449 UNREPLIED

Here is a FAILED UDP traceroute test to 1.1.1.1 from the intermittently offending AiMesh Node:

[FAILED NAT]

admin@RT-AC66U_B1-C293:/tmp/home/root# /usr/bin/time /usr/bin/traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 38 byte packets
1 192.168.0.1 (192.168.0.1) 4.446 ms 0.368 ms 0.283 ms
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
...
28 * * *
29 * * *
30 * * *
real 7m 30.69s
user 0m 0.01s
sys 0m 0.00s

The following is a FAILED tcpdump of the traceroute (above) from the main RT-AC66U B1 in Wireless router mode:

[Note: The related UDP packets can ONLY be seen connecting from the offending AiMesh Node and NOT the main RT-AC66U B1 in Wireless router's WAN address.]

[FAILED NAT]

admin@main-wap01:/tmp/home/root/# tcpdump -en -i any 'host 1.1.1.1 and udp'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
04:32:02.350596 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33435: UDP, length 10
04:32:17.366245 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33436: UDP, length 10
04:32:17.366756 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33437: UDP, length 10
04:32:17.367177 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33438: UDP, length 10
04:32:22.372353 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33439: UDP, length 10
...
04:39:07.676902 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33520: UDP, length 10
04:39:12.677387 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33521: UDP, length 10
04:39:17.682637 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33522: UDP, length 10
04:39:22.687006 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33523: UDP, length 10
04:39:27.687735 In 94:10:3e:82:c2:93 ethertype IPv4 (0x0800), length 62: 192.168.0.11.53975 > 1.1.1.1.33524: UDP, length 10
90 packets captured
238 packets received by filter
104 packets dropped by kernel

The following is a FAILED netstat-nat (NO RELATED NAT CONNECTIONS) of the traceroute (above) from the main RT-AC66U B1 in Wireless router mode:

[FAILED NAT]

admin@main-wap01:/tmp/home/root/# netstat-nat -n | grep 1.1.1.1
admin@main-wap01:/tmp/home/root/#

Is there an easy way to manually restart the NAT service without having to reboot the offending AiMesh Node each time?

Any other short-term solutions you might suggest?

Thanks, again.

Respectfully,


Gary

 

[RMerlin]

I don't know. I don't use AiMesh, and I don't devote any development time on it since it's entirely out of my control, and I have no idea how it's actually implemented. The first thing you should do is upgrade to an up to date firmware, as Asus made a ton of changes to the AiMesh implementation in 386.

 

[garycnew]

I don't know. I don't use AiMesh, and I don't devote any development time on it since it's entirely out of my control, and I have no idea how it's actually implemented. The first thing you should do is upgrade to an up to date firmware, as Asus made a ton of changes to the AiMesh implementation in 386.

My apologies. I'm still trying to sort out what portions of the Asuswrt code are open and closed to your improvements.

Thank you, again, for your GREAT work.

Respectfully,


Gary

 

[garycnew]

All:

I had a switch in-between the main RT-AC66U B1 in Wireless router mode and the offending AiMesh Node. When I removed the switch and directly connected the offending AiMesh Node to the main RT-AC66U B1 in Wireless router mode the issue seemed to resolve itself.

I'm still not entirely sure what caused the problem, but it appears to be resolved for now.

Respectfully,


Gary

 

[Datalink]

In the development for one of the latest 386 versions, there were problems noted with switches in the network. This appears to have been fixed, as indicated in the following thread:

Beta - ASUSWRT 386 RC2 public beta with full functions AiMesh 2.0 (continued)

Update on 2021/1/29 This version changed the DNSmasq version to 2.84 to fix CVE-2020-25681, CVE-2020-25682, CVE-2020-25683, CVE-2020-25687, CVE-2020-25684, CVE-2020-25685, CVE-2020-25686 aka DNSpooq https://drive.google.com/drive/folders/1TW9JXxWpM-qmgGv2BixibkACVki-9g1Q?usp=sharing The share...

www.snbforums.com

Quoted from that page:

"Update 2020/11/12 (9.0.0.4.386.40947)
https://drive.google.com/drive/folders/1UHZwR1buG-nei8fne9d9OktOS2gZy6Hs?usp=sharing

This version includes 19 models:
ZenWiFi: XT8(RT-AX95Q), XD4(RT-AX56U_XD4), CD6(RT-AC59_CD6)
GT series: GT-AX11000,GT-AC5300, GT-AC2900
RT series: RT-AX88U, RT-AX92U, RT-AX86U, RT-AX82U, RT-AX58U, TUF-AX3000, RT-AX56U, RT-AX55
RT series: RT-AC5300, RT-AC88U, RT-AC3100, RT-AC86U, RT-AC68U

Change log

1. Fixed Ethernet backhaul through switch issue
2. RT-AX86 and GT-AX11000 support 2.5G port as ethernet backhaul
3. Fixed RT-AX55 firmware update issue.
4. Fixed speedtest issue for android app.
5. Fixed mesh node guest network internet connection issue.
6. Fixed ap mode guest network issue.
7. Added security patch
8. Fixed repeater mode UI bug."

that was for development version 9.0.0.4.386.40947.

The latest official release for the RT-AC68U is a beta version,
Version 9.0.0.4.386.41994 Beta Version dated 2021/02/01

So the beta release on Asus's support page is behind the beta version shown in this forum. In any event, this should hopefully point out that the issue you have with a switch in the network has been fixed in the later version of AiMesh. Now all that has to happen is for Asus to release a final version for the 68U that includes the development code in version 9.0.0.4.386.40947.

 

[garycnew]

@Datalink

I appreciate the reference. It does sound like the same issue. I wish I had access to the bug report for that particular changelog entry.

Asus should probably mention, in their setup guide, that AiMesh 1.0 Nodes must be directly connected to the Main Router.

Thank you for your reply.

Respectfully,


Gary