ASUS RT-AC66U All AiMesh Nodes Stopped Working

[garycnew]

ASUS RT-AC66U All AiMesh Nodes Stopped Working. Link + Data Lights on Router Ethernet Ports Still Function.

WAN (eth0), Wi-Fi (br0), and VPN (tun12) Still Working. LAN Ethernet & Wi-Fi use the same interface br0.

DHCP seems to work with Wi-Fi. DHCP Range 192.168.0.193 - 192.168.0.253. Everything Lower Than 192.168.0.193 is Statically Assigned.

Statically Assigning IP Address to Remote Ethernet Desktop Interface Not Working.

Layer-2 Seems to Work.

Layer-3 Seems to Work for WAN (eth0), Wi-Fi (br0), and VPN (tun12), but Not for LAN Ethernet.

# robocfg showports
Switch: enabled
Port 0: 1000FD enabled stp: none vlan: 2 jumbo: off mac: 00:22:8e:d1:12:00
Port 1: 1000FD enabled stp: none vlan: 1 jumbo: off mac: 00:10:3e:82:c2:93
Port 2: 1000FD enabled stp: none vlan: 1 jumbo: off mac: 00:91:82:33:04:0c
Port 3: 1000FD enabled stp: none vlan: 1 jumbo: off mac: 00:56:27:c8:56:10
Port 4: 1000FD enabled stp: none vlan: 1 jumbo: off mac: 00:10:3e:18:3a:c2
Port 5: 1000FD enabled stp: none vlan: 2 jumbo: off mac: 00:96:93:e0:af:a4
Port 7: DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00
Port 8: DOWN enabled stp: none vlan: 1 jumbo: off mac: 00:00:00:00:00:00

# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
81.75.145.20 35.148.40.1 255.255.255.255 UGH 0 0 0 eth0
10.10.0.69 * 255.255.255.255 UH 0 0 0 tun12
35.148.40.1 * 255.255.255.255 UH 0 0 0 eth0
192.168.0.0 * 255.255.255.0 U 0 0 0 br0
35.148.40.0 * 255.255.248.0 U 0 0 0 eth0
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 35.148.40.1 0.0.0.0 UG 0 0 0 eth0

# arp -a
? (192.168.0.251) at 00:38:80:a9:f1:a9 [ether] on br0
? (192.168.0.31) at <incomplete> on br0
? (192.168.0.61) at <incomplete> on br0
? (192.168.0.217) at 00:d5:bd:f7:dc:9e [ether] on br0
? (192.168.0.121) at <incomplete> on br0
? (192.168.0.141) at <incomplete> on br0
? (35.148.40.1) at 00:22:8e:d1:12:00 [ether] on eth0
? (192.168.0.225) at 00:e2:2f:c3:56:6c [ether] on br0
? (192.168.0.91) at <incomplete> on br0
? (192.168.0.21) at <incomplete> on br0
? (192.168.0.41) at <incomplete> on br0
? (192.168.0.11) at <incomplete> on br0
? (192.168.0.51) at <incomplete> on br0
? (192.168.0.244) at 00:b5:7c:46:8e:57 [ether] on br0
? (192.168.0.81) at <incomplete> on br0
? (192.168.0.71) at <incomplete> on br0
? (192.168.0.223) at 00:96:93:e0:af:a4 [ether] on br0

Of the Successful ARP Responses Only SOME IP Addresses Can Be Pinged & Traced From The Router:

? (192.168.0.251) at 00:38:80:a9:f1:a9 [ether] on br0 (Phone: SUCCESS)
? (192.168.0.217) at 00:d5:bd:f7:dc:9e [ether] on br0 (Desktop: FAILED)
? (35.148.40.1) at 00:22:8e:d1:12:00 [ether] on eth0 (WAN: SUCCESS)
? (192.168.0.225) at 00:e2:2f:c3:56:6c [ether] on br0 (RokuExpress4K: SUCCESS)
? (192.168.0.197) at 00:42:01:6c:62:cc [ether] on br0 (Unknown: SUCCESS)
? (192.168.0.244) at 00:b5:7c:46:8e:57 [ether] on br0 (RokuUltra: SUCCESS)
? (192.168.0.223) at 00:96:93:e0:af:a4 [ether] on br0 (Unknown: FAILED)

[Note: The SUCCESSES are all above the DHCP 192.168.0.193 address minimum and have WLCEVENTD wlceventd_proc_event(529): eth2: Assoc 00:42:01:6C:62:CC, status: Successful (0) entries within the syslog that suggest they are Wi-Fi related devices.]

Desktop on Wi-Fi CAN Ping & Trace Router:

C:\Users\xxx>ping 192.168.0.1

Pinging 192.168.0.1 with 32 bytes of data:
Reply from 192.168.0.1: bytes=32 time=2ms TTL=64
Reply from 192.168.0.1: bytes=32 time=1ms TTL=64
Reply from 192.168.0.1: bytes=32 time=1ms TTL=64
Reply from 192.168.0.1: bytes=32 time=1ms TTL=64

Ping statistics for 192.168.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 2ms, Average = 1ms

C:\Users\xxx>tracert 192.168.0.1

Tracing route to router [192.168.0.1]
over a maximum of 30 hops:

1 <1 ms <1 ms 1 ms router [192.168.0.1]

Trace complete.

Any Ideas as to what I may be missing?

Your Assistance is Greatly Appreciated.

 

[garycnew]

UPDATE: It appears that all the LAN Ethernet Ports on the Primary Router Work Fine directly connected to an Apple or Windows Computer. It seems the issue is with the AiMesh Nodes originally connected to those same LAN ETH Ports. It must have something to do with the AiMesh implementation.

No Clients listed in the Client List and No AiMesh Nodes listed in the AiMesh Nodes List.

Any thoughts?

Thanks, again.

 

[garycnew]

UPDATE: I believe I have found another clue...

Syslog-NG won't start:

# /opt/etc/init.d/S01syslog-ng check
 Checking syslog-ng...              dead. 
# /opt/etc/init.d/S01syslog-ng start
 Starting syslog-ng...              failed.

The syslog points to the attached external drive (/tmp/mnt/HitachiHDD):

# ls -l /tmp/ | grep -i syslog
lrwxrwxrwx    1 admin    root            21 Jan  6 10:03 syslog.log -> /opt/var/log/messages

# ls -l / | grep -i opt
lrwxrwxrwx    1 admin    root             7 Aug 14  2020 opt -> tmp/opt

# ls -l /tmp/ | grep -i opt
lrwxrwxrwx    1 admin    root            27 Jan  6 10:03 opt -> /tmp/mnt/HitachiHDD/entware
lrwxrwxrwx    1 admin    root            21 Jan  6 10:03 syslog.log -> /opt/var/log/messages

# ls -l /opt/ | grep -i var
drwxr-xr-x    7 admin    root          4096 Jan  2 04:30 var

# ls -l /opt/var/ | grep -i log
drwxr-xr-x    3 admin    root          4096 Jan  2 04:30 log
srwxrwxrwx    1 admin    root             0 Oct 24 16:19 syslog-ng.ctl
-rw-------    1 admin    root         16384 Nov 28 00:00 syslog-ng.persist
-rw-rw-rw-    1 admin    root             5 Oct 24 16:19 syslog-ng.pid

The messages file hasn't been written to since Jan 6:

# ls -l /opt/var/log/
-rw-------    1 admin    root         99470 Nov 28 00:00 logrotate.log
-rw-------    1 admin    root        262155 Apr 13  2024 logrotate.log-20240414
-rw-------    1 admin    root        371922 Jan  6 10:03 messages

The attached external drive seems to have adequate Free Space:

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                29.1M     29.1M         0 100% /
devtmpfs                124.7M         0    124.7M   0% /dev
tmpfs                   124.8M      1.9M    122.9M   2% /tmp
/dev/mtdblock4           62.8M     11.3M     51.4M  18% /jffs
/dev/sda1                28.8G     21.0G      6.4G  77% /tmp/mnt/HitachiHDD

There seem to be adequate Free Inodes:

# tune2fs -l /dev/sda1
tune2fs 1.42.13 (17-May-2015)
Filesystem volume name:   HitachiHDD
Last mounted on:          /tmp/mnt/HitachiHDD
Filesystem UUID:          b47903cb-b92a-49fa-9062-f0a327fd30bf
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1917600
Block count:              7669021
Reserved block count:     383451
Free blocks:              2055985
Free inodes:              10787
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1022
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8160
Inode blocks per group:   510
Flex block group size:    16
Filesystem created:       Sun May 15 01:19:11 2022
Last mount time:          Mon Jan  6 10:03:10 2025
Last write time:          Mon Jan  6 10:03:10 2025
Mount count:              50
Maximum mount count:      -1
Last checked:             Tue Apr  9 02:19:18 2024
Check interval:           0 (<none>)
Lifetime writes:          2341 MB
Reserved blocks uid:      0 (user admin)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:              256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      6618767b-9c8b-4654-81de-918bfe1e8026
Journal backup:           inode blocks

I am able to create a sample file in the log directory:

# touch /opt/var/log/test

# ls -l /opt/var/log/test
-rw-rw-rw-    1 admin    root             0 Jan  8 03:08 /opt/var/log/test

I'm assuming the next step is to start Syslog-NG with the debug option?

Thanks, again.


Gary

 

[JGrana]

Maybe some possible filesystem issues?
Run AMTM, dc - disk check. Enable it and reboot
Then run AMTM and do dcl. (Disk Check Log) to see if it fixed anything.

 

[Ripshod]

I agree with @JGrana's view.
Since the drive has been in use since 2022, going through however many firmware upgrades, it may even be time to reset, format and start again (or even upgrade the drive).

 

[garycnew]

@JGrana and @Ripshod

The router is in a production environment and has seen only a few firmware upgrades.

It appears that klogd works during router startup, but exits shortly thereafter along with all longing:

May  4 23:05:05 kernel: klogd started
Jan  8 16:53:05 kernel: klogd: exiting

I was able to preform a clean Check Disk without issue:

Jan  8 16:52:42 amtm disk-check: Probing 'ext4' on device /dev/sda1
Jan  8 16:52:42 amtm disk-check: Running disk check v2.9, with command 'e2fsck -p' on /dev/sda1
Jan  8 16:52:53 amtm disk-check: Disk check done on /dev/sda1
Jan  8 16:52:55 usb: USB ext4 fs at /dev/sda1 mounted on /tmp/mnt/HitachiHDD.

tune2fs shows the Filesystem state as clean:

# tune2fs -l /dev/sda1
tune2fs 1.42.13 (17-May-2015)
Filesystem volume name:   HitachiHDD
Last mounted on:          /tmp/mnt/HitachiHDD
Filesystem UUID:          b47903cb-b92a-49fa-9062-f0a327fd30bf
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file uninit_bg dir_nlink extra_isize
Filesystem flags:         unsigned_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1917600
Block count:              7669021
Reserved block count:     383451
Free blocks:              2055937
Free inodes:              10787
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1022
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8160
Inode blocks per group:   510
Flex block group size:    16
Filesystem created:       Sun May 15 01:19:11 2022
Last mount time:          Wed Jan  8 16:52:55 2025
Last write time:          Wed Jan  8 16:52:55 2025
Mount count:              51
Maximum mount count:      -1
Last checked:             Tue Apr  9 02:19:18 2024
Check interval:           0 (<none>)
Lifetime writes:          2342 MB
Reserved blocks uid:      0 (user admin)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:              256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      6618767b-9c8b-4654-81de-918bfe1e8026
Journal backup:           inode blocks

When attempting to start Syslog-NG, in verbose mode, I found that the /opt/sbin/syslog-ng file was missing:

# opkg files syslog-ng | grep -i bin
/opt/sbin/syslog-ng-debun
/opt/bin/pdbtool
/opt/bin/slogverify
/opt/bin/persist-tool
/opt/bin/update-patterndb
/opt/sbin/logread
/opt/bin/slogkey
/opt/sbin/syslog-ng-ctl
/opt/bin/loggen
/opt/bin/dqtool
/opt/sbin/syslog-ng
/opt/bin/slogencrypt

# ls -l /opt/sbin/syslog-ng
ls: /opt/sbin/syslog-ng: No such file or directory

After reinstalling Syslog-NG, the daemon started up without issue. I'll watch the logs and see whether I can find any more clues as to what's going on.

I appreciate you assistance.

Respectfully,


Gary