Occasional hang

[Tim G]

Symptoms are:

• Lost WAN connection (LAN ok)
  
• Core-2 pinned at 100% load (Core-1 normal)
• Memory usage normal (30%), no errors in syslog
• Web interface still responding, but not possible to make SSH connection

Setup is RT-AC68U with two USB disks shared via Samba only. The hang has happened with Merlin firmwares 378.53_0 and 378.54_2.

Without SSH access I don't know how to debug this further. Any thoughts?
Many thanks, -Tim

 

[chamberc]

Yep, I have experienced this too... normally during the day when I'm at work and can't access it because the WAN connection is dead...

 

[RMerlin]

Symptoms are:

• Lost WAN connection (LAN ok)
  
• Core-2 pinned at 100% load (Core-1 normal)
• Memory usage normal (30%), no errors in syslog
• Web interface still responding, but not possible to make SSH connection

Setup is RT-AC68U with two USB disks shared via Samba only. The hang has happened with Merlin firmwares 378.53_0 and 378.54_2.

Without SSH access I don't know how to debug this further. Any thoughts?
Many thanks, -Tim

You will need SSH access to determine what process is crashed and taking 100% of one CPU core.

 

[Tim G]

You will need SSH access to determine what process is crashed and taking 100% of one CPU core.

Any idea what could take out the SSH daemon and the WAN at the same time? I assume that even with only one CPU core available, all non-crashed processes should remain responsive. I suppose I can try to write a logging script that stores to disk.

 

[john9527]

Just to double check....since you have gui access, anything in the syslog?

 

[Tim G]

Nope, only DHCP requests. The SSH connection attempt showed up too, but I couldn't actually get a prompt.

 

[john9527]

How are you trying access.....wired or wireless...windows or Linux or mac? Is it the same from multiple clients?

 

[Tim G]

Wired Debian. Didn't try other clients before rebooting the router. It only happens ~once a week, so difficult to investigate. Both crashes occurred whilst connected to my LAN from outside via SSH. I have two external ports forwarded to the same internal port on the same machine - can this cause problems?

 

[chamberc]

It always occurs while I am accessing the router from work..

 

[RMerlin]

Any idea what could take out the SSH daemon and the WAN at the same time? I assume that even with only one CPU core available, all non-crashed processes should remain responsive. I suppose I can try to write a logging script that stores to disk.

Seriously high CPU/IO load that might prevent the sshd daemon from properly opening and establishing a connections perhaps.

Try opening a SSH connection, and leave it open on your computer. With any luck, by the time it hangs again, you will still be able to check with "top" what is loading the CPU.

 

[Tim G]

After I last posted I set up automatic logging of 'top' to disk to try to find the cause of the problem. Since then my router has run for almost 2 weeks without problems, until today when it appears to have crashed and rebooted several times.

The 'top' logs don't seem to show anything suspicious:
https://drive.google.com/file/d/0B9_DGx19AWmoYnJxdjVibUo2T2s/view?usp=sharing

I'm not really sufficiently knowledgeable to read the system logs, but it seems very odd that there should be multiple reboots:
https://drive.google.com/file/d/0B9_DGx19AWmob09qc1lDcGowNnM/view?usp=sharing

Still running 378.54_2 for now.