Any way to reboot a remote router when reboot is not rebooting it?

[rjbu]

I have an RT-AC66U_B1 (384.15) router in a remote location - i.e. no one is there to power cycle it.
My devices in that location can connect to the router, but they can't get out to the internet anymore.
I cannot connect to the router thru https anymore (even if I could I doubt the reboot button would work)
but I can still ssh in and run commands.
I've tried scmerlin, and tell it to reboot the router; I've run "service reboot" (which is what scmerlin does);
I've tried "/sbin/reboot" and "/sbin/reboot -f" and nothing works.
If I look at what is in /tmp/syslog.log when I run /sbin/reboot -f I see
Mar 19 21:18:07 kernel: SysRq : Emergency Sync
but it doesn't reboot and stays up
Is there any other command to reboot the router?

In case anyone is wondering how the router gets in this state, I've seen this issue:
- The GUI System status screen shows 100% on Core 2
- top shows that it is not actual CPU usage on Core 2 (aka CPU1) but IO Wait

Mem: 211672K used, 44004K free, 908K shrd, 432K buff, 6620K cached
CPU0:  0.3% usr  0.0% sys  0.0% nic 97.0% idle  0.0% io  0.0% irq  2.5% sirq
CPU1:  0.7% usr  0.9% sys  0.0% nic  0.0% idle 97.4% io  0.0% irq  0.7% sirq
Load average: 1003.14 927.47 290.28 1/1185 23074
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
21424 21423 bob      R     1504  0.5   1  0.7 top
23004 23003 bob      S     1476  0.5   0  0.1 {scmerlin} /bin/sh /opt/bin/scmerl
23002   200 bob      S     1152  0.4   0  0.1 dropbear -p 2356 -s -a
  263     1 bob      D    23860  9.3   1  0.0 networkmap --bootwait
21431     1 bob      S     6124  2.3   1  0.0 httpd -i br0
 1278     1 bob      S     6000  2.3   1  0.0 /usr/sbin/smbd -D -s /etc/smb.conf
 1277     1 bob      S     5796  2.2   0  0.0 /usr/sbin/nmbd -D -s /etc/smb.conf
  393     1 bob      S     5640  2.2   1  0.0 amas_lib
  644   393 bob      S     5640  2.2   0  0.0 amas_lib
    1     0 bob      S     5624  2.2   1  0.0 /sbin/preinit
  194   187 bob      S     5624  2.2   0  0.0 /sbin/netool
  187     1 bob      S     5624  2.2   0  0.0 /sbin/netool
  195   194 bob      S     5624  2.2   0  0.0 /sbin/netool
  178     1 bob      S     5608  2.1   0  0.0 /sbin/wanduck
12816     1 bob      S     5604  2.1   0  0.0 bwdpi_wred_alive
  271     1 bob      S     5604  2.1   1  0.0 pctime
22926     1 bob      S     5604  2.1   0  0.0 watchdog
  270     1 bob      S     5604  2.1   0  0.0 hour_monitor
21308     1 bob      S     5604  2.1   0  0.0 usbled

- ps shows that a large number of processes show "D" state (I assume that is Disk Wait)
(I think whatever process gets in this state is just an innocent victim, not the cause)
- In the current case I see 1040 of these

  302 bob      1424 D    cp /tmp/syslog.log /tmp/syslog.log-1 /jffs

and now a number of other processes in "D" state

ps | grep D | less

    1 bob      5624 D    /sbin/preinit
   11 bob          0 DW  [sync_supers]
   36 bob          0 DW  [kworker/1:1]
   97 bob          0 DWN  [jffs2_gcd_mtd4]
  205 bob      5092 D    nt_center
  263 bob      23860 D    networkmap --bootwait

I've seen this now on 3 different RT-AC66U_B1 routers with 3 different software versions and 3 different usb sticks with 2 different file systems (once NTFS, twice ext4).
It's happens after the router has been up ~ 27 days.

 

[L&LD]

Are you able to 'Safely remove' the USB drive? A reboot may be possible then?

 

[JSinFCVA]

Are you able to 'Safely remove' the USB drive? A reboot may be possible then?

He's remote from the router. Nobody there to remove it, if I read correctly.

 

[dosborne]

Not sure if it would help, but could you "kill" the seemingly runaway processes?

 

[L&LD]

@JSinFCVA the USB doesn't need to be removed.

The ram and the processes just need to be 'released'.

 

[rjbu]

Not sure if it would help, but could you "kill" the seemingly runaway processes?

Sorry I didn't mention that I had tried that, but unfortunately the processes don't die and go away with "kill" or "kill -9"; think it's because they are in the "D" state waiting for disk io.

 

[rjbu]

Are you able to 'Safely remove' the USB drive? A reboot may be possible then?

I don't have access to the GUI to do that anymore. Is there a command that the GUI runs to do that?
I'd be interested in trying that to see if it helps, but in the command that was blocked "cp /tmp/syslog.log /tmp/syslog.log-1 /jffs"
I didn't see anything that would be using the USB drive.

 

[JDB]

I don't have access to the GUI to do that anymore. Is there a command that the GUI runs to do that?
I'd be interested in trying that to see if it helps, but in the command that was blocked "cp /tmp/syslog.log /tmp/syslog.log-1 /jffs"
I didn't see anything that would be using the USB drive.

You could try umount /tmp/mnt/<usbname>

Add -l and or -f options to help force it.


Sent from my iPhone using Tapatalk

 

[Dabombber]

You might need to rearrange the order of things in your unmount script. Looks like you've still got stuff running which references /opt, so entware and/or your swap file might not have unmounted properly. If "cat /proc/swaps" shows anything you could try "swapoff -a".

 

[CaptainSTX]

As another approach there are devices made for either automatically or from a remote location to power cycle network equipment in the case of failure, or using a schedule.

I happen to use an IPswitch which I got from 3GStore which continually pings five destinations and upon a certain number of failed pings to the five destinations power cycles both my modem and router until a connection is reestablished. It can also be set to power cycle the devices connected to its outlets. Finally it can be accessed remotely if you want force a reboot.

 

[rjbu]

As another approach there are devices made for either automatically or from a remote location to power cycle network equipment in the case of failure, or using a schedule.

I happen to use an IPswitch which I got from 3GStore which continually pings five destinations and upon a certain number of failed pings to the five destinations power cycles both my modem and router until a connection is reestablished. It can also be set to power cycle the devices connected to its outlets. Finally it can be accessed remotely if you want force a reboot.

Good idea, thanks. I used to have a similar device that I would use to reboot my HTPC when I couldn't access it remotely.

 

[rjbu]

I have an RT-AC66U_B1 (384.15) router in a remote location - i.e. no one is there to power cycle it.
My devices in that location can connect to the router, but they can't get out to the internet anymore.
I cannot connect to the router thru https anymore (even if I could I doubt the reboot button would work)
but I can still ssh in and run commands.
I've tried scmerlin, and tell it to reboot the router; I've run "service reboot" (which is what scmerlin does);
I've tried "/sbin/reboot" and "/sbin/reboot -f" and nothing works.
If I look at what is in /tmp/syslog.log when I run /sbin/reboot -f I see
Mar 19 21:18:07 kernel: SysRq : Emergency Sync
but it doesn't reboot and stays up
Is there any other command to reboot the router?

I finally noticed something in my original post that allowed me to answer my own question - SysRq
https://en.wikipedia.org/wiki/Magic_SysRq_key
specifically I was able to ssh in and reboot the router by running

echo b > /proc/sysrq-trigger

 

[Yota]

As another approach there are devices made for either automatically or from a remote location to power cycle network equipment in the case of failure, or using a schedule.

I happen to use an IPswitch which I got from 3GStore which continually pings five destinations and upon a certain number of failed pings to the five destinations power cycles both my modem and router until a connection is reestablished. It can also be set to power cycle the devices connected to its outlets. Finally it can be accessed remotely if you want force a reboot.

Just remember, that was last year, I was traveling in another country, and I used the router's openvpn server to protect my internet. when I left house, I added a smart plug to the router. so I can use my phone to control power on and off, and I think it will be good to restart the router.
The third day I was outside, I needed to restart my router. so I turned off the plug by app, and the plug and router never came online again. only then did I remember that the plug needed wifi.

I should use the routines function of the plug.

 

[CaptainSTX]

If necessary I can log into my account with my ISP and trigger a router reboot. This will in turn cause my switch to reboot both my modem and my router.

Also the switch I mentioned has an Ethernet connection to the router eliminates the possibility of WiFi connectivity issues.

The only time I have been locked out was when a GFI breaker tripped. When I got back home I changed the breaker out.