Are unmount, services-stop run at reboot?

[bigeyes0x0]

According to https://github.com/RMerl/asuswrt-merlin/wiki/User-scripts services-stop should be run when reboot, but I definitely do not see it run in my log on merlin 380.62_1 on my AC56U http://pastebin.com/DdCnMDdB (pasting the log here triggering sql injection block lol)

And having unmount script run after the kernel USB subsystem is brought down is not right, too.

Can you do something about these issues RMerlin, as I'm trying to unmount my USB HDD correctly so it doesn't have to recover everytime I reboot the router?

 

[RMerlin]

The firmware already unmounts partitions on reboot. If yours aren't cleanly unmounted, then your router must be crashing during the reboot.

 

[bigeyes0x0]

The problem is actually from line 28 down if you check the pastebin log. Seems like the kernel is already in the process of unloading various usb drivers right after closing nasapps thus causing any later command to fail including unmount custom script and where is services-stop was called, I thought it must be right after notify_rc reboot?

And if the router does crash then how should I go debug that? The log is consistent like I sent you from the reboot command till the last ext4 error due to usb subsystem is down at that time.

 

[RMerlin]

I thought it must be right after notify_rc reboot?

Could possibly be a timing issue indeed.

And if the router does crash then how should I go debug that?

The only way to really debug a kernel crash is through a serial connection.

 

[bigeyes0x0]

Could possibly be a timing issue indeed.

To make sure it's not some script of mine, I have disabled every script in /jffs/script and the same problem (kernel: EXT4-fs (sda1): recovery complete) still persists with my USB HDD which is a WD Elements 500GB. I tried a USB thumbdrive without mounting it on Windows as a drive, and it does not happen. If I do mount it then it does happen. So I guess we have some issue with timing and grace period here.

 

[bigeyes0x0]

Do you need further info on this? I can help with debugging.

 

[RMerlin]

Do you need further info on this? I can help with debugging.

No, I just need to find some time to test this, but currently busy with other things so that will have to wait.

 

[barbara]

Normally, you should not need to do this, however here's something to try.

/jffs/scripts/services-stop

# stop entware-ng
source /opt/etc/init.d/rc.unslung stop

# stop samba. release the locked files.
cd /
/sbin/service stop_samba

 

[bigeyes0x0]

@barbara: I did do this and to check if the services-stop has run I have tried to touch a file in jffs in this script and also added a logger output but alas this script wasn't run when I restart my router. It's likely a timing issue as I think this script should supposedly run right after "notify_rc reboot" in the log I included in the pastebin.

 

[Fitz Mutch]

In my custom firmware I have replaced the reboot and halt symlinks with scripts. And there's a user-defined script in JFFS that gets called automatically whenever these commands are run. So now, any weirdness that occurs during system shutdown, will happen after my user-defined shutdown script runs. Here's the replacement scripts. You must be able to re-build the firmware.

/sbin/reboot

#!/bin/sh
[ -x /jffs/home/system-shutdown ] && source /jffs/home/system-shutdown
[ -d /tmp/reboot ] && /bin/rm -rf /tmp/reboot
[ -e /tmp/reboot ] && /bin/rm -f /tmp/reboot
/bin/ln -sf /sbin/rc /tmp/reboot && /tmp/reboot

/sbin/halt

#!/bin/sh
[ -x /jffs/home/system-shutdown ] && source /jffs/home/system-shutdown
[ -d /tmp/halt ] && /bin/rm -rf /tmp/halt
[ -e /tmp/halt ] && /bin/rm -f /tmp/halt
/bin/ln -sf /sbin/rc /tmp/halt && /tmp/halt

I would have preferred to bind mount the reboot and halt commands to custom scripts. However, apparently it's not possible to bind mount over a symlink.

 

[bigeyes0x0]

Haven't tried bind mount on symlink ever but you can bind mount rc then use $0 in script to switch the logic duh . Will try this, thanks for the idea.

EDIT: just look at the layout of that directory, rc is a bunch of tools not just halt and reboot. Need to find another way. Can always extract the firmware trx but I don't want to go there if I can.

 

[Fitz Mutch]

Here's the bind mount approach for /sbin/rc. Unfortunately, I think this may cause more problems than it will solve. So I recommend my previous post.

Usage
/jffs/home/rc attach
/jffs/home/rc detach

/jffs/home/rc

#!/bin/sh
CMD=$(/usr/bin/basename $0)
SELF="/jffs/home/rc"

case $CMD in
  "rc" )
    ATTACHED=0
    /bin/mount | /bin/grep -q /sbin/rc && ATTACHED=1
    ARG1=$(/bin/echo $@ | /usr/bin/cut -f1 -d' ')
    case $ARG1 in
      "attach" )
        if [ $ATTACHED -eq 0 ]; then
          /bin/mount -o bind $SELF /sbin/rc
        else
          /bin/echo "Already attached."
          exit 1
        fi
        ;;
      "detach" )
        if [ $ATTACHED -eq 1 ]; then
          /bin/umount /sbin/rc
        else
          /bin/echo "Not attached."
          exit 1
        fi
        ;;
      * )
        /bin/echo "Bad option."
        exit 1
        ;;
    esac
    CMDLEVEL=0
    ;;
  "reboot" | "halt" )
    /bin/echo "TODO: script your system-shutdown here"
    CMDLEVEL=1
    ;;
  * )
    # all other rc commands
    CMDLEVEL=2
    ;;
esac

if [ $CMDLEVEL -gt 0 ]; then
  # run the rc command in the background
  ( /bin/umount /sbin/rc
    $0 $@
    [ $CMDLEVEL -eq 2 ] && /bin/mount -o bind $SELF /sbin/rc ) &
fi

 

[RMerlin]

The problem is actually from line 28 down if you check the pastebin log. Seems like the kernel is already in the process of unloading various usb drivers right after closing nasapps

Those modules are all related to USB modems/printers, none of them are related to USB storage. Here's the actual code in stop_usb(), which gets called early during a shutdown:

        stop_usb_program(0);

        remove_usb_modem_modules();
        remove_usb_prn_module();

First, it stops the various services, and only after that it remove those modules. Storage modules are moved a bit later.

As for services-stop, it definitely gets called, however Asus only calls this after doing a "stop_wan()" and a "stop_usb()". Once these two are done, control is passed back to init, which continues through its SIGTERM process, running stop_services() and doing other shutdown procedures.

This means that you cannot use stop_services() for anything related to USB disks - these are already unmounted at that point.

 

[RMerlin]

The actual shutdown/reboot process starts here:

https://github.com/RMerl/asuswrt-merlin/blob/master/release/src/router/rc/services.c#L4953

After that, the fallthrough lands here (with g_reboot being set to 1):

https://github.com/RMerl/asuswrt-merlin/blob/master/release/src/router/rc/init.c#L6335

From there you can see the rest of the shutdown process.

 

[bigeyes0x0]

Thanks RMerlin, I will research these and try to come up with better scripts.

 

[bigeyes0x0]

So this time I rewrite my unmount user script to do everything.
Case 1 I do it like before with a if as following:

#!/bin/sh

if [ -d $1/share ]; then
  touch /jffs/log/1.start
  [ -f /opt/etc/init.d/rc.unslung ] && /opt/etc/init.d/rc.unslung stop
  touch /jffs/log/2.unslung_stop
  sleep 1
  fuser -csk $1
  touch /jffs/log/3.kill_stray_process
  rm -f /mnt/usb
  sleep 1
  touch /jffs/log/4.done
fi

And it returns this log:

Oct 26 21:16:32 kernel: usbcore: deregistering interface driver cdc_mbim
Oct 26 21:16:32 kernel: usbcore: deregistering interface driver cdc_ncm
Oct 26 21:16:32 kernel: usbcore: deregistering interface driver qmi_wwan
Oct 26 21:16:32 kernel: usbcore: deregistering interface driver cdc_wdm
Oct 26 21:16:32 kernel: usbcore: deregistering interface driver rndis_host
Oct 26 21:16:32 kernel: usbcore: deregistering interface driver cdc_ether
Oct 26 21:16:32 kernel: usbcore: deregistering interface driver asix
Oct 26 21:16:32 kernel: cdrom: Uniform CD-ROM driver unloaded
Oct 26 21:16:32 kernel: usbcore: deregistering interface driver usblp
Oct 26 21:16:32 kernel: xhci_hcd 0000:00:0c.0: remove, state 1
Oct 26 21:16:32 kernel: usb usb1: USB disconnect, address 1
Oct 26 21:16:32 kernel: usb 1-1: USB disconnect, address 2
Oct 26 21:16:32 kernel: xhci_hcd 0000:00:0c.0: USB bus 1 deregistered
Oct 26 21:16:32 custom script: Running /jffs/scripts/unmount (args: /tmp/mnt/sda1)
Oct 26 21:16:32 kernel: EXT4-fs error (device sda1): ext4_find_entry:921: inode #2: comm [: reading directory lblock 0

And nothing was written to /jffs/log

Case 2 if I remove the if block wrap for this unmount script:

#!/bin/sh

touch /jffs/log/1.start
[ -f /opt/etc/init.d/rc.unslung ] && /opt/etc/init.d/rc.unslung stop
touch /jffs/log/2.unslung_stop
sleep 1
fuser -csk $1
touch /jffs/log/3.kill_stray_process
rm -f /mnt/usb
sleep 1
touch /jffs/log/4.done

I indeed see 4 log files under /jffs/log then again in both cases my USB HDD partition is still dirtied later. I think this is likely because for whatever reason the USB bus is actually still brought down before the actual unmount (line xhci_hcd 0000:00:0c.0: USB bus 1 deregistered perhaps, considering xhci is the usb3.0 controller). Otherwise the if condition in the first unmount script should have run without the EXT4-fs error.

If I use the unmount operation in webui, my unmount script works perfectly well. Do you have any other idea RMerlin?

 

[wocram]

Got similar phenomenon here.. When I reboot (through GUI) my router without manually un-mounting the connected drives prior the reboot, both drives on USB2- and 3 turn up red when restarted and need to be scanned.

 

[bigeyes0x0]

Got similar phenomenon here.. When I reboot (through GUI) my router without manually un-mounting the connected drives prior the reboot, both drives on USB2- and 3 turn up red when restarted and need to be scanned.

ext4 can usually handle recovery like this fine. Which filesystem do you use for your USB HDD?

 

[wocram]

ext4 can usually handle recovery like this fine. Which filesystem do you use for your USB HDD?

I have an ext2 formatted flash disk on USB2.0 and a ext4 formatted WD MyBook on USB3.0.

Will update router firmware with reset to factory default when Merlin's 380.63 (final) is available.
Initially I upgraded from stock to 380.61 without a full reset, that might be a possible cause as well.

 

[bigeyes0x0]

@wocram: You should do a factory reset now to make sure of that. You don't need to do a factory reset when .63 is out if your settings is clean from .61.