webgui - "Eject USB Disk" fails

[ColinTaylor]

Having read through your recent discussion, it makes me wonder what problems might result when powering down by switching off/unplugging. (I run Diversion, Skynet and Pixelserv, with the associated USB stick in the back of the router.)
Last week I had a glitch which would not clear with reboots from the GUI, but which did clear with a power-down and 20 second wait before powering up again.

With a USB drive plugged in and running such software, should we send a halt command before manually powering down, to prevent problems? (And add a small UPS to the router to prevent problems from short-term mains outages, too, eg https://www.amazon.co.uk/dp/B0187PWCLO/?tag=smallncom-21 after ensuring voltage, current, polarity and connectors all match, of course)

Once you start using something that was originally designed as an appliance as a server these sorts of problems are inevitable to some degree. In practice there's probably very few, if any, applications that can't cope with being unceremoniously killed (i.e. having the power removed).

The only problem I'm particularly aware of is the corruption of the filesystem on the USB device. So your biggest protection against that, and to save you having to repair it later, is to use a journaling filesystem.

I can't recommend that highly enough. So that means using something like ext4 instead of ext2. Some people recommend using ext2, or ext4 without its journal, for performance reasons. To that I would say; That's OK provided that there's nothing on your USB drive that you don't mind loosing.

 

[sfx2000]

Now - if the webgui unmount actually worked for me - I'd be a happy camper ... but as a noob with the add-ons in my signature - it fails to successfully umount despite my best efforts.

If you have entware and/or other things that are working on that disk/volume - you will not be able to unmount that drive until processes are killed and the file system sync'ed up...

Even then, one can run into the unmount issue where a reboot can hang, as the kernel still have the file system open on that mount...

 

[thelonelycoder]

If you have entware and/or other things that are working on that disk/volume - you will not be able to unmount that drive until processes are killed and the file system sync'ed up...

Even then, one can run into the unmount issue where a reboot can hang, as the kernel still have the file system open on that mount...

That's a good point.
Even the guys in Redmond can't handle that gracefully, announcing that that the device could not be ejected because a file is in use by a process. Windows would not tell which process, you'd have to figure it out yourself.

 

[kernol]

Diversion all starts it in post-mount by sourcing post-mount.div, which starts all in one file:
....

I have the same scenario which seems nearly impossible. Diversion relies on Dnsmasq. To offload the logfile writing I would have to restart Dnsmasq but ignoring dnsmasq.conf.add (if found) and dnsmasq.postconf or else the files content would be added to the dnsmasq.conf again.

As for now, I keep following whatever discussion I see and hope for a good solution from someone else.

#thelonelycoder ... I note that in your app under option d - 1 it is possible to "Disable" Diversion. Having tried it I know it does not break the router's functionality ... just seems to remove Diversion activity [along with pixelserv-tls]. Dnsmasq of course continues to run and so does Entware - which leaves us with the logging problem!.

Killing Dnsmasq will I assume also break connections - so we need a method of diverting log file back to /tmp?

Leaving aside the logging to USB disk folder problem for now - would it not be possible to follow your Disable Diversion script with script commands to Disable Skynet [Option 9 in the Skynet menu].

That leaves me with only YazFi [which I don't think uses the USB after it is loaded] and also FreshJR which I know has some sort of timer to verify QOS settings periodically but not sure whether that lives in RAM or is called from USB disk.

In the meantime I will apply Colin's suggestion of enabling journaling on my ext4 USB in the hope that it prevents data loss - even at the expense of a little performance. Have you been running your attached drive in ext4 with journaling? You said you never experienced any disk corruption when dropping power - I do every time [without journaling] :-(

 

[thelonelycoder]

#thelonelycoder ... I note that in your app under option d - 1 it is possible to "Disable" Diversion. Having tried it I know it does not break the router's functionality ... just seems to remove Diversion activity [along with pixelserv-tls]. Dnsmasq of course continues to run and so does Entware - which leaves us with the logging problem!.

Killing Dnsmasq will I assume also break connections - so we need a method of diverting log file back to /tmp?

Leaving aside the logging to USB disk folder problem for now - would it not be possible to follow your Disable Diversion script with script commands to Disable Skynet [Option 9 in the Skynet menu].

That leaves me with only YazFi [which I don't think uses the USB after it is loaded] and also FreshJR which I know has some sort of timer to verify QOS settings periodically but not sure whether that lives in RAM or is called from USB disk.

In the meantime I will apply Colin's suggestion of enabling journaling on my ext4 USB in the hope that it prevents data loss - even at the expense of a little performance. Have you been running your attached drive in ext4 with journaling? You said you never experienced any disk corruption when dropping power - I do every time [without journaling] :-(

Disabling Diversion in d stops all Diversion services. That includes Dnsmasq writing to the USB device as it is Diversion that adds the log facility. If Diversion is disabled, Dnsmasq will write the much sparser output to the routers syslog.

You could add a command such as "diversion disable" to an unmount script. But then you'd have to re-enable it once the device is mounted. Diversion does not load services if it is set as disabled. The enable command is "diversion enable".
Maybe a good idea would be to have a command such as "diversion disable once". Next time the device is mounted, it would auto-enable Diversion services.

 

[thelonelycoder]

Looking at the Asuswrt-Merlin wiki, a "diversion disable once" command would really be very helpful.

unmount
Called just before unmounting a partition. Like pre-mount, this is a blocking script, so be careful with it. The script is passed the mount point as an argument which can be used in the script using $1.

I should make that a default feature in Diversion. Will look into it as soon as time allows.

 

[visortgw]

Looking at the Asuswrt-Merlin wiki, a "diversion disable once" command would really be very helpful.

I should make that a default feature in Diversion. Will look into it as soon as time allows.

Yes, it would be extremely helpful as I have found that Diversion needs to be temporarily disabled on some routers (e.g., RT-AC68U) in order to free up enough RAM to update firmware.

 

[thelonelycoder]

Yes, it would be extremely helpful as I have found that Diversion needs to be temporarily disabled on some routers (e.g., RT-AC68U) in order to free up enough RAM to update firmware.

Coded, 90% tested and coming in Diversion 4.0.7.
Diversion will write /jffs/scripts/unmount with the following content, with the swapoff line only written if swap is enabled:

#!/bin/sh
diversion unmount # Added by Diversion
swapoff -a # Added by Diversion

It can also be invoked manually:

 

[sfx2000]

Diversion will write /jffs/scripts/unmount with the following content, with the swapoff line only written if swap is enabled:

Might consider the lazy unmount option in the script...

      -l, --lazy
              Lazy  unmount.   Detach  the  filesystem from the file hierarchy
              now, and clean up all references to this filesystem as  soon  as
              it is not busy anymore.  (Requires kernel 2.4.11 or later.)

Lazy unmounts have saved me more than a couple of times...

 

[kernol]

Many thanks #thelonelycoder #ColinTaylor and #sfx2000 - looks like a really useful adaptation is in the offing, so will await Diversion 4.0.7 with much anticipation .

Now we just need the other app developers - Skynet / YazFi and FreshJR to review their offerings to determine whether they too need to add script lines to unmount so that "noobs" like me don't have to fiddle about and inadvertently make a mess of the settings.

IMHO all app developers should accept the responsibility to NOT break the functionality of the Webgui's "Eject Disk" feature.
Being able to gracefully umount USB drives - especially for firmware updates - may well result in far more stable installations of Merlin-ware .

It will also make a real difference to those of us who need to call routines from UPS devices to prepare for a power out.

 

[FreshJR]

FreshJR which I know has some sort of timer to verify QOS settings periodically but not sure whether that lives in RAM or is called from USB disk.

I don't use a USB drive for anything.

The script does not require graceful shutdowns. It runs once and is purged from memory. The daily check is done by a built in Linux task scheduler and not the script continually running.

No issues on this end.

 

[Jack Yaz]

No USB usage for YazFi, my script is only run when firewall restarts, with no lingering processes - i.e. nothing to kill before an unmount

 

[thelonelycoder]

I don't use a USB drive for anything.

No USB usage for YazFi, my script is only run when firewall restarts, with no lingering processes - i.e. nothing to kill before an unmount

So, we should almost completely be covered.
Since "diversion unmount" also runs "/opt/etc/init.d/rc.unslung stop" to stop all Entware services, @Xentrk 's Stubby will stop properly too. That is, if Diversion is installed.
Unmounting a device through the WebUI does not run /jffs/scripts/services-stop, so @Adamm will have to do his own thing for Skynet.

As a remark: In my tests when rebooting, /jffs/scripts/services-stop is consistently run before /jffs/scripts/unmount, by about two or three seconds.

 

[kernol]

Thanks to #FreshJR and #JackYaz for swift responses - and confirmation no issues with USB umount.

My grateful thanks to all for contributions so far ... a long time problem looks close to being resolved for all users of these great additions .

 

[kernol]

So, we should almost completely be covered.
Since "diversion unmount" also runs "/opt/etc/init.d/rc.unslung stop" to stop all Entware services, @Xentrk 's Stubby will stop properly too. That is, if Diversion is installed.
Unmounting a device through the WebUI does not run /jffs/scripts/services-stop, so @Adamm will have to do his own thing for Skynet.

As a remark: In my tests when rebooting, /jffs/scripts/services-stop is consistently run before /jffs/scripts/unmount, by about two or three seconds.

@Adamm should have an easy time of it because he already has a "Temp disable Skynet" routine in his menu [as you do more permanently in yours].

 

[thelonelycoder]

Thanks to #FreshJR and #JackYaz for swift responses - and confirmation no issues with USB umount.

My grateful thanks to all for contributions so far ... a long time problem looks close to being resolved for all users of these great additions .

I'm glad we finally got that covered. It's been in the back of my mind for a long time to unmount Diversion properly when using the Eject button in the WebUI.
It just needed a trigger like your post and a few ideas and facts thrown into the discussion to actually look into it.

 

[kernol]

@thelonelycoder - I'm please I pulled the trigger - and really chuffed with your responses ... Diversion has made a HUGE difference to my home setup - and I will soon be deploying it in my business ahead of the Sophos Firewall which, despite its hefty functionality ... doesn't clean up like Diversion does!

The same applies to Skynet, YazFi and FreshJR's - fully useful add-ons.

 

[Adamm]

There is no need to “unmount” Skynet, there is no constantly running process accessing any files on the USB. Everything iptables/ipset related is stored in the ram.

 

[thelonelycoder]

The only content in my unmount script is currently ...

#!/bin/sh

awk '/SwapTotal/ {if($2>0) {system("swapoff /opt/swap")} else print "Swap not mounted"}' /proc/meminfo

Thanks

Swapoff does this without the need to add the location, which I use in Diversion if a supported swapfile is found.

swapoff -a

Supported swapfiles are detected with code that would output the location with this line:

echo $(grep -o "swapon .*" /jffs/scripts/post-mount | awk '{print $2}')

Depending on that, you might have to add swapoff -a yourself after Diversion 4.0.7 release.
Or I could just grep for swapon and and add the line in /jffs/scripts/unmount, regardless if Diversion supports it.
Let me know.

@Adamm and I agreed on some standards we support for Skynet, amtm and Diversion, that includes the swapfile detection, creation and management.
I understand Skynet detects swap partitions but Diversion and amtm do not. I have no intention ATM to add this as well.

 

[kernol]

Swapoff does this without the need to add the location, which I use in Diversion if a supported swapfile is found.

swapoff -a

Supported swapfiles are detected with code that would output the location with this line:

echo $(grep -o "swapon .*" /jffs/scripts/post-mount | awk '{print $2}')

Depending on that, you might have to add swapoff -a yourself after Diversion 4.0.7 release.
Or I could just grep for swapon and and add the line in /jffs/scripts/unmount, regardless if Diversion supports it.
Let me know.

@Adamm and I agreed on some standards we support for Skynet, amtm and Diversion, that includes the swapfile detection, creation and management.
I understand Skynet detects swap partitions but Diversion and amtm do not. I have no intention ATM to add this as well.

I don't create custom swap - and suspect most [if not all] AMTM / Diversion and/or Skynet users simply create their swaps via your supplied utilities ... which means they would reside in /opt/swap.

Your call on the likelihood of some users creating swaps elsewhere. In my view best to play it "safe" and close the swap wherever it may be found?

Are you planning in 4.0.7 to do the "disable once" script so that if USB is mounted again Diversion will autorun?
I am relaxed about that - quite happy to do a reboot after returning USB to ensure everything kick starts properly.