In Entware, why is rc.unslung called from services-start?

[latenitetech]

First, my congratulations on the superb implementation of Entware and integration into AsusWRT-Merlin. With just a few hooks at the firmware level, it truly is magic how well it all works together to greatly extend the functionally of an already fine router.

One thing I just can't get my head wrapped around though, why is the call to 'rc.unslung start' placed in services-start rather than in post-mount? Services-start is called long before the entware USB volume is mounted, and so it's forced to just camp out in a loop waiting for the entware filesystem to become available. I've had to extend that wait loop several times because it kept timing out as I was adding more "stuff" to my config.

It finally dawned on me that if I just put the call to 'rc.unslung start' after swapon in post-mount, then it doesn’t matter how long the boot takes, entware will always still start. One-liner change and problem solved.

A couple of other observations:
- with swapon being placed in post-mount, and rc.unslung in services-start, there is a bit of a race condition. I was seeing my entware services started before swap was activated. I'd prefer to get swap in place before I start services that might depend on the swap being there. Moving rc.unslung to after swapon in post-mount addresses that.

- I've recently started doing an automatic fsck on USB drives in pre-mount. Normally it goes by quickly if the drives are clean. But if I accidentally pulled out a USB drive while it's active (yes, I've been known to remove the wrong USB stick), then I want fsck to take the time to clean things back up before it goes back into service. In that case, the delay before the entware volume is mounted can be excessive, long past when the services-start loop times out. But by putting the 'rc.unslung' in post-mount, it doesn't matter how long the fsck takes, I can be assured that's once entware is finally mounted, the router will return to normal with all extra applications started correctly.

Anyway, I've made the change in my rig, and am very happy with it. Just offering it up as perhaps a change to be made to the official entware installation scripts. Or am I missing something?

For completeness, here is my post-mount script. And I've completely eliminated the services-start script.

if [ "$1" = "/tmp/mnt/sys" ] ; then
  # set up and run Entware
  ln -nsf $1/opt /tmp/opt
  logger "$0:" "Enabling swap on /opt/swap"
  swapon /opt/swap
  logger "$0:" "Running Entware services-start ..."
  /opt/etc/init.d/rc.unslung start
fi

 

[heysoundude]

A couple of other observations:
- with swapon being placed in post-mount, and rc.unslung in services-start, there is a bit of a race condition. I was seeing my entware services started before swap was activated. I'd prefer to get swap in place before I start services that might depend on the swap being there. Moving rc.unslung to after swapon in post-mount addresses that.

- I've recently started doing an automatic fsck on USB drives in pre-mount. Normally it goes by quickly if the drives are clean. But if I accidentally pulled out a USB drive while it's active (yes, I've been known to remove the wrong USB stick), then I want fsck to take the time to clean things back up before it goes back into service. In that case, the delay before the entware volume is mounted can be excessive, long past when the services-start loop times out. But by putting the 'rc.unslung' in post-mount, it doesn't matter how long the fsck takes, I can be assured that's once entware is finally mounted, the router will return to normal with all extra applications started correctly.

Anyway, I've made the change in my rig, and am very happy with it. Just offering it up as perhaps a change to be made to the official entware installation scripts. Or am I missing something?

For completeness, here is my post-mount script. And I've completely eliminated the services-start script.

if [ "$1" = "/tmp/mnt/sys" ] ; then
  # set up and run Entware
  ln -nsf $1/opt /tmp/opt
  logger "$0:" "Enabling swap on /opt/swap"
  swapon /opt/swap
  logger "$0:" "Running Entware services-start ..."
  /opt/etc/init.d/rc.unslung start
fi

I think I'm following (and like) your logic, if I'm reading and understanding it correctly: do one thing at a time, do it properly/fully, then move on. Yes?

Does it speed up boot/launch, especially on older routers with slower/fewer processors and less RAM, like the n66u?

 

[elorimer]

One thing I just can't get my head wrapped around though, why is the call to 'rc.unslung start' placed in services-start rather than in post-mount? Services-start is called long before the entware USB volume is mounted, and so it's forced to just camp out in a loop waiting for the entware filesystem to become available. I've had to extend that wait loop several times because it kept timing out as I was adding more "stuff" to my config.

A kindred spirit: ublockr - a minimalists approach to adblocking

One counter argument is if entware is on an SD card: Why does entware rc.unslung run from services-start?

 

[latenitetech]

Does it speed up boot/launch, especially on older routers with slower/fewer processors and less RAM, like the n66u?

No, that's not the motivation, although there might be a small savings in not running the extra services-start script, and its time-wasting loop watching for entware to mount. The real motivation is just a more reliable boot sequence.

 

[thelonelycoder]

services-start runs only once when the router boots up.
post-mount is run everytime a new device is plugged in.
This may not be beneficial for the services you kick off in it.

 

[latenitetech]

One counter argument is if entware is on an SD card: Why does entware rc.unslung run from services-start?

Aha, I knew there had to be something driving that original design. I guess I should have done a better job looking for that question being previously asked before I posted. Sorry about that.

But that being said, I still maintain that for those of us with USB-mounted entware volumes, running rc.unslung in post-mount is a far superior design. Perhaps the installation script could provide both alternatives, either by asking in the install script, or figuring it out automatically based on where entware is located.

 

[latenitetech]

services-start runs only once when the router boots up.
post-mount is run everytime a new device is plugged in.
This may not be beneficial for the services you kick off in it.

I don't see the issue here. Post-mount checks for the specific volume being mounted, so it only takes action if the entware volume is involved.

But you do point out another benefit of having rc.unslung in post-mount: if you remove/insert that drive while the router is already up, it will automatically "restart" entware, something you don't get from rc.unslung being in services-start. If you truly had an application that didn't depend on entware being mounted (and was somehow bothered by it being "started" again if the entware volume was re-mounted and rc.unslung called again), I would argue it should just be started directly in services-start and not by entware's rc.unslung.

In my case, everything I start by rc.unslung is absolutely dependent on the entware volume being mounted, and would benefit from the post-boot auto-start "feature" inherent in having it called in post-mount.

 

[elorimer]

There is a little war going on here involving TLC's very useful ab-solution script, entware's default scripts, and what you are proposing. For a good while I started entware in post-mount because it made the same sense to me as it does to you.

services-start runs early in the boot process, before the thumb drive partitions are mounted, so it needs a sleep loop to hang around for a good long while until the entware partition mounts. (See what I did there? )

Some examples have that loop testing whether the partition that you think has entware on it has mounted, using the $1 parameter that gets passed. Some have it testing to see if the partition actually has an entware directory on it by using -d. Some have it testing to see if that directory actually has the executable rc-unslung script on it using -x. And some have it testing to see if there is the /opt/etc link active with that executable script on it.

So you have to be careful, I think, when you are moving some of those services-start things to post-mount. By way of example, my setup is totally wrong. My post-mount checks to see if the partition that is mounted is the thumb drive on which I expect entware to be, using $1, and if it is, forms the link to /opt/etc for that entware directory. My services start hangs around to see if /opt/etc/init.d/rc-unslung exists and is executable, then runs it. That comes from ab-solution. So I've created an unnecessary race condition. (Don't tell TLC. He'll be most vexed with me.)

The other thing that might be going on is if you are using services-start to stop a system service and replace it with an equivalent from entware. Above my pay grade.

 

[thelonelycoder]

Whatever runs and works on your system @latenitetech and @elorimer is likely the best for you.

But for AB-Solution with thousands of installations I had to go with a way that works for all cases and on a large variety of router models and firmware versions.
My own testing showed that the services-start is the reliable way to go. There is not a single complaint in the AB3 thread about Entware (and therefore pixelserv-tls) not starting.
This is prove enough to me that the decision I made was correct.

 

[latenitetech]

For a good while I started entware in post-mount because it made the same sense to me as it does to you.

What made you go away from that? I'm going to take a guess it was wanting to run AB, as it looks like the install script for AB assumes/enforces the use of services-start to kick off rc.unslung. Or was there something else that didn't work right that was fixed by moving it back to services-start?

 

[thelonelycoder]

What made you go away from that? I'm going to take a guess it was wanting to run AB, as it looks like the install script for AB assumes/enforces the use of services-start to kick off rc.unslung. Or was there something else that didn't work right that was fixed by moving it back to services-start?

@elorimer and I had that discussion before, as he hints at it. He may have given up on it because:
AB will silently append or write a new file if the entries are missing.
This is part of the installation check when the AB UI is started or shut down.

 

[latenitetech]

Whatever runs and works on your system @latenitetech and @elorimer is likely the best for you.

But for AB-Solution with thousands of installations I had to go with a way that works for all cases and on a large variety of router models and firmware versions.
My own testing showed that the services-start is the reliable way to go. There is not a single complaint in the AB3 thread about Entware (and therefore pixelserv-tls) not starting.
This is prove enough to me that the decision I made was correct.

Hey TLC - Thanks for the exchange. I feel like I'm the new kid poking the old guard. I certainly don't mean to challenge all the great work you've obviously done in this space, and although I've not personally installed AB, it looks like an amazing piece of work. But sometimes a noob can lend a new perspective.

AsusWRT-Merlin's customization hooks open up the possibility for any number of new and creative applications that can significantly change the boot timing, causing the primitive wait loop in services-start to break (i.e. timeout prematurely). And that failure can be frustrating for a noob to troubleshoot and fix.

This is clearly a very old issue … googling around and I'm finding posts dating back to at least 2013 where the original solution was a fixed 10 or 20 second delay in services-start, then somewhere along the way it was converted to a wait loop with either a 1 second or 5 second delay between checks, then it got extended to 30 seconds (and that's what's currently baked into the firmware's install script), and then lots of posts where it's 60 seconds.

A boot-time fsck can easily extend the post-mount delay into the minutes if the volume is dirty (not hard to do if one yanks the wrong USB stick out while it's active), so that would argue for making the loop timeout 5 mins or more. But why bother when moving the "entware start" to post-mount solves all that.

Looking at 'ab-solution-3.sh' it's clear you've put a lot of work into ensuring the environment is setup correctly for AB. Although I've not dissected it in detail, it appears the assumption and enforcement for having rc.unslung called from services-start is included there. So for example, if I wanted to use AB, and still keep my preferred post-mount start of rc.unslung, I would need to edit your 'ab-solution-3.sh' and customize it for me. Correct?

If you haven't noticed, I love a spirited technical discussion. Obviously I can do whatever I like on my own router, but I also am trying to do some "give back" and offer ideas and suggestions for improvements that can make it better for others to come, especially noobs who may be daunted by obscure failures that can readily be fixed. Just asking you (and other members of the "establishment") to consider my suggestion. Thx!

- Mark

 

[elorimer]

I've followed TLC's work now for a while, and I've had ab-solution on an N16, an N66, an AC87 and an AC56. I know I'm going to leap on whatever improvement he next chooses to make, and it is easier to do it his way, and the way the entware install script sets it up.

 

[thelonelycoder]

Looking at 'ab-solution-3.sh' it's clear you've put a lot of work into ensuring the environment is setup correctly for AB. Although I've not dissected it in detail, it appears the assumption and enforcement for having rc.unslung called from services-start is included there. So for example, if I wanted to use AB, and still keep my preferred post-mount start of rc.unslung, I would need to edit your 'ab-solution-3.sh' and customize it for me. Correct?

There are some lines that need out-commenting or deletion after AB and pixelserv-tls is installed (services-start is only checked and created if (ps) is installed) if you want to modify the rc.unslung start place.
While this is possible I rather not post here where it's done and would hate if anyone posted it.
The reason is, I don't want any support questions if someone does the editing and then it may not work at some point.

This would be a text-book PEBKAC error and the BKAC part of it would likely omit mentioning the edit done to ab-solution.sh.
A furious search for the cause of the problem would trigger unnecessary posts and confusion.

Not wanting to come across as an old bearded unix guru (I'm not a guru and no beard) I run it with your posted change on some of my testrouters to see if Entware comes up every time.
I see the advantage of starting rc.unslung right after the device is mounted and not 'pre start' it and wait until the device is mounted.

As this would be a larger change for AB, I need to do more testing and have the beta testers have a go at it first before I release it in the upcoming AB3.1 update.

 

[heysoundude]

(I'm not a guru and no beard)

The no beard might explain the lonely...

 

[thelonelycoder]

The no beard might explain the lonely...

I might have to include the letbeardgrow(now, killrazor) function in my code somewhere then.

 

[latenitetech]

... I rather not post here where it's done and would hate if anyone posted it.

TLC -I respect your position, and would never consider doing that. I appreciate you being open to the change though.

I just posted my first Merlin wiki contribution today, describing a way to run fsck on boot. But in it, I cautioned that the delayed boot could impact certain applications started from entware. And then I described the post-mount entware start change as a possible solution. But I also added the caveat that it may not be appropriate for some users, including current users of AB. Take a look and see if you think my wording is suitable if you get a chance:
https://github.com/RMerl/asuswrt-me...eck-at-Boot-(and-other-noobie-hints-and-tips)
The reference to AB-Solution is in the Gotchas section.

And I too could qualify for the "old enough for a gray beard," but have never gone that route!

 

[thelonelycoder]

My testing concluded that I will leave things as they are in AB-Solution.
That means rc.unslung will be started from services-start.

As RMerlin stated several times, the order in which these custom config files get run is not predictable and depends on other stuff going on while booting the router.
Out of the 20 reboots I did with with my testrouters, with rc-unslung started from post-mount, about 10% failed to start pixelserv. The reason was the wan-start script kicked in too late and the virtual interface for it was not ready. On one particular firmware version this is almost always the case.
Reverting back to services-start always works.

I may revisit this if I have more time for testing, but for now, AB3.1 will not introduce your findings.
Thanks for your write-up anyway, I've bookmarked this thread and the GitHub page for later.

 

[latenitetech]

That's really interesting. All the questions around this issue have always been "why was it done that way in the first place?" So you may have uncovered it. Thinking about it, the only way what you described could happen is if post-mount is called before services-start (certainly the opposite of what I always observe on my RT-AC68U), but also that rc.unslung always occurs after the WAN is up.

Could you share the logs of the two cases you observe (on the firmware you see consistent failure)? The one where rc.unslung is called from services-start and the one where it's called from post-mount?

If it's simply a matter of ensuring services-start has been called before releasing rc.slung from post-mount, it would be trivial to add setting a flag (a file under /tmp, for example) in services-start, then checking that flag in post-mount. If we also had to check on WAN up, that could be a bit more challenging (not sure how that is known).

And one more point, the "firmware" you're referring to, is that a particular router model, or is it older versions of AsusWRT-Merlin firmware on the same model? If it's the model that's different, is the problem an RT-N66 by chance? That's the one I've been led to believe has an internal SD card that entware can be installed on, and behaves differently than other USB-only routers.

 

[thelonelycoder]

I'm saving the files from the devices while I do other tests. This might take some days to complete.
The original logs were not saved.