General CRU/SSH weirdness and questions

[Viktor Jaep]

I have a couple of things that are irking me, and hoping someone might have a decent explanation for them.

1.) CRU Weirdness RESOLVED -- I had created a CRU job a while back to run a script at the same time each night. About 4 weeks ago, I no longer had need for this script, and deleted the CRU job (cru -d <unique id>). I checked the cru -l (list), and sure enough it's gone. But then, a week or so goes past, and all of a sudden this cru job is back in my list, running this nightly job again! I did not add it back, and came back on its own. Considering it a fluke, I deleted it again, and double checked, it's no longer listed. Again, randomly 2 nights ago, it was back in my cru jobs list. What's going on? Why won't it stay gone?

2.) SSH Weirdness RESOLVED -- I have always run long-running scripts, like @Martineau's VPN Failover script, in a PuTTY window running on a PC directly attached to the router. Never seemed to have a problem doing it this way. A few years later, on a different router now, I started running my VPNMON-R2 script in this same fashion, and started randomly running into what seem to be hangs, or disconnections. The window would not completely redraw on a refresh of the screen every 60 seconds. I figured, OK, maybe I'm losing connection somehow, and was delighted to find out from @eibgrad that you could also use the "Screen" function, which runs using a daemon directly on the router itself... no more need for the external PuTTY! I thought for sure this would solve my issue. The strange thing is, I'm running into the same exact issue. The screen just hangs on a redraw... and will find that its just sitting there for hours after reconnecting to the screen window again. I've tried to add some debugging to the script thinking perhaps it's having trouble pinging, or running an API call, but nothing is coming up as a red flag. It will just randomly happen, sometimes after a few hours, other times, after a few days. What could cause screen to just hang the script? And is there a better way to get to the root of this problem?

Thanks in advance for your expert advice...

EDIT: Might be wondering why SSH Weirdness was resolved? Well, it was all because this was dealing with that strange AC86U script locking bug that has been discussed ad naseum. The only fix that worked for me sofar was using the "timeout" command whenever I had to do an "NVRAM GET" command.

 

[ColinTaylor]

1. Adding or deleting cron jobs interactively using cru is only temporary until the router is rebooted. To make cron jobs persistent you have to add them to a system startup script. So look for cru commands you added to a user script.

 

[Viktor Jaep]

1. Adding or deleting cron jobs interactively using cru is only temporary until the router is rebooted. To make cron jobs persistent you have to add them to a system startup script. So look for cru commands you added to a user script.

Good catch! For some reason, and I don't know if I added this manually... but there was a cru statement in my "services-start" file that looks like it was adding it to the list! Mystery #1 resolved! Thanks @ColinTaylor!

 

[eibgrad]

There's no reason to believe that screen is going to correct a display problem w/ the script, which this seems to be. Whatever problems exists prior to using screen will persist.

 

[Viktor Jaep]

There's no reason to believe that screen is going to correct a display problem w/ the script, which this seems to be. Whatever problems exists prior to using screen will persist.

Thanks @eibgrad - how would you troubleshoot something like this?

 

[eibgrad]

Thanks @eibgrad - how would you troubleshoot something like this?

I could take a look and see if I notice something obvious. If it's less than obvious, then it could take some effort to come up to speed on the script and hopefully reproduce it. Sometimes it's something unique to your particular setup. So the simpler you can make the configuration while still triggering the problem, the better.

 

[Viktor Jaep]

I could take a look and see if I notice something obvious. If it's less than obvious, then it could take some effort to come up to speed on the script and hopefully reproduce it. Sometimes it's something unique to your particular setup. So the simpler you can make the coniguration while still triggering the problem, the better.

I truly appreciate your expertise and trained eye on this, @eibgrad ... They are not very big or complex scripts - I'm still a complete n00b!

Please take a look at this one: https://github.com/ViktorJp/VPNMON-R2/blob/main/vpnmon-r2-0.6.sh

I have made some modifications to this one (0.6) here to try to sort out what may be causing a hang and posted it as well: https://github.com/ViktorJp/VPNMON-R2/blob/main/vpnmon-r2-0.7beta.sh ... I went through the process of adding echo's before and after major items, as well as adding more sleep statements to give things more time just to see if it was failing at the same spot each time... but after finding that none of these really mattered, I took those out, and instead tried to turn off error checking before major items using "set -o pipefail -e" and then re-enabling it again with "set +o pipefail +e". But even with this, it has frozen... more often than not, it freezes when running the checkvpn() function.

Thank you!

 

[eibgrad]

I took a VERY quick glance and had an immediate comment.

One thing I've noticed most script writers do NOT do is make their scripts debuggable! IOW, there's little if any consideration given to what would help debug a problem. Not unless the problem is already manifest. And then, like you, the typical response is to plaster echo commands all over the place hoping to happen upon the culprit.

If you examine my own scripts, I always plan ahead and assume I will have to debug it at some point. And it gets especially tough when you're dealing w/ scripts that run w/o the benefit of a terminal, such as the various event scripts (init-start, firewall-start, etc.). So what you'll find w/ my scripts is a DEBUG option that when enabled issues the 'set -x' shell command to dump all the actions performed by the interpreter. And I force ALL the output from the script to the syslog (at least for event driven scripts) and/or a log file, by placing the entire script in a block, either { } or ( ) (the latter being necessary if I plan to run the script as a standalone process, which is typically the case when I want to run it in the background).

The script will typically follow the pattern below.

#!/bin/sh
#DEBUG=; set -x # uncomment/comment to enable/disable debug mode
LOG="$([ ${DEBUG+x} ] && echo '/tmp/$(basename $0).$$.log' || echo '/dev/null')"
{
# everything the script does goes here!
} 2>&1 | tee $LOG | logger -t $(basename $0)[$$]

IOW, at any time, I (or the user) can simply uncomment that DEBUG line and be able to capture pretty much everything necessary to debug the problem, and usually w/o the need for modifying the script.

For a standalone script such as yours, the logger isn't necessary. But I included it to show what I often do when dealing w/ the event scripts.

This has saved me INCALCULABLE time and energy. I just enable DEBUG mode and ALL my output, including that from the interpreter, is dumped to the log file and/or syslog. Easy. Granted, that log can get mighty big in some instances. But it's a helluva lot easier than trying to pick your way through the code adding echo commands all over the place. And if a user claims to have an issue w/ the script, I can simply tell them to enable DEBUG mode and send me the log. Probably 90% of the time that will provide all the information I need.

So get into that habit! Because right now, that's what I plan to do w/ your script! LOL

 

[Viktor Jaep]

The script will typically follow the pattern below.

#!/bin/sh
#DEBUG=; set -x # uncomment/comment to enable/disable debug mode
LOG="$([ ${DEBUG+x} ] && echo '/tmp/$(basename $0).$$.log' || echo '/dev/null')"
{
# everything the script does goes here!
} 2>&1 | tee $LOG | logger -t $(basename $0)[$$]

This is such great advice... thank you very much. I will start experimenting with this and see how it can benefit my troubleshooting capabilities. I am still so new to shell scripting, I'm still not sure what's possible and what's not... what's bash and what's not... it's really been a hitting-the-wall and trying, then trying again kind of learning process for me.

So get into that habit! Because right now, that's what I plan to do w/ your script! LOL

I appreciate any and all feedback you have... I truly appreciate your help!

 

[Viktor Jaep]

So get into that habit! Because right now, that's what I plan to do w/ your script! LOL

I'm still too unfamiliar with how this exactly works... but I had it working. Unfortunately, it was not displaying anything onscreen, but logging everything to a .log file under /tmp, and dumping everything to the syslog. I tried manipulating it a bit to also show screen output, and failed... so I just added this to the start of my main loop:

DEBUG=; set -x # uncomment/comment to enable/disable debug mode
{
 #main loop
}

So now at least I'm seeing everything on screen, and hope to catch it in the act... Thanks again!

 

[Weblee2407]

OK..I have some general questions that seem to fit here.

1 - I have read several articles online about writing a cron entry (cru a) that will execute the last particular day (lets say Sat) of the month.. I cant get anything I find to work like 6L or SatL; when I search for examples of cron/cru entries what implementation of cron should I be searching for?

2 - using a date command in a script that fires every Saturday has "get date then add X days" logic that doesnt seem to work either. What particular implementation of date is in use iin Merlin?

3 - if I double tab and get a list of commands - are those busy box commands? I have found I can generate a list of busy box commands too. How do you know what command you are calling when they have the same name?

4 - is there any documentation for these commands in a Wiki somewhere? ..Because I cant get a man page to present either.

Please help - I need the big picture.

Oh forgot to add - my ISP has Gig internet rolling out and they are rebooting the cable modem on a somewhat regular basis...each time they do that all the CRU entries from everything except diversion are lost. I found an old discussion about events causing this to happen but its from 2020 or something. Is this a known issue in the current revision? All of my cru adds are in post-mount

ref

CRON jobs keep deleting?

I noticed the other day that the LED scheduler on my router stopped working. I found this was due to all CRON jobs being deleted apart from 5 Diversion related ones. I did a forced update to all my addons (scMerlin, ntpMerlin, conmon, Diversion, etc.) and they all seemed to be restored. I’ve...

www.snbforums.com

 

[ColinTaylor]

1. crond, like most services on the router, is a fairly basic implementation and doesn't support many of the non-standard extended options. So look online for the standard cron man pages.

2. Again, the built-in date command is a fairly minimal implementation from busybox.

# /bin/date -?
/bin/date: invalid option -- '?'
BusyBox v1.25.1 (2022-03-25 10:23:25 EDT) multi-call binary.

Usage: date [OPTIONS] [+FMT] [TIME]

Display time (using +FMT), or set time

        [-s,--set] TIME Set time to TIME
        -u,--utc        Work in UTC (don't convert to local time)
        -R,--rfc-2822   Output RFC-2822 compliant date string
        -I[SPEC]        Output ISO-8601 compliant date string
                        SPEC='date' (default) for date only,
                        'hours', 'minutes', or 'seconds' for date and
                        time to the indicated precision
        -r,--reference FILE     Display last modification time of FILE
        -d,--date TIME  Display TIME, not 'now'
        -D FMT          Use FMT for -d TIME conversion

Recognized TIME formats:
        hh:mm[:ss]
        [YYYY.]MM.DD-hh:mm[:ss]
        YYYY-MM-DD hh:mm[:ss]
        [[[[[YY]YY]MM]DD]hh]mm[.ss]
        'date TIME' form accepts MMDDhhmm[[YY]YY][.ss] instead

3. Mostly, plus some Asus or Merlin additions.

# ls -l /bin/date
lrwxrwxrwx    1 admin    root             7 Mar 25  2022 /bin/date -> busybox

4. Not really. Either use a command's built-in help or look at the standard Linux man pages. Other than that you'll need to look at the source code.

 

[JGrana]

4. Not really. Either use a command's built-in help or look at the standard Linux man pages. Other than that you'll need to look at the source code.

Or, for personal scripts, consider installing the Entware coreutils full version.

opkg list | grep coreutils

Keeping in mind the long debated issues with $PATH and the dangers of having a script that uses the busybox version might not expect the "full" version.

(procps-ng is another set of commands that are useful - I install procps-ng-ps as soon as I install Entware).

 

[Martineau]

2 - using a date command in a script that fires every Saturday has "get date then add X days" logic that doesnt seem to work either. What particular implementation of date is in use iin Merlin?

You can manually calculate the date that is 7 days in the future using the ASUS firmware version of date

i.e.

TODAYS_MONTH=$(date +%B)                                                     # Extracted from Today's date e.g. 'April' (or use +%m for '04' value)
MONTH_IN_7_DAYS=$(date -D %s -d $(( $(date +%s) + $((7*86400)) )) +%B)       # + 7 days

if [ "$TODAYS_MONTH" == "$MONTH_IN_7_DAYS" ];then
    echo "$(date "+%A %F") is not the last $(date +%A) in $TODAYS_MONTH."
    exit
fi
echo "<your desired Action(s) go here>"

 

[Weblee2407]

First - let me simply say THANK YOU for this wonderful example. I will spend the coming weeks dissecting this snippet to find out how it works. I had just started to investigate the -D, but I was no where close to what this script does.

Secondly - Coming from a PowerShell background: OH MY GOSH!

 

[Weblee2407]

Well, as is the tradition on this forum, I have been taught to fish. I can understand the script and what it is doing and why. But yes - one final question. How do I learn how to use the various parts of syntax ('(+ for date without accidentally crashing NTPD or setting my clock to some time that is too far out of range for autocorrect by NTP?

Since I know what this does, I'm good with using it, but I would like to safely poke around more.

 

[Martineau]

Well, as is the tradition on this forum, I have been taught to fish. I can understand the script and what it is doing and why. But yes - one final question. How do I learn how to use the various parts of syntax ('(+ for date without accidentally crashing NTPD or setting my clock to some time that is too far out of range for autocorrect by NTP?

Since I know what this does, I'm good with using it, but I would like to safely poke around more.

AFAIK, unless you specifically use the '-s'/'--set' directive, you will never alter the system clock, nor crash NTPD.

The Linux/Unix date web tutorials list the available FMT modifiers some of which may not be available in the ASUS Busybox version of date.

The most common are:

So the FMT definition is preceded by the initial '+' character

e.g. for the current date, customise the text string returned (where the '+' sign indicates the quoted string contains % placeholders to be substituted by actual values)

date +"This is a custom FMT %F for %A and allows 'date' to return any part of the supplied date e.g. Epoch seconds returns %s"

This is a custom FMT 2023-04-14 for Friday and allows 'date' to return any part of the supplied date e.g. Epoch seconds returns 1681471741

or what day was 1984-04-08?

date -d "1984-04-08" +"%A"

Sunday

Switching between the numerical value i.e. Epoch seconds and the human-readable format(s) is key in overcoming the lack of convenient date manipulation directives such as '+7 days' or even 'next week' that are only available on the full Linux/Unix implementation of date on other platforms.

e.g.

date -d "1984-04-08 01:30" +"%s"

450232200

so you could then add/subtract the desired number of days (in seconds) then pass the result back as the -d argument to convert back into the desired human-readable format.

 

[Weblee2407]

AFAIK, unless you specifically use the '-s'/'--set' directive, you will never alter the system clock, nor crash NTPD.

The Linux/Unix date web tutorials list the available FMT modifiers some of which may not be available in the ASUS Busybox version of date.

The most common are:

View attachment 49310

So the FMT definition is preceded by the initial '+' character

e.g. for the current date, customise the text string returned (where the '+' sign indicates the quoted string contains % placeholders to be substituted by actual values)

date +"This is a custom FMT %F for %A and allows 'date' to return any part of the supplied date e.g. Epoch seconds returns %s"

This is a custom FMT 2023-04-14 for Friday and allows 'date' to return any part of the supplied date e.g. Epoch seconds returns 1681471741

or what day was 1984-04-08?

date -d "1984-04-08" +"%A"

Sunday

Switching between the numerical value i.e. Epoch seconds and the human-readable format(s) is key in overcoming the lack of convenient date manipulation directives such as '+7 days' or even 'next week' that are only available on the full Linux/Unix implementation of date on other platforms.

e.g.

date -d "1984-04-08 01:30" +"%s"

450232200

so you could then add/subtract the desired number of days (in seconds) then pass the result back as the -d argument to convert back into the desired human-readable format.

Outstanding! Thanks

 

[Viktor Jaep]

So to close out this thread and the [OP]...

You might be wondering why my SSH Weirdness was resolved (from the original question in the [OP]? Well, it was all because this was dealing with that strange AC86U script locking bug that has been discussed ad naseum. The only fix that worked for me sofar was using the "timeout" command whenever I had to do an "NVRAM GET" command.