How/when is ntp supposed to trigger/run (ntp_ready=1)

[bengalih]

I've been getting error recently (from Skynet) about NTP failing to start.
I don't see any ntp type process running in a ps output and after confirming `ntp_ready=1` on my box after boot, it seems to revert to `ntp_ready=0` after some time.

Can someone delineate how and when the box should be doing time syncs and a few troubleshooting steps?

I will note that I did implement a workaround on a box which cannot get it's internet connection prior to disk mount, and the amtm checks require time sync in order not to hang.
So I do manually set ntp_ready during a pre-mount script and then revert it afterwards in order for the system to properly sync its time after disk mount.

I do get the `ntpd: Initial clock set` after boot, but from then on it seems not be occurring. This seems to be a more recent issue for me, as I've had this same configuration running for quite a while, but have done some tweaking (upgraded packages, etc), IPv6 tweaks, ddns changes, but nothing I can trace to what can be causing this.

To be clear: I'm not seeing *anything* in the logs that ntp is doing anything (no records at all). I only see complaints from Skynet that time isn't synced and when I check I see that ntp_ready has been reset to 0. So I'm trying to understand when/what/it I should see in the logs in terms of ntp firing and what I might be able to do to force it to run under close to normal circumstances so I can view what is going on.

Thanks.

 

[ColinTaylor]

I do get the `ntpd: Initial clock set` after boot, but from then on it seems not be occurring.

When you see the "ntpd: Initial clock set" message it is up and running and should stay that way. You won't see any subsequent messages. If the ntp service is stopped (perhaps because the WAN interfaced bounced) then I would expect to see a "ntpd: Stopped ntpd" message.

 

[bengalih]

When you see the "ntpd: Initial clock set" message it is up and running and should stay that way. You won't see any subsequent messages. If the ntp service is stopped (perhaps because the WAN interfaced bounced) then I would expect to see a "ntpd: Stopped ntpd" message.

Ok. So assuming I get the initial message and ntp_ready=1, you are saying that I won't see any more messages?
I still assume that ntp client is running occasionally in the background to check the time status/drift? I don't see anything in "ps | grep ntp" so I assume something is either triggered at set times or else built into another service?

In short, I'm looking for what could be the culprit that sets ntp_ready back to 0 after it has been already set to 1.
My theory is that some service is being restarted (either under normal cycling of service or possibly something malfunctioning/crashing) and when they restart they are either directly setting this back to 0 or else attempting another sync which is failing.

I can sync time manually with a peer:

root@router-asus:/jffs/scripts# /usr/sbin/ntp -t -n -q -p 10.10.10.100
ntp: setting time to 2023-04-27 12:29:17.103485 (offset +1.565801s)

And if I issue this command it will set the ntp variable:

keymaster@router-asus:/jffs/scripts# nvram get ntp_ready
0
keymaster@router-asus:/jffs/scripts# nvram get ntp_ready
0
keymaster@router-asus:/jffs/scripts#  /sbin/ntpd_synced step
keymaster@router-asus:/jffs/scripts# nvram get ntp_ready
1

But something seems to be resetting this to 0 and hanging up the scripts.

Is there a specific service I can bounce that would force the ntp code to execute without running the direct ntp_sycned command?

Also, is there necessarily supposed to be an etc/ntp.conf file? because I don't have one.

 

[ColinTaylor]

Ok. So assuming I get the initial message and ntp_ready=1, you are saying that I won't see any more messages?

Yes.

I still assume that ntp client is running occasionally in the background to check the time status/drift?

Yes.

I don't see anything in "ps | grep ntp" so I assume something is either triggered at set times or else built into another service?

The ntpd process should always be running. If you're not seeing it then something has gone wrong. I suspect you have some third-party scripts that are interfering with its normal operation.

Is there a specific service I can bounce that would force the ntp code to execute without running the direct ntp_sycned command?

service restart_ntpc

Also, is there necessarily supposed to be an etc/ntp.conf file? because I don't have one.

No.

 

[EmeraldDeer]

I am using chrony but it appears the builtin NTP is used on boot. The settings are in nvram.

# nvram getall | grep -E "ntp_|ntpd_" | sort
size: 84845 bytes (46227 left)
ntp_ready=1
ntp_server0=192.168.50.230
ntp_server1=time.apple.com
ntp_server_tried=192.168.50.230
ntpd_enable=0
ntpd_server_redir=0

 

[bengalih]

The ntpd process should always be running. If you're not seeing it then something has gone wrong. I suspect you have some third-party scripts that are interfering with its normal operation.

You are exactly right on both counts. I'm not 100% sure what happened but my customization used to work and something, somehow was changed and was causing this issue.

To be clear, my time did get initially synced at boot-up, but somehow the ntp_ready was being reverted and this was causing problems with Skynet which checks this variable *even after it is started* and will log errors every hour. Personally, I think the checks in that software are too aggressive, but I don't want to make the fixes there because they will be reverted on updates.

Basically, my problem stems both from AMTM and Skynet. AMTM will wait 5 for a 5 minute timeout if ntp_ready isn't set (and the Skynet issue detailed above). My issue is that I require a special copy of wpa_supplicant on my drive in order to initialize my WAN connection. And since I can't sync my time by built-in means until after the WAN is up, I had a 5 minute longer boot time. I got around this by using a pre-mount script to set time with a local box and then revert these changes after mount so that the system could continue to sync with a public ntp server as normal.

Again, this used to work, but something about this was now being finicky. I've rewritten some of my scripts to better deal with this, but I do want to make note of some things I discovered about the time service and these nvram variables that perhaps helps someone else out:

First, I'm not 100% sure how the time service is meant to kick off normally, but I suspect it is somehow bound to the WAN interface going up and/or being connected. Reason being (as stated above) it won't try to sync time until it knows it can reach the Internet.

What I have found is that if "ntp_ready=1" when the WAN interface comes then the system does not start the time server in the background. On a normal system as @ColinTaylor states you should see something like this:

root@router-asus:/jffs/scripts# ps | grep ntp
 2272 keymaste  1460 S    /usr/sbin/ntp -t -S /sbin/ntpd_synced -p pool.ntp.org
 6643 keymaste  2660 R    grep ntp

In my tests if I kill the ntp service and then restart my WAN (service restart_wan), the ntp service will not relaunch if "ntp_ready=1".
To the best of my knowledge ntp_ready should get set to 1 on the initial time sync and it should never go back to 0.
I'm curious if someone manually changed the time on the server and then ntp was unable to reach a time server if it would ever change this value back to 0. I don't think so.
This seems problematic to me, and I'm going to do a few more tests to see if anything on the box will relaunch ntp if it isn't running.

The other thing I noticed is that if ntp isn't running and "ntp_ready=0", when you start ntp it will not necessarily change the value to 1.
It will only change it if it actually syncs the time, which it won't do if the system time is close enough to the time ntp detects.
This also seems a bit problematic to me since launching the ntp client and that client verifying that time is "good enough" should tell the system that ntp is indeed "ready."

I suspect all of this behavior is default ASUS and doesn't cause problems when people aren't hacking their own boxes. But to those of us who are I hope these observations help someone else.

 

[Ripshod]

I suspect all of this behavior is default ASUS and doesn't cause problems when people aren't hacking their own boxes. But to those of us who are I hope these observations help someone else.

I wonder if I'm seeing the effects of this. After a reboot ntp doesn't sync which holds back a load of scripts, but I've not waited for any timeouts. I just go to Administration>System and go straight to the bottom and click apply. Like magic ntp syncs instantly.
Not really being a coder beyond copy/paste I haven't looked too far into it.
You think this behaviour is linked?

 

[bengalih]

I wonder if I'm seeing the effects of this. After a reboot ntp doesn't sync which holds back a load of scripts, but I've not waited for any timeouts. I just go to Administration>System and go straight to the bottom and click apply. Like magic ntp syncs instantly.
Not really being a coder beyond copy/paste I haven't looked too far into it.
You think this behaviour is linked?

Its hard to say. If you are an AMTM user then yes, there is a 5 minute timeout for the disk check to run if "ntp_ready" isn't set.

I suggest the first thing you do is after reboot you should ssh into the system and execute:

nvram get ntp_ready

If it shows "0" that means that your box hasn't synced time yet. You can see if ntp is running by issuing:

keymaster@router-asus:/tmp/home/root# pidof ntp
10315

If it returns any number like above then ntp is running. If it doesn't return anything then ntp isn't.

You can wait 5 extra minutes and see if things continue. If so it is almost definitely the amtm check. You can also manually set the variable with:

nvram set ntp_ready=1

And that should almost immediately cause amtm to process and continue. You can also check if your disk already mounted (another indicator the amtm script is holding it up) by issuing:

root@router-asus:/tmp/home/root# mount | grep /dev/sd
/dev/sda1 on /tmp/mnt/flash_drive type ext4 (rw,nodev,relatime,barrier=1,data=ordered)

You should see your sdX device mounted similar to mine above.

What I can also tell you is that on my box just clicking "APPLY" on that page you mention (even without changes) executes the following:

rc_service: httpd 975:notify_rc restart_time;restart_leds;restart_usb_idle;restart_firewall;

So it does execute a restart of time which would cause ntp to load and sync time if it wasn't already. The question then becomes why isn't your time service loading prior to you having to do this?

 

[bengalih]

I suspect all of this behavior is default ASUS and doesn't cause problems when people aren't hacking their own boxes. But to those of us who are I hope these observations help someone else.

Just to confirm my additional suspicions/observations. If ntp is somehow killed, it does not appear that there is any built-in process which will start it back again. This is true regardless of the value of "ntp_ready".
Again, this seems problematic to me that the system has no way to recover if ntp were to somehow crash. As stated, I suspect this is default Asus behavior and they probably assume ntp will never crash on the box as it is more of a closed environment by default. Additionally, the restart of several services do appear to also restart time service. However I don't know how many of those services (if any) will ever perform automatic restarts on a regular basis.
Having had my system sit for 24+ hours now and nothing has forced ntp back into a running mode. Additionally, under normal circumstances it is unlikely time would ever drift enough from initial sync until the router is next rebooted to make this an issue. It only really becomes an issue when adding on 3rd part tools which require it.

In short - as long as ntp_ready is set to 1 at the initial boot and sync on the router, all native services should continue to function and time drift shouldn't be an issue - even if ntp is killed and doesn't sync again. Outside configurations will have to deal with all the outlying eccentricities I've outlined in this thread.

 

[dave14305]

So I do manually set ntp_ready during a pre-mount script and then revert it afterwards

What did this script look like?

 

[bengalih]

eWhat did this script look like?

I've modified it now, but I pulled this from a backup. I admit with my latest testing results from above, I'm not entirely sure how this worked (will explain again in-line), but I only recently started seeing incidents of ntp_ready set to 0 and Skynet complaining. When I look at the logic now with my results it did seem to be error-prone.

I started off with the following in my pre-mount:

/usr/sbin/ntp -t -n -q -p 10.10.10.100 > /tmp/ntp.check 2>&1
grep "setting time to" /tmp/ntp.check
if [ "$?" = "0" ]; then
    /sbin/ntpd_synced step
else
    nvram set ntp_ready=1 #we force this here just in case to bypass timeout check in amtm disk-check
fi

That synced time with my local Domain Controller. Assuming that my DC was reachable on the network then ntp should always sync time which in this case will always meant an adjustment on the Router because the router always starts with the time/date way in the past. So on a successful sync it would log that string which I capture in the .check file. Then, I verified through the output file that the time was set and, if it was (again it always was) I would call "ntpd_synced step" which is what the system itself calls and that calls the underlying process which sets ntp_ready=1.
If for some reason time didn't set, I would force it to 1 as per the comment.

Then, immediately after in post-mount I had:

grep "setting time to" /tmp/ntp.check
if [ "$?" = "1" ]; then
        nvram set ntp_ready=0
fi
rm /tmp/ntp.check

Which would check to see if the time had not actually been set (through the .check file) and therefore revert ntp_ready back to 0. The idea here was to allow the system itself to move forward with its own time syncing procedure once the internet connection was established. In my case, if the time really wasn't synced within some reasonable time frame then my certificate negotiation for my wpa_supplicant would fail and I would never get internet access anyway.

Reviewing it again now I see several scenarios that could be problematic, but it seemed to work fine for about a year before I just started noticing issues. My goal was to bypass "normal operation" long enough to get my internet connection established and then try to revert back to letting the system manage it. That proves difficult because what I found (or what I'm now finding) is that if you set ntp_ready to 1, the system will seemingly never launch ntp (I don't see it in a ps list). And, if you set ntp_ready to 0 the system will launch ntp, but it seemingly will never send the command to set ntp_ready back to 1 because it never actually sets the time - at least in my case because the time was already set in pre-mount.

I have since changed the logic. The pre-mount is same - attempt to sync with my network time and, in any event force ntp_ready to 1 so amtm disk check doesn't stall. But I have changed my post-mount to this:

# If time not set by custom NTP, load time from file to hopefully allow wpa_supplicant to connect
grep "setting time to" /tmp/ntp.check
if [ "$?" = "1" ]; then
        /tmp/mnt/flash_drive/entware/bin/fake-hwclock load force
fi
rm /tmp/ntp.check

Here I am checking to see if time was actually set through the existence of the .check file and if it wasn't (i.e. I just forced ntp_ready=1), to go ahead and load data from the fake-clock file (which is saved hourly on my system via cron job). This would hopefully get my clock close enough timewise to allow the certificate exchange to work. Additionally I added the following in a wan-event user script:

if ([ "$1" = "0" ] && [ "$2" = "connected" ]); then service start_time; fi

This will force ntp to launch once my WAN is connected, otherwise it doesn't seem to do it.

Again, this may be more than needed - quite frankly if I just set my time using my local DC at boot at issued an "nvram set ntp_ready=1" it would probably be enough 99.9% of the time. I wanted a little extra for those situations that I need to reboot remotely when I'm on the road.

 

[bengalih]

I wonder if I'm seeing the effects of this. After a reboot ntp doesn't sync which holds back a load of scripts, but I've not waited for any timeouts. I just go to Administration>System and go straight to the bottom and click apply. Like magic ntp syncs instantly.
Not really being a coder beyond copy/paste I haven't looked too far into it.
You think this behaviour is linked?

bengalih:

Its hard to say. If you are an AMTM user then yes, there is a 5 minute timeout for the disk check to run if "ntp_ready" isn't set.

Just wanted to correct myself here. I remembered incorrectly. The amtm script timeout is I believe only 100 seconds, not 5 minutes. So if you have having delays longer than about 2 minutes at that point it is likely not the amtm script holding you up.