NTP does not work in asuswrt

[kvic]

My second post on this forum. Hello everyone.

A while back I had my RT-AC56U running for over two months without reboot. I was surprised by how much the clock has drifted during the period. It's more than a minute. Lately i got sometime to look into the cause and found the issue puzzling.

By design asuswrt has a NTP daemon that will sync up my router's clock against a time server on reboot. After that, the daemon will re-sync the clock every 12 hours. If things work properly, the router's clock should show little drift from the time server at anytime.

I can check for clock drift by issuing

ntpclient -i 3 -d -h stdtime.gov.hk​

where std time.gov.hk is a time server near me.

Output will include something like this:

Total elapsed: 5947.00
Server stall: 81.06
Slop: 5865.94
Skew: 430724.57​

It shows my router's clock has drifted by 430724.57 micro-seconds. That's a drift of ~0.43s since my manual sync about 10 hours ago.

Upon further experiments, I'm quite sure that asuswrt NTP daemon's per-12-hour sync does not happen. I can see the problem in merlin 43_2 and merlin 44. It could exist in less recent merlin versions (which were further complicated by other NTP related issue) as well as asus stock versions.

By looking at the NTP daemon code here, with my experiments at the moment, I could only conclude that the sleep function never returns. Hard to believe if so. But maybe possible after reading this?

Hope the sharp eyes here can spot the cause. Or maybe it's only my AC56U?

Cheers

 

[cjason]

I confirm this problem exists in Asuswrt Merlin 376.45
My AC56U is set to sync time with pool.ntp.org and after 1 days 22 hours 14 minutes uptime I get this when running

ntpclient -i 3 -d -h pool.ntp.org

Total elapsed:  19431.00
Server stall:      32.67
Slop:           19398.33
Skew:          1375517.72

 

[kvic]

Thanks for confirming. Do people running Asus stock firmware see the issue?

 

[CooCooCaChoo]

After reading this thread I just put this together to update the time at midnight everyday to keep things running on the correct time:

METHOD 1:

In the directory location /jffs a file with the same name as the login, which I will refer to as $NAME:

0 0 * * * ntpclient -s -h time.nist.gov

The line means run at 0 minutes at 0 hour (12:00 AM) every day of every week of every month the command "ntpclient -s -h time.nist.gov". The command sets the time received from the time server time.nist.gov. You can use an NTP server closer to you in lieu of time.nist.gov.

In the directory location /jffs/scripts a file called "init-start" with contents:

#!/bin/sh

cp /jffs/$NAME /var/spool/cron/crontabs/$NAME

Basically this copies the cron job from the first file you created over to the crontabs directory so that it can be run.

METHOD 2:

Alternately, you can attempt to set the cron job directly from the init-start script if you have only 1 cron job to run:

In the directory location /jffs/scripts a file called "init-start" with contents:

#!/bin/sh

cru a UpdateTime "0 0 * * * ntpclient -s -h time.nist.gov"

This does the same exact thing as the first method without using a crontabs file. Here it is adding a job through the cru (cron utility) a job named UpdateTime using the same parameters of the job from the first method.

Only benefit of the first method is if you need more than 1 cron job to run. Alternately you can also add multiple cron jobs using the init-start script as in method 2.

On both methods, if you already have stuff in the init-start script, just add the command to the end of the file.

 

[kvic]

Thanks for the workaround. Setting up a cron job for time sync will remediate the drifting clock.

I found a piece of GNU document that describes a bit more detail on potential issue of mixing sleep and SIGALRM. The potential issue is more on premature wake up from sleep(). Also if asuswrt is using GNU libc (which I believe it's) then we won't see pre-mature wakeup.

But rather the issue we're seeing in the "NTP daemon" is that the sleep() in the infinite while() loop never wakes up on predefined interval's to sync up the time. That still remains a puzzle and draws my immense curiosity.

 

[kvic]

Fundamental elements such as libc shall receive more respect than people would like to give. Having doubt in sleep() embarrasses me now when I see the root cause.

After its first sync, the "NTP daemon" is suspected of being trapped in line 131 - the pause()!

The design of ntp.c is less than desirable.

 

[CooCooCaChoo]

Hrm, after setting this up, it appears that cron doesn't seem to be executing the job. Need to look into it some more.

 

[kvic]

To remediate the drifting clock with a workaround, I would try a simple cron job e.g. adding the below line into "service-start" (if you're using merlin build).

cru a SyncTime "* */6 * * * ntpclient -s -h stdtime.gov.hk"

Replace stdtime.gov.hk with a time server physically near you. This will setup a job that synchronises the router's clock every six hours.

The bug in ntp.c need to be fixed. Or else we have a piece of 2MB junk in RAM doing no good.

 

[CooCooCaChoo]

Hrm, after setting this up, it appears that cron doesn't seem to be executing the job. Need to look into it some more.

Ok, figured out why it didn't work. Apparently the file has to have the name of the user name that you use to log on to the router.

After observing the router's time keeping, I determined that the time goes off very quickly. After a 24-hour period, it can be off by as much as 3 seconds. Having the ntpclient sync with an ntp server every day is what I would recommend. More often might be necessary, especially if you are OCD about time.

Myself, I have ntpclient sync every hour:

0 * * * * ntpclient -s -h time.nist.gov

And after some research I would recommend time.nist.gov for North America users. Those servers are what's called stratum-1 servers; machines that have a connection to machines that themselves are updated by a very reliable and accurate clock source.

 

[kvic]

The stuck-forever bug that we found in "ntp daemon" is not the only problem. ntpclient is not reliable either. It has two issues and the second one makes me uneasy.

I found that ntpclient does not succeed always. It's only about 87.5%. One may argue this also depends on the time server that ntpclient interacts with or network conditions when it runs. I'm not so convinced.

To improve this situation, I wrote a little script that re-tries when ntpclient fails. This works well. The clock syncs on each scheduled run.

A bigger surprise is I suspect ntpclient leaks memory! The more you run the less memory you're left with. To be fair, below is a list of candidates:

• ntpclient
• busybox
• nvram util

and I would bet on ntpclient.

The ironic thing is that the stuck-forever bug in "ntp daemon" saves the router from diving into instability. That's not a perpetual wait. By its design of a time sync every 12 hours, if it had worked, the day would come in about one and a half month.

 

[kvic]

The memory leak in ntpclient is a red herring because I couldn't reproduce.

Below is the script that I used to sync my time. It uses the time server you specify in firmware GUI. It tries until success or reaches max five attempts. The status will be logged in syslog.log.

#!/bin/sh

attempts=1
ntp_server=`nvram get ntp_server0`
while [ $attempts -le 5 ]
do
    nvram set ntp_ready=0
    ntpclient -s -h $ntp_server >& /tmp/status.txt
    if [ `nvram get ntp_ready` -eq 1 ]
    then
            echo "Succeeded after $attempts attempt(s)" >> /tmp/status.txt
            attempts=6
    else
            sleep 1
            let attempts++
    fi
done
logger -t "ntpclient.sh" < /tmp/status.txt
rm /tmp/status.txt

Save the script in, say, /jffs/scripts/ntpclient.sh. And set a cron job such as below

cru a SyncTime "* */6 * * * /jffs/scripts/ntpclient.sh"

This will sync your router's clock every six hours with fair confidence.

I have ideas on how to fix ntp.c, and hope to create a patch when I get time to setup my Linux build environment.

For now, I'm satisfied with the findings and workaround.

 

[kvic]

I have been using the script in #11 to sync time in the past year. Happy to report after my recent re-visit of the issue that Asus NTP fix is shaping up.

At least for my AC56U, I can confirm it's updating clock every 12 hours - the designated frequency in Asuswrt's code. People like me may prefer a shorter interval, say every 6 hours or even 2 hours. To do so, we can use a feature in Asus implementation. Set up a cron job with a frequency of your liking and run the following command:

killall -s ALRM ntp

This sends an alarm expiry signal to the NTP process and wakes it up to sync the clock. A bonus by doing this way is a round robin of using two NTP servers in every successive run. The time servers are stored in NVRAM variables, ntp_server0 and ntp_server1.

Happy tuning your router's clock.

 

[KevTech]

Never had this issue myself and it seems to update or at least check once an hour.

Jul 21 11:59:57 ntp: start NTP update
Jul 21 12:59:57 ntp: start NTP update
Jul 21 13:59:57 ntp: start NTP update
Jul 21 14:59:57 ntp: start NTP update

 

[kvic]

Never had this issue myself and it seems to update or at least check once an hour.

Jul 21 11:59:57 ntp: start NTP update
Jul 21 12:59:57 ntp: start NTP update
Jul 21 13:59:57 ntp: start NTP update
Jul 21 14:59:57 ntp: start NTP update

You may have time_zone_x=DST in nvram. By design then ntp will sync the clock every hour.

One side note: I still don't figure why a few people are getting "ntp: start NTP update" in syslog.log. By design, this message should not print in every sync but only the first time.

 

[KevTech]

Stock firmware with log_level set to 7 instead of the default of 6.

 

[kvic]

Asuswrt Merlin has default 7.

Only reasonable guess (until someone running Merlin see the same logs) is the stock firmware can be slightly different from codes released under GPL.

 

[joltdude]

My second post on this forum. Hello everyone.

A while back I had my RT-AC56U running for over two months without reboot. I was surprised by how much the clock has drifted during the period. It's more than a minute. Lately i got sometime to look into the cause and found the issue puzzling.

By design asuswrt has a NTP daemon that will sync up my router's clock against a time server on reboot. After that, the daemon will re-sync the clock every 12 hours. If things work properly, the router's clock should show little drift from the time server at anytime.

I can check for clock drift by issuing

ntpclient -i 3 -d -h stdtime.gov.hk​

where std time.gov.hk is a time server near me.

Output will include something like this:

Total elapsed: 5947.00
Server stall: 81.06
Slop: 5865.94
Skew: 430724.57​

It shows my router's clock has drifted by 430724.57 micro-seconds. That's a drift of ~0.43s since my manual sync about 10 hours ago.

Upon further experiments, I'm quite sure that asuswrt NTP daemon's per-12-hour sync does not happen. I can see the problem in merlin 43_2 and merlin 44. It could exist in less recent merlin versions (which were further complicated by other NTP related issue) as well as asus stock versions.

By looking at the NTP daemon code here, with my experiments at the moment, I could only conclude that the sleep function never returns. Hard to believe if so. But maybe possible after reading this?

Hope the sharp eyes here can spot the cause. Or maybe it's only my AC56U?

Cheers

KVIC

You Sir have explained what iv seen for some time, but couldn't seem to figure out why, but I didn't dig into the code...

The original answer I got to my complaint was .. Its a router not a timeserver, plus it doesn't have a RTC.... I still thought there was something amiss with NTP implementation... Thank you for explaining it and finding out the cause...(Was trying to figure out a workaround with using a GPS on one of the USB ports)....or perhaps use a Dallas and GPS on a Raspberry Pi, plug it into the network with the routers and set it up as a timeserver or broadcast NTP

BTW it affects other ASUS routers, I think it goes all the way back to the RTAC66.. perhaps even earlier

-J

 

[RMerlin]

Stock firmware with log_level set to 7 instead of the default of 6.

There was a reason why we had different default log levels, however I can't remember what they were (it was over a year or two ago that I last played with that particular setting).

 

[KevTech]

There was a reason why we had different default log levels, however I can't remember what they were (it was over a year or two ago that I last played with that particular setting).

Not sure what else is not shown but I do know that log_level 6 on stock firmware will not show any dnsmasq-dhcp messages.

I like to keep track of what connected and when.

 

[RMerlin]

Not sure what else is not shown but I do know that log_level 6 on stock firmware will not show any dnsmasq-dhcp messages.

I like to keep track of what connected and when.

If I remember, Asus changed the log level of those messages rather than provide an option to hide them. It wasn't a change in the default logging level of the whole system.