I found that it helps to redirect the DNS lookups for NTP server names, to the router's NTP server. It is for when your Internet connection is unstable or your upstream DNS server is flaky.
So funny - ran into this the other day, and came to same conclusion to solve a similar problem
Just using 192.168.0.1 as an example... myrouter.lan is 192.168.0.1
iptables -t nat -A PREROUTING -i br-lan -p udp --dport 123 -j DNAT --to 192.168.0.1
edit /etc/dnsmasq.conf - assuming that hostname is myrouter.lan
cname=time.nist.gov,myrouter.lan
cname=time-a.nist.gov,myrouter.lan
cname=time-b.nist.gov,myrouter.lan
cname=time-nw.nist.gov,myrouter.lan
cname=pool.ntp.org,myrouter.lan
cname=android.pool.ntp.org,myrouter.lan
Add/Ensure to slash etc slash hosts # cloudflare stuff
192.168.0.1 myrouter.lan
Bonus Points
Add/Edit ntp configs... below is for
openwrt
/etc/config/system as it runs busybox for ntp
config timeserver 'ntp'
option enabled '1'
option enable_server '1'
list server 'time1.google.com'
list server 'time2.google.com'
list server 'time3.google.com'
list server 'time4.google.com'
For real NTP - just add those as servers to slash etc slash ntp.conf
Some folks worry about google time* - and that's ok, just change it out for the pools for local regions.
Google public NTP servers work well with Google public DNS - short hops, lower latency
* google public NTP smear time to cover for things like leap seconds - when one has 100,000 plus servers synced up globally, doing a gentle smudge over 20 hours prior to the leap second jump makes sense.
for ntp.conf is really running gps/pps - as mentioned - add the stanza for PPS and NMEA - there's an appropriate REFID for those services...
@kvic - seems good enough for gobernent work... the ASCII codes for REFID are fun
This alone is sufficient. No need to add entries to dnsmasq..
The DNSmasq entries do keep things local... it's complicated with real ntp in this setup...
From a cold startup - it's good, but over time, ntp sorts itself - but the initial DNS lookups come into play - host thinks it's talking to a remote, but the real data is coming from the local timeserver