D3, Yes, usually the time variances will settle down and it all depends on the load, or how much is going on with the router. That the symptoms have happened with two different models you have is indeed perplexing.
Administration-Settings is where to enter/apply/test time servers, pool or otherwise. You may/may not have to reboot the router after you apply a new entry, but it will give you a clean start to see how a new server settles down or affects the router in your log. Note any use restrictions single time servers list, since using anything other than national/global pools isn't widely recommended practice. If your router pings the server too much, your router will be flagged. If everyone had to try this the servers would become useless in short order. When there's an isolated or good reason to test, and if it solves your issue, it's worth trying.
Through v380.68_4, our routers used the north American pool. The time issue cropped up in October with the 3200, the only model/build we saw this in. We read isolated reporting of almost identical symptoms, which others (and myself) at first thought could be a bug or worse, such as a hack directed at certain routers; difficult to prove. Similar bugs have disappeared after an upgrade.
Since the single time server fix worked on our 3200, we've seen this instead as a specific set of circumstances that triggers the time variance, ie, how the router is setup, powered-up, routed, tunnels, load, etc. Logs showed that time variances were worse when the router was loaded with new configs or anything requiring a reset or factory default, and that's when the time or date was skewed. Trying to find related posts with sufficient details, logs, photos, to narrow it down took a long time.
We tried different servers in zones where the VPN tunnels exited before settling on the one we stayed with; the problem stopped and we're still on the same server but I still manually check the router and host each cycle to be everything is synced. If the modem appears that it's synced but nothing is routing, I always suspect a timing issue and check the router.
Here are a few links that may give you some ideas. Don't give up on your 3200 too soon. Didn't intend to bury you with details, and good luck.
https://www.snbforums.com/threads/asus-rt-ac3200-issues-and-impressions.22414/ (setting time/date RT-AC3200, #103, 104, 105)
NIST has a note re upgrades:
https://www.nist.gov/pml/time-and-frequency-division/services/internet-time-service-its
http://support.ntp.org/bin/view/Servers/NTPPoolServers
Current nist.gov list:
https://tf.nist.gov/tf-cgi/servers.cgi
http://brentfidler.blogspot.com/2015/02/getting-ntp-server-to-work-on-asus-rt.html
The following is you want to dig deep;
http://www.satsignal.eu/ntp/setup.html