Datalogger - 9dof - Lagging timestamp

I’m struggling with a lagging timestamp on my Sparkfun datalogger IoT - 9dof device.
I know that the board is online and I know that the server settings are correct (pool.ntp.org).
I’ve tried many, many things (disabling/enabling NTP, system reset, factory reset, changing update interval), but nothing seems to work. I got a new board and it seemed to be in sync for the first couple of times that I used it, but now I’m facing the same issues.
The device is not being handled particularly rough. Is the onboard clock very sensitive ? How can I tell whether or not the board is damaged ?

Did you try setting a primary and secondary NTP? It works as a failover, I believe Configuration - SparkFun DataLogger IoT Hookup Guide

You could add in an external RTC if ensuring timestamp accuracy is paramount

The secondary NTP server is time.nist.gov, but that’s not the issue.
The NTP startup is working, so I know the NTP client is connecting at least once at startup. However, the clock doesn’t stay in sync. Even if I set the update interval to 1 minute.