I have cron command that runs a shutdown bash script at 12:10am every day.
10 0 * * * /bin/bash /var/www/scripts/special-reboot.sh
The script is very basic, all it does is shutdown nginx and moves data from MEMORY tables in MySQL to normal tables. Then it performs
sudo /sbin/shutdown -r now 'Daily restart.'
It's been running well for about a month or two, however last night something very strange happened and I need to know why.
I have a python logging script which checks that important processes are running every 15 minutes. After the reboot last night, the log looked like this (note the dates and times).
[2019-02-12 23:45] ... [2019-02-13 00:00] ... [2016-11-04 03:30] ... [2016-11-04 03:45] ... [2016-11-04 04:00] ... [2016-11-04 04:15] ... [2016-11-04 04:30] ... [2016-11-04 04:45] ... [2016-11-04 05:00] ... [2016-11-04 05:15] ... [2016-11-04 05:30] ... [2019-02-13 02:30] ... [2019-02-13 02:45] ... [2019-02-13 03:00]
As you can see, it suddenly starts counting from a different date time for about two hours, and then it goes back to normal.
The scariest part is that it wasn't only the python script that was affected. Another C script that tracks electrical pulses is missing data from that period of time.
I don't know what could've caused this. I know, due to my logging script that all of the important programs were running absolutely fine during this time period. I believe this is most likely a problem with the system rather than my programs.