0

I'm using a program named zBackup over a lot (100+ >90GB each) of tar files. Every five or so, the system reboots. I've used

last -x

Which shows a crash

kayot    pts/2        --.--.--.--      Wed Dec  5 14:19   still logged in
runlevel (to lvl 5)   4.15.0-42-generi Wed Dec  5 14:19   still running
reboot   system boot  4.15.0-42-generi Wed Dec  5 14:18   still running
kayot    pts/2        --.--.--.--      Wed Dec  5 11:59 - crash  (02:19) <-- Here
runlevel (to lvl 5)   4.15.0-42-generi Wed Dec  5 11:58 - 14:19  (02:20)
reboot   system boot  4.15.0-42-generi Wed Dec  5 11:58   still running
shutdown system down  4.15.0-39-generi Wed Dec  5 11:57 - 11:58  (00:00)
kayot    pts/2        --.--.--.--      Wed Dec  5 11:57 - 11:57  (00:00)
kayot    pts/2        --.--.--.--      Tue Dec  4 10:23 - 10:35  (00:11)

** I replaced my IP address with --.--.--.--

Which coincides with the reboot.

However checking the kern.log and syslog around those times doesn't tell me what happened that is causing this issue.

Dec  5 12:05:55 core 50-motd-news[9567]:    - <Removed MOTD HTML Link>
Dec  5 12:05:55 core systemd[1]: Started Message of the Day.
Dec  5 12:13:54 core systemd[1]: Starting Cleanup of Temporary Directories...
Dec  5 12:13:54 core systemd[1]: Started Cleanup of Temporary Directories.
Dec  5 12:17:01 core CRON[29385]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  5 13:17:01 core CRON[25753]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  5 14:17:01 core CRON[8785]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
<Rebooted Here>
Dec  5 14:19:01 core systemd-modules-load[511]: Inserted module 'iscsi_tcp'
Dec  5 14:19:01 core kernel: [    0.000000] microcode: microcode updated early to revision 0x25, date = 2018-04-02
Dec  5 14:19:01 core systemd-modules-load[511]: Inserted module 'ib_iser'

There are no hourly cron tasks.

Dec  5 11:59:03 core kernel: [   62.374831] audit: type=1400 audit(1544029143.830:51): apparmor="STATUS" operation="profile_load" label="lxd-website_</var/lib/lxd>//&:lxd-website_<var-lib-lxd>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=5802 comm="apparmor_parser"
Dec  5 11:59:03 core kernel: [   62.374833] audit: type=1400 audit(1544029143.830:52): apparmor="STATUS" operation="profile_load" label="lxd-website_</var/lib/lxd>//&:lxd-website_<var-lib-lxd>:unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=5802 comm="apparmor_parser"
<Rebooted Here>
Dec  5 14:19:01 core kernel: [    0.000000] microcode: microcode updated early to revision 0x25, date = 2018-04-02
Dec  5 14:19:01 core kernel: [    0.000000] Linux version 4.15.0-42-generic (buildd@lgw01-amd64-023) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 (Ubuntu 4.15.0-42.45-generic 4.15.18)

The system isn't overheating. I've watched the CPU temps in the SuperMicro IPMI tool and they aren't going up.

My OS is Ubuntu 18.04 LTS. This data is on a ZFS and being written to the same ZFS.

What should be my next diagnostic step?

  • Why are you backing things up onto the same drive? – Michael Prokopec Dec 6 '18 at 1:55
  • I dumping the files off a damaged 4TB NTFS deduplicated drive that has >26TB of backup data on it. My array only has 6TB free so I'm using zBackup to try and get all the data off the drive. So far the backup is only 120GB and it already has a few years in it. This drive was part of three drives that were rotated. Due to the low disk space usage of zBackup I'm considering making a backup folder that will keep all the backups, and rsync'ing it to the backup disks. Basically a Windows to Linux migration path with three copies, if it stops crashing. – Kayot Dec 6 '18 at 14:39
  • 26TB doesn't fit in 4TB, yes it is possible to make a backup and have it store the changes for one or even multiple systems and have it able to reconstitute the information to each system as it was. That doesn't mean that the data backed up is the size of all the systems when restored. It simply copies the information that is not the same and makes note of what system the changes are present on. So it isn't even a compression scheme it's compression by the omission of unnecessary duplicates. You are good however in the fact that the 4TB will fit in that 6TB. – Michael Prokopec Dec 6 '18 at 16:29
  • You're thinking incremental backup. These are full backups. The data when not deduplicated is >26TB. zBackup deduplicates using a sliding window hash. The drives in question all use Windows Server 2016 deduplication. I'm pulling the data off with a WS2016 Machine to a Ubuntu 18.04 file share. Then I'm deduplicating the data when I fill the array up. I can't compress the tar data because I'd lose deduplication abilities, but I can compress the zBackup as it's post deduplication. However, this question is in regards to an undiagnosable crash. Once I get that fixed the rest is simple. – Kayot Dec 6 '18 at 17:14
  • The crash only happens when you are backing up and there is no trace of it in your logs. You have a lot of disk activity when backing up I assume? It seems that if nothing software wise is causing the issue. Look at the hardware. What kind of power supply are you running? Do you have a UPS? It is acting like the power is just going out for a second then resetting. May not seem like much but look into it. – Michael Prokopec Dec 6 '18 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.