1

Problem

Suddently, all the files from feb-2015 to nov-2016 have disappeared from the filesystem.

Context

  • Over the last 2 years, the system repeatedly terminated abruptly for power-fail reasons or for USB-hardware disconnections.
  • This loss does not seem because of bad-blocks or things like this. It seems more something software-related.
  • The disks are flat full-size vmdk files on a USB hardware-based RAID-5 connected to a VirtualBox Ubuntu.
  • The host holding the RAID and the VirtualBox is a Windows.
  • The RAID in windows is NTFS and the VirtualBox runs Ubuntu accessing the vmdk formatted in ext4. The Windows does not do other function than just containing a bunch of virtual machines and no user enters that Windows except to launch or stop virtual machines.

Situation

I've been using those disks (from the Ubuntu) on an intensive usage on a daily basis. And I have successfully used everyday the files from the previous day, and the previous week, and the previous month, and so on... Even after reboots... So the data must be there.

But suddently... my Dropbox started to delete files, I went to see what happened, and the fact is that... I can't tell if this happened "on the fly" or after a reboot.

But the thing is that... the latest file I can see is from jan-2015 while I've successfully been working with those disks until nov-2016 every single day. The files probably are in the blocks and for some reason I've lost all the index.

I can easily imagine that for some reason, the ext4 conserves backup-copies of i-node tables or something like this, and for some reason, the linux decided to "restore" the mapping to an old-one.

I've seen there are a bunch of tools. From running fsck to extundelete, and others.

What I see

Doing:

find . -newermt 20150201

gives me zero results.

In addition,

  • For files I know they existed, I cd into the corresponding directories and I can't see the files there.
  • I can see a directory structure I know I deleted.
  • This has happened to several disks attached to the virtual machine.

ext4 or NTFS?

I have unmounted the RAID that contains the vmdk virtual disks from the Windows and I have mounted it read-only on a linux.

If I list the files, I can see this:

root@vagrant:/mnt/raid5/mnt# ls -la
total 1572864014
drwxrwxrwx 1 root root         4096 Jan 29  2015 .
drwxrwxrwx 1 root root         4096 Oct 31 21:14 ..
-rwxrwxrwx 2 root root 268435456000 Jan 26  2015 LinuxData250G_A-flat.vmdk
-rwxrwxrwx 2 root root          651 Jan 26  2015 LinuxData250G_A.vmdk
-rwxrwxrwx 2 root root 268435456000 Jan 26  2015 LinuxData250G_B-flat.vmdk
-rwxrwxrwx 2 root root          558 Jan 26  2015 LinuxData250G_B.vmdk
-rwxrwxrwx 2 root root 536870912000 Jan 26  2015 LinuxData500G_A-flat_3.vmdk
-rwxrwxrwx 2 root root 536870912000 Jan 26  2015 LinuxData500G_A-flat.vmdk
-rwxrwxrwx 2 root root          559 Jan 26  2015 LinuxData500G_A.vmdk

where the problematic disks are those LinuxDataXXXG_Y-flat.vmdk

What surprises me is that... the modification date of all those is Jan-26 of 2015... exactly "before" the time-loss in the content of the disks... I can't see any file from feb-2015 but I do see files from jan-2015.

I wonder if instead of being a problem with the EXT4 from inside the linux it could be a problem with the NTFS itself. What I can't tell is if the VirtualBox low-level drivers would update the time for the host files or that is "creation" time.

There is no space in the RAID to have a full-copy of the vmdks, so the lost data MUST be there somehow.

Working on an imaged-copy

I have already done an "image" of the original vmdk and I have unmounted the full RAID so I may be a bit "destructive" on the copy.

The thing is that it lasts 20 hours to copy the full backup image so I don't want to just be kidding with trial-and-error and I want to use the proper tools in the proper order.

Any failure in the process will be 1 day delay.

Questions

Assuming the data MUST be there... Assuming that from ext4 I can play around with i-node tables...

  • Q1) What is the toolset I need to try to restore i-node mappings or re-create a new i-node table from nothing?
  • Q2) Particularly, what is the order I have to run the tools? I mean, if I run fsck maybe I destroy things for extundelete or viceversa or for other tools.
  • Q3) Is it true that i-node tables are backed-up from time to time into special partition areas reserved to that? Where? May I be lucky and find a backup of the tables if not from the "last day" at least from mid nov/2016 or so?

Maybe I am asking the wrong questions...

So one ring to rule them all: Q) How can I recover my lost files?

migrated from serverfault.com Dec 5 '16 at 2:30

This question came from our site for system and network administrators.

  • 6
    You restore your important data from the backups you took. – yoonix Nov 29 '16 at 19:57
  • 1
    Maybe don't build your system to look like an intricate and delicate puzzle - serverfault is a site for professional sysadmins, who inherently build support and supportable systems - we also test our backups and run DR trials routinely too. This site may not be for you. – Chopper3 Nov 29 '16 at 19:59
  • 1
    A RAID 5 of USB hard drives is truly frightening. I'm a little surprised it lasted two years, especially with all the warnings it gave you over that time. If you want any hope of recovering anything, contact a professional data recovery company immediately...but don't hold much hope. – Michael Hampton Nov 29 '16 at 20:29
  • 2
    That's a hilarious daisy-chain of storage you have there. Stop using it immediately so your data can actually be recovered, and submit the pile of abstractions to a data recovery professional. – Spooler Nov 29 '16 at 20:31
  • Sorry about the posting. I wrongly thought SF was for general system administration. I see now that home and devel environments are explicitly off-topic. I've posted a question on meta for help on migration: meta.serverfault.com/questions/8969/migrate-question-to-ul – Xavi Montero Dec 5 '16 at 0:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.