Suddently, all the files from feb-2015 to nov-2016 have disappeared from the filesystem.
- Over the last 2 years, the system repeatedly terminated abruptly for power-fail reasons or for USB-hardware disconnections.
- This loss does not seem because of bad-blocks or things like this. It seems more something software-related.
- The disks are flat full-size
vmdkfiles on a USB hardware-based RAID-5 connected to a VirtualBox Ubuntu.
- The host holding the RAID and the VirtualBox is a Windows.
- The RAID in windows is NTFS and the VirtualBox runs Ubuntu accessing the vmdk formatted in ext4. The Windows does not do other function than just containing a bunch of virtual machines and no user enters that Windows except to launch or stop virtual machines.
I've been using those disks (from the Ubuntu) on an intensive usage on a daily basis. And I have successfully used everyday the files from the previous day, and the previous week, and the previous month, and so on... Even after reboots... So the data must be there.
But suddently... my Dropbox started to delete files, I went to see what happened, and the fact is that... I can't tell if this happened "on the fly" or after a reboot.
But the thing is that... the latest file I can see is from jan-2015 while I've successfully been working with those disks until nov-2016 every single day. The files probably are in the blocks and for some reason I've lost all the index.
I can easily imagine that for some reason, the
ext4 conserves backup-copies of i-node tables or something like this, and for some reason, the linux decided to "restore" the mapping to an old-one.
I've seen there are a bunch of tools. From running fsck to extundelete, and others.
What I see
find . -newermt 20150201
gives me zero results.
- For files I know they existed, I
cdinto the corresponding directories and I can't see the files there.
- I can see a directory structure I know I deleted.
- This has happened to several disks attached to the virtual machine.
ext4 or NTFS?
I have unmounted the RAID that contains the
vmdk virtual disks from the Windows and I have mounted it read-only on a linux.
If I list the files, I can see this:
root@vagrant:/mnt/raid5/mnt# ls -la total 1572864014 drwxrwxrwx 1 root root 4096 Jan 29 2015 . drwxrwxrwx 1 root root 4096 Oct 31 21:14 .. -rwxrwxrwx 2 root root 268435456000 Jan 26 2015 LinuxData250G_A-flat.vmdk -rwxrwxrwx 2 root root 651 Jan 26 2015 LinuxData250G_A.vmdk -rwxrwxrwx 2 root root 268435456000 Jan 26 2015 LinuxData250G_B-flat.vmdk -rwxrwxrwx 2 root root 558 Jan 26 2015 LinuxData250G_B.vmdk -rwxrwxrwx 2 root root 536870912000 Jan 26 2015 LinuxData500G_A-flat_3.vmdk -rwxrwxrwx 2 root root 536870912000 Jan 26 2015 LinuxData500G_A-flat.vmdk -rwxrwxrwx 2 root root 559 Jan 26 2015 LinuxData500G_A.vmdk
where the problematic disks are those LinuxDataXXXG_Y-flat.vmdk
What surprises me is that... the modification date of all those is Jan-26 of 2015... exactly "before" the time-loss in the content of the disks... I can't see any file from feb-2015 but I do see files from jan-2015.
I wonder if instead of being a problem with the EXT4 from inside the linux it could be a problem with the NTFS itself. What I can't tell is if the VirtualBox low-level drivers would update the time for the host files or that is "creation" time.
There is no space in the RAID to have a full-copy of the vmdks, so the lost data MUST be there somehow.
Working on an imaged-copy
I have already done an "image" of the original
vmdk and I have unmounted the full RAID so I may be a bit "destructive" on the copy.
The thing is that it lasts 20 hours to copy the full backup image so I don't want to just be kidding with trial-and-error and I want to use the proper tools in the proper order.
Any failure in the process will be 1 day delay.
Assuming the data MUST be there... Assuming that from ext4 I can play around with i-node tables...
- Q1) What is the toolset I need to try to restore i-node mappings or re-create a new i-node table from nothing?
- Q2) Particularly, what is the order I have to run the tools? I mean, if I run fsck maybe I destroy things for extundelete or viceversa or for other tools.
- Q3) Is it true that i-node tables are backed-up from time to time into special partition areas reserved to that? Where? May I be lucky and find a backup of the tables if not from the "last day" at least from mid nov/2016 or so?
Maybe I am asking the wrong questions...
So one ring to rule them all: Q) How can I recover my lost files?