2

I had a volume of 350Gb attached to my Linode server. It was running out of space so I resized it to 400GB from the dashboard and restarted my server. Then I used the df -h command to check if the disk has been resized or not but it was still showing 96% used.

So I first unmounted my volume using umount /var/www/disk and then ran this command sudo mkfs -t ext4 /dev/sdc and after I remounted my volume and used ls command to check the file but there was nothing inside except lost+found file and when I used the df -h it shows that now the disk is 400gb and out of it 20gb of disk is being used but no file can be seen except lost+found file.

I am still connected to my terminal till the time I restarted my server after resizing the volume from dashboard and I have copied the entire commands I used and their output scrolling up. Please help me to get my data back as it is extremely important. I would be very grateful of you.

root@ubuntu:/var/www/html/processing# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        78G   48G   27G  65% /
devtmpfs        2.0G  4.0K  2.0G   1% /dev
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            392M  392K  392M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sdc        345G  315G   15G  96% /var/www/disk

root@ubuntu:/var/www/html/processing# lsblk
NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb    8:16   0  1004M  0 disk [SWAP]
sdc    8:32   0   400G  0 disk /var/www/disk
sda    8:0    0    79G  0 disk /

root@ubuntu:/var/www/html/processing# sudo mkfs -t ext4 /dev/sdc
mke2fs 1.42.9 (4-Feb-2014)
/dev/sdc is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sdc is mounted; will not make a filesystem here!
root@ubuntu:/var/www/html/processing# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        78G   48G   27G  65% /
devtmpfs        2.0G  4.0K  2.0G   1% /dev
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            392M  392K  392M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sdc        345G  315G   15G  96% /var/www/disk

root@ubuntu:/var/www/html/processing# sudo mkfs -t ext4 /dev/sdc
mke2fs 1.42.9 (4-Feb-2014)
/dev/sdc is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sdc is mounted; will not make a filesystem here!

root@ubuntu:/var/www/html/processing# sudo killall php screen

root@ubuntu:/var/www/html/processing# sudo umount /disk
umount: /disk: not found

root@ubuntu:/var/www/html/processing# sudo umount /var/www/disk
umount: /var/www/disk: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

root@ubuntu:/var/www/html/processing# fuser -kim /var/www/disk
/var/www/disk:        1206
Kill process 1206 ? (y/N) y

root@ubuntu:/var/www/html/processing# sudo umount /var/www/disk

root@ubuntu:/var/www/html/processing# sudo mkfs -t ext4 /dev/sdc
mke2fs 1.42.9 (4-Feb-2014)
/dev/sdc is entire device, not just one partition!
Proceed anyway? (y,n) y
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
26214400 inodes, 104857600 blocks
5242880 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3200 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done     


root@ubuntu:/var/www/html/processing# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        78G   48G   27G  65% /
devtmpfs        2.0G  4.0K  2.0G   1% /dev
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            392M  392K  392M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user

root@ubuntu:/var/www/html/processing# lsblk
NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb    8:16   0  1004M  0 disk [SWAP]
sdc    8:32   0   400G  0 disk 
sda    8:0    0    79G  0 disk /

root@ubuntu:/var/www/html/processing# sudo mount /dev/sdc/ /var/www/disk

root@ubuntu:/var/www/html/processing# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        78G   48G   27G  65% /
devtmpfs        2.0G  4.0K  2.0G   1% /dev
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            392M  392K  392M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sdc        394G   71M  374G   1% /var/www/disk

root@ubuntu:/var/www/html/processing# cd /var/www/disk

root@ubuntu:/var/www/disk# ls
lost+found

root@ubuntu:/var/www/disk# sudo umount /var/www/disk
umount: /var/www/disk: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

root@ubuntu:/var/www/disk# fuser -kim /var/www/disk
/var/www/disk:        1573c
Kill process 1573 ? (y/N) y
Connection to 45.79.22.163 closed.

zeeshan@zeeshan-VirtualBox:~$ sudo ssh [email protected]
[email protected]'s password: 
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 5.4.10-x86_64-linode132 x86_64)

 * Documentation:  https://help.ubuntu.com/
New release '16.04.6 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Thu Feb 27 07:40:09 2020 from 119.155.0.120

root@ubuntu:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        78G   48G   27G  65% /
devtmpfs        2.0G  4.0K  2.0G   1% /dev
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            392M  392K  392M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sdc        394G   71M  374G   1% /var/www/disk

root@ubuntu:~# fuser -kim /var/www/disk

root@ubuntu:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        78G   48G   27G  65% /
devtmpfs        2.0G  4.0K  2.0G   1% /dev
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            392M  392K  392M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sdc        394G   71M  374G   1% /var/www/disk

root@ubuntu:~# sudo killall php screen
php: no process found
screen: no process found
Improve this question
1
  • Whatever you do, unmount the partition to prevent accidental write access.
    – AdminBee
    Commented Feb 27, 2020 at 10:20

1 Answer 1

Reset to default
6

I'm sorry, but it is all gone, and unless you (or your hoster) have backups, there is no way to get it back.

Recovering after mkfs is hard enough as it is, but utterly impossible when this message appears:

Discarding device blocks: done

With that, all data is gone. And this message only appears when blocks have actually been discarded. On devices that do not support discard, it is not shown at all.

mke2fs (mkfs.ext4 and others) always imply a full discard. If you do not want that to happen you have to explicitely provide the nodiscard option. From the manpage:

   -E extended-options
   [...]
               discard
                      Attempt to discard blocks at mkfs time (discarding
                      blocks initially is useful on solid state devices
                      and sparse / thin-provisioned storage). When the
                      device advertises that discard also zeroes data (any
                      subsequent read after the discard and before write
                      returns zero), then mark all not-yet-zeroed inode
                      tables as zeroed. This significantly speeds up
                      filesystem initialization. This is set as default.

               nodiscard
                      Do not attempt to discard blocks at mkfs time.

Discard is supported by SSD, but also by image files (on filesystems that support sparse files), loop devices, and commonly virtual hard drive images. This allows the VM to discard unused blocks and the host to use such unused space for other purposes. Some hosters allow you to do create snapshots of your server state as long as you have enough discarded-thus-free storage capacity. But that's usually a manual process and doesn't help you now.

Technically, the data might still exist somewhere in the "free space" of the host machine HDD which does not support discard. Or the host machine SSD if host discards happen at a later time with fstrim. So if this scenario happened in a self-hosted VPS (you control the host server) you could attempt to sift through that, but on a VPS/cloud hosting service it's impossible to get them to do it. They can not differentiate between data discarded by you or discarded by other customers, so impossibility of the task aside they'll refuse for privacy reasons. Constant write activity by other customers on the same host would also lead to your data being overwritten in reality so it's a moot point in either case.

Improve this answer
6
  • Okay but why it is still showing 20gb used space? any idea?
    – zish
    Commented Feb 27, 2020 at 10:55
  • 1
    @zish ext4 has by default a 5% root reserve, for 400GB that comes out as 20GB, you can disable the root reserve with tune2fs -m 0, that leaves you with full capacity minus regular filesystem metadata/journal/overhead.
    – frostschutz
    Commented Feb 27, 2020 at 10:56
  • to verify you can run strings -w /dev/sdc or hexdump -C --skip 20G /dev/sdc and see if there is anything other than 00 00 00 00 or ff ff ff ff patterns throughout the entire device. If the discard worked, there should be nothing. If there is data left you'd see random data and bits of text fluttering by. strings should output "lost+found" as that is the one string in a newly created ext4 filesystem that's always there. If you set a label you'd also find that. But not much else.
    – frostschutz
    Commented Feb 27, 2020 at 11:05
  • It's not all gone. I was in the same situation and was able to reciver files using photorec.
    – elip
    Commented Sep 5 at 19:34
  • But did you get the ‘‘Discarding device blocks: done’’ message?
    – G-Man Says 'Reinstate Monica'
    Commented Sep 5 at 20:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .