1

TL;DR

I have a dd image disk.dd which has multiple partitions. The end goal is to reduce the size of this dd image file.

After deleting and recreating a higher numbered partition with a start sector offset lower than it was before, (i.e expanding the partition to the left) I have a partition which has a filesystem in it and whose primary superblock is somewhere inside this partition, and I know the sector at which this primary superblock resides.

How can I e2fsck this filesystem so that it moves to the beginning of the partition ?

So that afterwards I can shrink this filesystem with resize2fs and then shrink this partition from right, i.e (recreating this partition with a lower end sector offset)

Then I'll repeat this process with the partitions after that until the last partition, effectively shrinking all partitions and hence reducing the size of dd image

Please do not suggest gparted . I'm looking for a command line solution

Also, I know this would've been easier with LVM . But this legacy system


Long version


I have a dd image disk.dd that I took using the following

dd if=/dev/sda of=/path/to/disk.dd

of a system which has the following layout

Disk /dev/loop15: 465.78 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x54093dd5

Device        Boot     Start       End   Sectors   Size Id Type
/dev/loop15p1 *         2048  81922047  81920000  39.1G 83 Linux
/dev/loop15p2       81922048 143362047  61440000  29.3G 82 Linux swap / Solaris
/dev/loop15p3      143362048 163842047  20480000   9.8G 83 Linux
/dev/loop15p4      163842048 976773167 812931120 387.7G  5 Extended
/dev/loop15p5      163844096 976773119 812929024 387.7G 83 Linux

Now, on a different system, I'm accessing disk.dd through a loop device using

losetup --find --partscan disk.dd

I resized all of the ext4 filesystems with

resize2fs -M /dev/loopNpartX
resize2fs /dev/loopNpartX FSsize

i.e the partitions p1, p3 and p5

With dumpe2fs, I can see logical block size of filesystem which is 4096 bytes for all ext4 filesystems which in my case as I shown above are hosted on 3 partitions

Now if I'm verbally reading this correctly (correct me if I'm wrong here)

The primary superblock of a filesystem is "usually expected" to be located at block 0 of the partition

So, I can dump superblock information with

dumpe2fs -h -o superblock=0 -o blocksize=4096 /dev/loopNpartX

Now it's time to shrink partitions in order to reduce the size of disk.dd file


I got the block count for each file system again using dumpe2fs

fdisk works on physical block size OR sectors of device which in my case is 512 bytes

So, in order to find how many sectors should be enough to accommodate the filesystem, I used the following formula

Required Sectors = ( ( Block Count + 100 ) * Logical Block Size ) / Physical Block Size

100 acting as a buffer just in case I'm missing something about the organization of filesystem which should be enough

I did this for every filesystem


Now


With lsblk -f, I get the UUIDs of existing filesystems

With fdisk -l, I get which partition to keep the boot flag on

Now to shrink partitions, I would delete and recreate them using fdisk

-- First partition

start sector offset = 2048
last sector offset  = 2048 + "Required Sectors" for this filesystem

-- Second partition

Second partition on existing disk is swap, so I'll not shrink it, just move it left

start sector offset = "last sector offset" of first partition + 1
last sector offset  = "start sector offset" + Total sectors as as on existing partition

I then change it's type to Swap And then with tune2fs -U change the UUID back to what was on dd image

-- Third partition

start sector offset = "last sector offset" of second partition + 1
last sector offset  = "start sector offset" + "Required Sectors" for this filesystem

Here is where I'm stuck


After expanding third partition to the left, this partition has a filesystem whose starting sector I know (i.e sector having the primary superblock)

But I don't know how to e2fsck this filesystem to correct it on the partition so that the filesystem is moved left to the beginning of the partition

1
  • Would it not be easier to dd the partitions to separate images, shrink them separately and then rebuild the disk from these separate shrunk partitions? Commented Sep 6, 2021 at 14:01

1 Answer 1

1

It's not possible with fsck. In a filesystem, everything has offsets and if you change the start sector, all of these offsets change. fsck simply has no facility to re-write all offsets for everything (superblocks, journals, directories, file segments, etc.). And even if you could do that, it would only work if the new start sector aligns with internal filesystem structures.

So this is not done.

Instead, you'd have to shift all data to the left with dd (essentially what gparted does). Only by shifting the filesystem entirely, would the offsets within it remain intact.

In principle the dd command could work like this. It reads and writes to the same device, at different offsets. This can only work for shifting to the left, so seek (write to) must be smaller than skip (read from). All units in 512b sectors (if you specify bs=1M, your partitions must be MiB aligned and all units in MiB instead)

dd if=/dev/sdx of=/dev/sdx \
   seek=newpartitionstart \
   skip=oldpartitionstart \
   count=filesystemsize

However, this is very dangerous. Use it at your own risk. Do take the time to backup your data first.

Shifting to the right would be more complicated. You'd have to work backwards, otherwise you overwrite data that has yet to be read, and corrupt everything in the process.


The only tool I know that does it (more or less) without shifting data is blocks --lvmify, which achieves it by converting the existing filesystem partition to LVM. With LVM, you can logically expand to the right while it's physically stored on the left. Without LVM, you could also set up a linear device mapping manually, but then you are stuck with a non-standard solution.


The most sensible approach to this type of problem (if you don't want to use gparted) would be to backup all data, then make new partitions and filesystems in any layout you like, and then restore your data.

If this dd image is your approach to a backup solution, consider backing up files instead. Disk images can be hard to handle, especially if you want to transform them afterwards.


If your main goal is reduce the storage requirement of the image file, what you could do is fstrim (for loop mounted filesystem - losing all free space), or blkdiscard (for loop swap partition - losing all data).

Provided the filesystem that stores the image supports sparse files and hole punching, it would make the dd image use less storage space w/o changing any layout, as any free space within the image would also be freed for the backing filesystem.

Similarly, this is dangerous, if you discard the wrong parts of the image file, the image file is irrecoverably damaged. The simple act of creating a loop device for an image file, and mounting it, already modifies/damages the image file.

If the source disk is SSD, and it's already using fstrim regularly, and reads trimmed areas as binary ZERO, you can create an already sparse dd image in the first place using dd conv=sparse if=/dev/ssd of=ssd.img. This way any binary zero area would not take up space in the ssd.img file. Note that conv=sparse can lead to corrupt results when used in the other direction when restoring to a non-zero target drive.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .