I have a system image sparse file which actual size is only a few MBs but the "apparent size" is about 1GB. I'm trying to write it to a block device efficiently (without the holes). Here are some none working solutions I've tried:

  • dd if=sparse_file of=/dev/some_dev processes the whole file including the holes, so at the end I'm getting something like 1007869952 bytes (1,0 GB) copied, 22,0301 s, 45,7 MB/s
  • cp --sparse=always sparse_file /dev/some_dev seems also is not working as it takes long time for few MBs (~13s)
  • ddrescue --sparse --force sparse_file /dev/some_dev fails with a message ddrescue: Only regular files can be sparse. (Note: it works in the opposite direction as covered here).

There are 2 other ways covered here but I'd like to use only the standard tools which are the part of Linux distribution.

So is there a way to write the sparse file to a block image skipping the holes?


This tool written in C may be helpful for you. It is not a standard tool, but it is very simple and easy to compile.

You are right that cp doesn't support sparse output to block devices (confirmed in man page).

| improve this answer | |
  • 1
    This tool doesn't skip holes, rather it skip zeroes. So target device should be zeroed before(otherwise significant zeroes will be lost). – Yuriy Nazarov Aug 31 '15 at 15:46

I encountered a problem that is related to this thread: I created a LUKS crypto block device with integrity. I'd want to wipe each sector (i.e. logical block) so the sector's integrity checks, before I could read from the sector. Upon "luksFormat", cryptsetup(8) offers to wipe all sectors by default. If I'd overwrite a sector, the sector's integrity bits'd be overwritten, too, so I'd no need to care whether its previous integrity checks. However, if a partial write'd cause the kernel to read this sector, and possibly along with neighbor sectors within page size, and the related integrity'd fail to check, the write'd fail. Unfortunately, mke2fs writes partial sectors. So I could not mke2fs on a not wiped such device. So I wrote this "sparseblk" utility. I could create a fs image on a plain file, then write the plain file to the device. This "sparseblk" utility skips holes in its input by lseek(fd, offset, SEEK_DATA), and pads data to the sector boundary when writing. The kernel writes whole sectors after the fs is mounted.

hypothetical scenario: https://gitlab.com/cryptsetup/cryptsetup/issues/335#note_270050959

truncate -s 1G /tmp/1.img /tmp/2.img
truncate -s 980M /tmp/1.img /tmp/2.img
mke2fs /tmp/2.img
cryptsetup luksFormat \
    --key-file /etc/motd \
    --sector-size $(getconf PAGESIZE) \
    --cipher chacha20-plain \
    --integrity poly1305 \
    --integrity-no-wipe \
cryptsetup open --key-file /etc/motd /tmp/1.img dm-0
argv[0] < /tmp/2.img > /dev/mapper/dm-0
mount /dev/mapper/dm-0 /mnt/1

I've not tested it with e2fsck and resize2fs, etc., yet.

Anyway, if I'd be serious about security, I'd wipe the whole device.

You may use this "sparseblk" utility for other purposes. You may ignore the "logical block size not a multiple of page size" warning if you use this "sparseblk" utility for other purposes.

posix is adopting SEEK_DATA SEEK_HOLE: http://austingroupbugs.net/view.php?id=415

"sparseblk": https://gist.github.com/insulsa/18b7d31bd82ddade14db07f413c0b2d2

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.