23

I have 200 GB free disk space, 16 GB of RAM (of which ~1 GB is occupied by the desktop and kernel) and 6 GB of swap.

I have a 240 GB external SSD, with 70 GB used1 and the rest free, which I need to back up to my disk.

Normally, I would dd if=/dev/sdb of=Desktop/disk.img the disk first, and then compress it, but making the image first is not an option since doing so would require far more disk space than I have, even though the compression step will result in the free space being squashed so the final archive can easily fit on my disk.

dd writes to STDOUT by default, and gzip can read from STDIN, so in theory I can write dd if=/dev/sdb | gzip -9 -, but gzip takes significantly longer to read bytes than dd can produce them.

From man pipe:

Data written to the write end of the pipe is buffered by the kernel until it is read from the read end of the pipe.

I visualise a | as being like a real pipe -- one application shoving data in and the other taking data out of the pipe's queue as quickly as possible.

What when the program on the left side writes more data more quickly than the other side of the pipe can hope to process it? Will it cause extreme memory or swap usage, or will the kernel try to create a FIFO on disk, thereby filling up the disk? Or will it just fail with SIGPIPE Broken pipe if the buffer is too large?

Basically, this boils down to two questions:

  1. What are the implications and outcomes of shoving more data into a pipe than is read at a time?
  2. What's the reliable way to compress a datastream to disk without putting the entire uncompressed datastream on the disk?

Note 1: I cannot just copy exactly the first 70 used GB and expect to get a working system or filesystem, because of fragmentation and other things which will require the full contents to be intact.

  • Why would you back up a whole filesystem like that, instead of just user directories, and perhaps a list of non-standard software that's installed? – jamesqf Jun 18 '17 at 18:23
  • 5
    @jamesqf Eg. because it's much easier to restore... – deviantfan Jun 18 '17 at 18:25
  • 4
    @jamesqf Because then I also get the boot sector, and the swap partition, so that I can recreate the disk exactly instead of having a billion annoying files. – cat Jun 18 '17 at 18:26
  • 3
    Random tip: look into lzop instead of gzip; it compresses much faster with only a slighly lower compression ratio. I find it ideal for disk images where compression speed can be a real bottleneck. – marcelm Jun 18 '17 at 22:19
  • 1
    "What when the program on the left side writes more data more quickly than the other side of the pipe can hope to process it?" The kernel will cause the writing process to sleep until there is more room in the pipe. – Tavian Barnes Jun 19 '17 at 17:04
16

Technically you don't even need dd:

gzip < /dev/drive > drive.img.gz

If you do use dd, you should always go with larger than default blocksize like dd bs=1M or suffer the syscall hell (dd's default blocksize is 512 bytes, since it read()s and write()s that's 4096 syscalls per MiB, too much overhead).

gzip -9 uses a LOT more CPU with very little to show for it. If gzip is slowing you down, lower the compression level, or use a different (faster) compression method.

If you're doing file based backups instead of dd images, you could have some logic that decides whether to compress at all or not (there's no point in doing so for various file types). dar (tar alternative`) is one example that has options to do so.

If your free space is ZERO (because it's an SSD that reliably returns zero after TRIM and you ran fstrim and dropped caches) you can also use dd with conv=sparse flag to create an uncompressed, loop-mountable, sparse image that uses zero disk space for the zero areas. Requires the image file to be backed by a filesystem that supports sparse files.

Alternatively for some filesystems there exist programs able to only image the used areas.

  • 1
    "If you do use dd, you should always go with larger than default blocksize like dd bs=1M" - You can, but don't expect too much. On my PC, dd will do about 2GB/s with 512-byte blocks. That won't be the bottleneck; gzip will be. – marcelm Jun 18 '17 at 22:13
  • @marcelm We never know what kind of machine people are using. If you have dd going 2GB/s with 512-byte blocks, I would be surprised if it didn't max out one CPU core 100% in the process. Now if your box is a quadcore that just sits idle anyway, you might not notice a difference. Everybody else still does, though. – frostschutz Jun 18 '17 at 22:26
  • 9
    Sigh. Every time dd blocksize is mentioned, people come nitpicking. gzip being cpu intensive was also part of my answer, okay? And sorry, I disagree with "negligible". It might only add 1-2s per gig with gzip -9 (but that still amounts to minutes when processing hundreds of gigs) but taking your advice with lzop -1 it's 1s per gig vs. 4s per gig. Tested on a potato (single core vserver). Adding a sane blocksize to dd costs nothing and has zero downsides. Don't nitpick. Just do it. ymmv – frostschutz Jun 19 '17 at 0:02
19

dd reads and writes data one block at a time, and it only ever has one block outstanding. So

valgrind dd if=/dev/zero status=progress of=/dev/null bs=1M

shows that dd uses approximately 1MB of memory. You can play around with the block size, and drop valgrind, to see the effect on dd’s speed.

When you pipe into gzip, dd simply slows down to match gzip’s speed. Its memory usage doesn’t increase, nor does it cause the kernel to store the buffers on disk (the kernel doesn’t know how to do that, except via swap). A broken pipe only happens when one of the ends of the pipe dies; see signal(7) and write(2) for details.

Thus

dd if=... iconv=fullblock bs=1M | gzip -9 > ...

is a safe way to do what you’re after.

When piping, the writing process ends up being blocked by the kernel if the reading process isn’t keeping up. You can see this by running

strace dd if=/dev/zero bs=1M | (sleep 60; cat > /dev/null)

You’ll see that dd reads 1MB, then issues a write() which sits there waiting for one minute while sleep runs. That’s how both sides of a pipe balance out: the kernel blocks writes if the writing process is too fast, and it blocks reads if the reading process is too fast.

  • 1
    That's pretty cool. By which mechanism does dd know to slow down to match gzip's speed? It is automatic, like by the kernel, or does it calculate from metadata about its output file descriptor? – cat Jun 18 '17 at 14:57
  • 9
    @cat It's automatic; dd calls write() to put data into the pipe. write() actually transfers control to the kernel so it can manipulate the pipe memory. If the kernel sees the pipe is full, it will wait ("block") until the pipe has enough room. Only then will the write() call finish and transfer control back to dd, which will then write data to the pipe again. – marcelm Jun 18 '17 at 22:17
9

There are no negative implications other than performance: the pipe has a buffer, which is usually 64K, and after that, a write to the pipe will simply block until gzip reads some more data.

8

Answering the actual question as to how it works: "what if the program on the left side writes more data more quickly than the other side of the pipe can hope to process it?"

This doesn't happen. There is a fairly small, limited size buffer in the pipe; see How big is the pipe buffer?

Once the pipe buffer is full, the sending program blocks. When it makes a write call, the kernel will not return control to the program until the data has been written into the buffer. This gives the reading program CPU time in which to empty the buffer.

3

Maybe you only need the files, then use tar. You can fill with zeros the blocks that don't contain anything you want, somebody has already asked about it. Clear unused space with zeros (ext3,ext4)

Then, there's pigz which is usually faster than gzip.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.