2

This question already has an answer here:

I'm writing some images back to block devices, with dd, and getting very strange output I've never seen before.

# xz -dc goren.img.xz | dd bs=1M of=/dev/storage2/goren
35+2475166 records in
35+2475166 records out
21474836480 bytes (21 GB) copied, 222.912 s, 96.3 MB/s
# xz -dc gronn.img.xz | dd bs=1M of=/dev/storage2/gronn
50+2413782 records in
50+2413782 records out
21474836480 bytes (21 GB) copied, 233.478 s, 92.0 MB/s
# xz -dc grummle.img.xz | dd bs=1M of=/dev/storage2/grummle
63+2443466 records in
63+2443466 records out
21474836480 bytes (21 GB) copied, 222.898 s, 96.3 MB/s
# xz -dc hozen.img.xz | dd bs=1M of=/dev/storage2/hozen
19+2556787 records in
19+2556787 records out
21474836480 bytes (21 GB) copied, 250.989 s, 85.6 MB/s

The output I expected to see in each case (and what I got when creating the image files) is:

20480+0 records in
20480+0 records out

As far as I can tell the images are being written correctly, though I am concerned with the number of records shown. This is obviously not correct in any of the cases. Though, as I said, the written images match the originals, pass filesystem checks, etc.

I'm using Fedora 21 x86_64 with coreutils 8.22.

marked as duplicate by Gilles linux May 6 '15 at 22:43

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • I suspect dd isn't reading 1M blocks reliably from the pipe but I'm too busy to dig up the source code and look. Try ... | dd ibs=1024 obs=1m ... to change the input block size to a smaller value and see what you get. – Andrew Henle May 6 '15 at 21:39
  • 1
    avoid dd with pipes – Stéphane Chazelas May 6 '15 at 21:56
  • 2
    Use the right tool for the right job. If the job doesn't involve tapes or seeking, dd isn't the right tool. – Gilles May 6 '15 at 22:43
  • @Gilles What relevance is the marked duplicate to this question? It discusses a completely different issue. – Michael Hampton May 6 '15 at 22:53
  • @MichaelHampton Same issue: dd does not, in general, just copy its input to its output. It copies bits and pieces, here and there, in a difficult-to-control fashion, unless you use the right parameters (absolutely avoid bs) with the right file types (plain files are block devices can be ok, forget about pipes and character devices). – Gilles May 6 '15 at 22:54
4

Those are incomplete reads. It should go away if you add iflag=fullblock.

By default, dd will happily accept smaller blocks from a pipe, if there isn't more data readily available. With the iflag, dd will wait until a full block of data has been gathered, or EOF.

In regards to data consistency there should be no issue, so you should be getting correct results either way.

The question is why are you using dd at all, your example can just as well be reduced to:

xz -dc goren.img.xz > /dev/storage2/goren
  • Because > gives me no feedback. And this will eventually be part of a larger backup script. – Michael Hampton May 6 '15 at 21:39
  • 1
    @MichaelHampton perhaps consider pv instead of dd – roaima May 6 '15 at 22:35
  • Hm. iflag=fullblock makes dd report the expected numbers, but then it's almost 20% slower. – Michael Hampton May 6 '15 at 22:52
  • With a larger block size and iflag=fullblock there will be longer gaps between writes to disk hence the slow down. I see no need for fullblock here. Note the next version of dd (version 8.24) will have a status=progress option to update the status line every second. – Pádraig Brady May 7 '15 at 9:26
  • @MichaelHampton you could try bs=64k or something and see if that helps ... dd blocksize is a bit of a mystery, see also unix.stackexchange.com/a/183920/30851 – frostschutz May 7 '15 at 10:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.