10

Consider a 100MB raw block device as a simple example. That is 204800 blocks of 512 bytes each for a total of 102760448 bytes.

The challenge is to shift the first 98MB (200704 blocks) so there is a gap of 2MB (4096 blocks) in front of it. To do this in-place requires that nothing is written to a sector that has not been read. One way to achieve this is to introduce a buffer:

$ dd if=/dev/sdj2 count=200704 | mbuffer -s 512 -b 4096 -P 100 | dd of=/dev/sdj2 seek=4096

The expectation is that mbuffer will store 4096 blocks before passing anything to the writer, thus ensuring that nothing is written to an area that has not been read and that the writer lags the reader by the size of the buffer. The buffer should allow the reader and writer to operate as fast as possible within those constriants.

However, it doesn't seem to work reliably. I've tried using real devices but it never works on them, whereas experiments with a file worked on my 64-bit box but not on my 32-bit box.

First, some preparation:

$ dd if=/dev/sdj2 count=200704 | md5sum
0f0727f6644dac7a6ec60ea98ffc6da9
$ dd if=/dev/sdj2 count=200704 of=testfile

This doesn't work:

$ dd if=/dev/sdj2 count=200704 | mbuffer -s 512 -b 4096 -P 100 -H | dd of=/dev/sdj2 seek=4096
summary: 98.0 MiByte in  4.4sec - average of 22.0 MiB/s
md5 hash: 3cbf1ca59a250d19573285458e320ade

This works on 64-bit system but not on 32-bit system:

$ dd if=testfile count=200704 | mbuffer -s 512 -b 4096 -P 100 -H | dd of=testfile seek=4096 conv=notrunc
summary: 98.0 MiByte in  0.9sec - average of  111 MiB/s
md5 hash: 0f0727f6644dac7a6ec60ea98ffc6da9

How can this be done reliably?


notes

I have read other questions about buffering and looked at pv, buffer and mbuffer. I could only get the latter to work with the required buffer size.

Using intermetiate storage is an obvious solution to the problem that always works but it isn't practical when sufficient spare capacity isn't available.

Test platforms running Arch Linux with mbuffer version 20140302.

  • I don't suppose it would solve the problem, but out of curiosity why use mbuffer at all? Why not instead make dd read the entire contents of the block device in one go using dd bs=102760448? Of course, one way or the other it's buffered in RAM. – Celada Nov 13 '14 at 16:50
  • @Celada - the 100MB example was just an example. Reading 1TB, for example, in one go wouldn't be such a good idea. – starfry Nov 13 '14 at 16:58
  • 2
    Ah, I understand now, thanks. The mbuffer should actually force the second dd to lag behind for first and you only need enough RAM to buffer the size of the shift. Too bad dd doesn't support reading and writing blocks in backwards order since that would eliminate the problem! – Celada Nov 13 '14 at 17:08
  • You didn't list how you computed the second md5sum – psusi Nov 14 '14 at 16:13
  • @psusi, the second md5 is output by mbuffer (its -H argument enables this feature). – starfry Nov 14 '14 at 17:35
2

Without a buffer, you could go backwards, one block at a time.

for i in $(seq 100 -1 0)
do
    dd if=/dev/thing of=/dev/thing \
       bs=1M skip=$i seek=$(($i+2)) count=1
done

Please note that this example is dangerous due to lack of error checking.

It's also slow due to the amount of dd calls. If you have memory to spare, you could use a larger blocksize.

With a buffer, beware pitfalls. It is not sufficient to guarantee a 100% prefill. What you need is a minimum fill throughout the entire process. The buffer must never ever drop below 2M because otherwise you will have overwritten your yet-to-be-read data again.

So while in theory you could do without any kind of buffer and just chain dd:

dd if=/dev/thing bs=1M | \
dd bs=1M iflag=fullblock | \
dd bs=1M iflag=fullblock | \
dd of=/dev/thing bs=1M seek=2

In practice this does not work reliably because there is no guarantee the first dd manages to keep reading data, while the last dd (with 2M of "buffer" in between) is already writing.

You can increase your chances considerably by making the in between buffer considerably larger, but even so, it's not reliable.

Unfortunately I do not know a good buffer program with minimum fill property. You need one that stops output as long as there is less than your safety margin within the buffer.

  • I accepted this because it answers the original question by demonstrating how dd could be used. I think, however, that the real solution is not to use dd but instead opt for something that is designed to run backwards like ddrescue. I've described a way to do that in an answer. – starfry Nov 20 '14 at 10:31
  • 1
    @starfry: sure, a program that just does it will be a nice solution. However I'm not at all sure about ddrescue here. Not if it expects to be working on different devices, and you have to trick it into accepting your arguments. It might not not have the "minimum buffer fill" property internally either (since with different devices it's not needed), so again it could corrupt your data. You'd have to check in the sourcecode whether it's actually designed for your use case. – frostschutz Nov 20 '14 at 11:10
1

A reliable solution requires that you ensure that nothing writes to an area that might not have been read and the only real way to achieve that is to perform the copy in a reverse direction.

The ddrescue tool can work in a reverse direction but it refuses to run with the input and output being the same. However it's possible to trick it by duplicating the device node.

I have performed some quick experiments and it appears to work. The command-line is:

$ ddrescue -f -R -s 200704s -o 4096s /dev/sdj11 /dev/sdj11_copy

The arguments are

  • -f is required to force it to write to an existing output device
  • -R tells it to work in a reverse direction
  • -s tells it how much of the input to copy (I used the s suffix to specify the number of sectors)
  • -o tells it to seek forwards in the output device before writing (specified in sectors again with the s suffix)
  • /dev/sdj11 is the block device to read
  • /dev/sdj11_copy is the block device to write

I created /dev/sdj11_copy with mknod to match the parameters of /dev/sdj11.

I've only done some very quick tests but this does appear to work ok to copy a raw device. It does not work on a file (I could not trick it into going beyond the files being the same)

This doesn't answer my original question which asked how to achieve this with dd but I think, having read the other answers, the answer to that is that dd cannot do it.

  • What happens if ddrescue discovers a bad block in this scenario? If it jumps to another area of the disk (to avoid bad blocks), and continues copying from there, it would again overwrite not yet copied parts of your data. If it doesn't expect to work with the same device, it has no reason to take any special measures to prevent various possible data corruption cases. – frostschutz Nov 20 '14 at 11:20
  • I agree that this is a potential issue but I haven't looked at the edge cases, since I was able to use it to do what I needed. There are ddrescue options to limit its attempts to recover bad data but I haven't looked into using them. – starfry Nov 20 '14 at 12:00
  • The fact that it refuses to operate if the input and output are the same is probably a good indication that it isn't safe. – psusi Nov 20 '14 at 16:32
0

You are reading 4096 blocks, and then writing those 4096 blocks to the next 4096 blocks of the disk, thus overwriting the second 4096 blocks before they can be read. You need to read 8129 blocks to get those second 4096 before starting any writing, and then you need to only write 4096 blocks before reading the next 4096.

You didn't mention what kind of filesystem this is. If it is ext[234], and you have a recent version of e2fsprogs, then you can use e2image -ra -O 512 /dev/sdj2. This also has the added benefit of being smart enough to skip the free space in the volume.

  • That makes sense when reading it and I'm going to take another look based upon that. But it doesn't explain why it worked on the test file. – starfry Nov 14 '14 at 17:37
  • Re the filesystem, are you referring to the filesystem containing my test file? That's ext4 but for the block device copy, any filesystem should be irrelevant. – starfry Nov 14 '14 at 17:37
  • @starfry, the only way I know of to do this in a generic way is to use the algorithm Emmanuel suggested ( work backwards from the end ), which is what gparted does. – psusi Nov 14 '14 at 18:26
  • re the block size, I had tried larger blocks (I should have written that in the question). I found that it didn't become more reliable even a 64K sector buffer. The reliable solution is to run backwards, something that dd doesn't do. – starfry Nov 20 '14 at 10:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.