Hot answers tagged

156

In appearance, dd is a tool from an IBM operating system that's retained its foreign appearance (its parameter passing), which performs some very rarely-used functions (such as EBCDIC to ASCII conversions or endianness reversal… not a common need nowadays). I used to think that dd was faster for copying large blocks of data on the same disk (due to more ...


139

Since the servers are physically next to each other, and you mentioned in the comments you have physical access to them, the fastest way would be to take the hard-drive out of the first computer, place it into the second, and transfer the files over the SATA connection.


109

If your intent is to backup a remote computer's HDD A via SSH to a single file that's on your local computer's HDD, you could do one of the following. Examples run from remote computer $ dd if=/dev/sda | gzip -1 - | ssh user@local dd of=image.gz run from local computer $ ssh user@remote "dd if=/dev/sda | gzip -1 -" | dd of=image.gz Live example $ ssh ...


101

Under OS X (didn't try on Linux), you can simply type Ctrl+T in the terminal running dd. It will print the same output as kill -INFO $PID, plus the CPU usage: load: 1.40 cmd: dd 34536 uninterruptible 3.49u 64.58s 5020305+0 records in 5020304+0 records out 2570395648 bytes transferred in 4284.349974 secs (599950 bytes/sec) I found out about it reading ...


72

Let's try it. Here's a trivial C program: #include <stdio.h> int main(int argc, char **argv) { puts("/usr/tmp"); } We'll build that into test: $ cc -o test test.c If we run it, it prints "/usr/tmp". Let's find out where "/usr/tmp" is in the binary: $ strings -t d test | grep /usr/tmp 1460 /usr/tmp -t d prints the offset in decimal into the ...


70

netcat is great for situations like this where security is not an issue: # on destination machine, create listener on port 9999 nc -l 9999 > /path/to/outfile # on source machine, send to destination:9999 nc destination_host_or_ip 9999 < /dev/sda # or dd if=/dev/sda | nc destination_host_or_ip 9999 Note, if you are using dd from GNU coreutils, you ...


65

It depends on whether the disk image is a full disk image, or just a partition. Washing the partition(s) If the disk is in good working condition, you will get better compression if you wash the empty space on the disk with zeros. If the disk is failing, skip this step. If you're imaging an entire disk then you will want to wash each of the partitions on ...


58

dd was useful in the old days when people used tapes (when block sizes mattered) and when simpler tools such as cat might not be binary-safe. Nowadays, dd if=/dev/sdb of=/dev/sdc is a just complicated, error-prone, slow way of writing cat /dev/sdb >/dev/sdc. While dd still useful for some relatively rare tasks, it is a lot less useful than the number of ...


52

It's because you're doing it wrong. You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read. And then you have conv=sync which complements incomplete reads with zeroes. 0+15281 records in 15280+0 records out dd received 0 full and 15281 incomplete reads, and wrote 15280 full ...


50

dd if=boot1h of="/dev/r$temp1" status=none from dd (coreutils) 8.21 docs: 'status=LEVEL' Transfer information is normally output to stderr upon receipt of the 'INFO' signal or when 'dd' exits. Specifying LEVEL will adjust the amount of information printed, with the last LEVEL specified taking precedence. 'none' Do not ...


39

Apple court, Apple rules. Try diskutil: $ diskutil list ... # if mounted somewhere $ sudo diskutil unmount $device # all the partitions (there's also a "force" option, see the manual) $ sudo diskutil unmountDisk $device # remember zip drives? this would launch them. good times! $ sudo diskutil eject $device (In the case of a disk image, the hdiutil ...


38

dd if=/dev/zero of=/dev/sda bs=4096 count=4096 Q: why 4096 is particularly used for counter? This will zero out the first 16 MiB of the drive. 16 MiB is probably more than enough to nuke any "start of disk" structures while being small enough that it won't take very long. dd if=/dev/zero of=/dev/sda bs=512 count=4096 seek=$(expr blockdev --getsz /dev/...


36

If you use the conv=notrunc argument, you can replace just the first however many bytes. e.g. dd conv=notrunc if=small.img of=large.img root@debian:~/ddtest# dd if=/dev/zero of=file1.img bs=1M count=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.14556 s, 9.2 MB/s root@debian:~/ddtest# dd if=/dev/urandom of=file2.img bs=1M count=1 1+0 ...


35

Summary: dd is a cranky tool which is hard to use correctly. Don't use it, despite the numerous tutorials that tell you so. dd has a “unix street cred” vibe attached to it — but if you truly understand what you're doing, you'll know that you shouldn't be touching it with a 10-foot pole. dd makes a single call to the read system call per block (defined by ...


33

Do use fast compression. Whatever your transfer medium - especially for network or usb - you'll be working with data bursts for reads, caches, and writes, and these will not exactly be in sync. Besides the disk firmware, disk caches, and kernel/ram caches, if you can also employ the systems' CPUs in some way to concentrate the amount of data exchanged per ...


29

Well, assuming you have stat and bash, you can get the file size with: stat -c %s your_file If you want to extract the last $amount bytes for that file with dd, you could: dd if=your_file of=extracted_part \ bs=1 count=$amount \ skip=$(( $(stat -c %s your_file) - $amount )) But the saner approach would be to use tail: tail -c $(( 1024*1024 )) ...


29

You're observing a combination of the peculiar behavior of dd with the peculiar behavior of Linux's /dev/random. Both, by the way, are rarely the right tool for the job. Linux's /dev/random returns data sparingly. It is based on the assumption that the entropy in the pseudorandom number generator is extinguished at a very fast rate. Since gathering new ...


26

As far as the end result is concerned, they will do the same. The difference is in how dd would process data. And actually, both your examples are quite extreme in that regard: the bs parameter tells dd how much data it should buffer into the memory before outputting it. So, essentially, the first command would try to read 2GB in two chunks of 1GB, and the ...


26

All of the following commands are equivalent. They read the bytes of the CD /dev/sr0 and write them to a file called image.iso. cat /dev/sr0 >image.iso cat </dev/sr0 >image.iso tee </dev/sr0 >image.iso dd </dev/sr0 >image.iso dd if=/dev/cdrom of=image.iso pv </dev/sr0 >image.iso cp /dev/sr0 image.iso tail -c +1 /dev/sr0 >image....


25

There are several limitations that could be limiting the transfer speed. There is inherent network overhead on a 1Gbps pipe. Usually, this reduces ACTUAL throughput to 900Mbps or less. Then you have to remember that this is bidirectional traffic and you should expect significantly less than 900Mbps down. Even though you're using a "new-ish router" are you ...


24

I really don't know how to explain this better than the manpage does. bs= sets the blocksize, for example bs=1M would be 1MiB blocksize. count= copies only this number of blocks (the default is for dd to keep going forever or until the input runs out). Ideally blocks are of bs= size but there may be incomplete reads, so if you use count= in order to copy a ...


23

dd has many (weird) options, see dd(1). You should explicitly state the buffer size, so try dd if=/dev/sda of=/dev/sdb bs=16M IIRC, the default buffer size is only 512 bytes. The command above sets it to 16 megabytes. You could try something smaller (e.g. bs=1M) but you should use more than the default (especially on recent disk hardware with sectors of ...


23

Will dd if=/dev/zero of=/dev/sda wipe out a pre-existing partition table? Yes, the partition table is in the first part of the drive, so writing over it will destroy it. That dd will write over the whole drive if you let it run (so it will take quite some time). Something like dd bs=512 count=50 if=/dev/zero of=/dev/sda would be enough to overwrite the ...


21

The dd command includes LOTS of options that cat is not able to accommodate. Perhaps in your usage cases cat is a workable substitute, but it is not a dd replacement. One example would be using dd to copy part of something but not the whole thing. Perhaps you want to rip out some of the bits from the middle of an iso image or the partition table from a hard ...


21

You want dd_rescue. dd_rescue -a -b 8M /dev/sda1 /mount/external/backup/sda1.raw


20

No one has yet mentioned that you can use dd to create sparse files, though truncate can also be used for the same purpose. dd if=/dev/zero of=sparse-file bs=1 count=1 seek=10GB This is almost instant and creates an arbitrary large file that can be used as a loopback file for instance: loop=`losetup --show -f sparse-file` mkfs.ext4 $loop mkdir myloop ...


20

dd is designed to copy blocks of data from an input file to an output file. The dd block size options are as follows, from the man page: ibs=expr Specify the input block size, in bytes, by expr (default is 512). obs=expr Specify the output block size, in bytes, by expr (default is 512). bs=expr Set both input and output block sizes to expr bytes,...


20

This is slow because of the small block size. Using a recent GNU dd (coreutils v8.16 +), the simplest way is to use the skip_bytes and count_bytes options: in_file=1tb start=12345678901 end=19876543212 block_size=4096 copy_size=$(( $end - $start )) dd if="$in_file" iflag=skip_bytes,count_bytes,fullblock bs="$block_size" \ skip="$start" count="$...


20

If you want to overwrite only at the start of the file, and leave the rest intact, use the conv=notrunc option to prevent truncation: dd conv=notrunc if=/dev/zero of=test bs=1024 count=1024 notrunc means: Do not truncate the output file. Preserve blocks in the output file not explicitly written by this invocation of the dd utility. It is in POSIX and ...


20

Use unzip -p: unzip -p 2015-11-21-raspbian-jessie.zip 2015-11-21-raspbian-jessie.img | dd of=/dev/sdb bs=1M


Only top voted, non community-wiki answers of a minimum length are eligible