Hot answers tagged

131

Since the servers are physically next to each other, and you mentioned in the comments you have physical access to them, the fastest way would be to take the hard-drive out of the first computer, place it into the second, and transfer the files over the SATA connection.


107

In appearance, dd is a tool from an IBM operating system that's retained its foreign appearance (its parameter passing), which performs some very rarely-used functions (such as EBCDIC to ASCII conversions or endianness reversal… not a common need nowadays). I used to think that dd was faster for copying large blocks of data on the same disk (due to more ...


105

You can send dd a certain signal using the kill command to make it output its current status. The signal is INFO on BSD systems (including OSX) and USR1 on Linux. In your case: kill -INFO $PID You can find the process id ($PID above) with the ps command; or see pgrep and pkill alternatives on mac os x for more convenient methods. More simply, as AntoineG ...


67

netcat is great for situations like this where security is not an issue: # on destination machine, create listener on port 9999 nc -l 9999 > /path/to/outfile # on source machine, send to destination:9999 nc destination_host_or_ip 9999 < /dev/sda # or dd if=/dev/sda | nc destination_host_or_ip 9999 Note, if you are using dd from GNU coreutils, you ...


60

Let's try it. Here's a trivial C program: #include <stdio.h> int main(int argc, char **argv) { puts("/usr/tmp"); } We'll build that into test: $ cc -o test test.c If we run it, it prints "/usr/tmp". Let's find out where "/usr/tmp" is in the binary: $ strings -t d test | grep /usr/tmp 1460 /usr/tmp -t d prints the offset in decimal into ...


51

Under OS X (didn't try on Linux), you can simply type Ctrl+T in the terminal running dd. It will print the same output as kill -INFO $PID, plus the CPU usage: load: 1.40 cmd: dd 34536 uninterruptible 3.49u 64.58s 5020305+0 records in 5020304+0 records out 2570395648 bytes transferred in 4284.349974 secs (599950 bytes/sec) I found out about it reading ...


40

You can switch bs and skip options: dd bs=1131 skip=1 if=filtered.dump of=trimmed.dump This way the operation can benefit from a greater block. Otherwise, you could try with tail (although it's not safe to use it with binary files): tail -c +1132 filtered.dump >trimmed.dump Finally, you may use 3 dd instances to write something like this: dd ...


37

There's but one way to determine the optimal block size, and that's a benchmark. I've just made a quick benchmark. The test machine is a PC running Debian GNU/Linux, with kernel 2.6.32 and coreutils 8.5. Both filesystems involved are ext3 on LVM volumes on a hard disk partition. The source file is 2GB (2040000kB to be precise). Caching and buffering are ...


31

It depends on whether the disk image is a full disk image, or just a partition. Washing the partition(s) If the disk is in good working condition, you will get better compression if you wash the empty space on the disk with zeros. If the disk is failing, skip this step. If you're imaging an entire disk then you will want to wash each of the partitions on ...


26

There are several limitations that could be limiting the transfer speed. There is inherent network overhead on a 1Gbps pipe. Usually, this reduces ACTUAL throughput to 900Mbps or less. Then you have to remember that this is bidirectional traffic and you should expect significantly less than 900Mbps down. Even though you're using a "new-ish router" are you ...


25

As far as the end result is concerned, they will do the same. The difference is in how dd would process data. And actually, both your examples are quite extreme in that regard: the bs parameter tells dd how much data it should buffer into the memory before outputting it. So, essentially, the first command would try to read 2GB in two chunks of 1GB, and the ...


23

If your intent is to backup a remote computer's HDD A via SSH to a single file that's on your local computer's HDD, you could do one of the following. Examples run from remote computer $ dd if=/dev/sda | gzip -1 - | ssh user@local dd of=image.gz run from local computer $ ssh user@remote "dd if=/dev/sda | gzip -1 -" | dd of=image.gz Live example $ ...


22

For dd, you can send a signal. For other commands that are reading or writing to a file, you can watch their position in the file with lsof. lsof -o -p1234 # where 1234 is the process ID of the command lsof -o /path/to/file If you plan in advance, pipe the data through pv.


22

When you use dd on /dev/sdb instead of /dev/sdb1 or /dev/sdb2, you copy all the partitions from the said drive into one file. You must mount each partition separately. To mount a partition from a file, you must first find out where in the file that partition resides. Using your output from file -s sdb.img we find the startsectors for each partition: ...


22

You're observing a combination of the peculiar behavior of dd with the peculiar behavior of Linux's /dev/random. Both, by the way, are rarely the right tool for the job. Linux's /dev/random returns data sparingly. It is based on the assumption that the entropy in the pseudorandom number generator is extinguished at a very fast rate. Since gathering new ...


20

If you use the conv=notrunc argument, you can replace just the first however many bytes. e.g. dd conv=notrunc if=small.img of=large.img root@debian:~/ddtest# dd if=/dev/zero of=file1.img bs=1M count=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.14556 s, 9.2 MB/s root@debian:~/ddtest# dd if=/dev/urandom of=file2.img bs=1M count=1 1+0 ...


19

Though the “best answer” was given, this site states otherwise: Actually, it stands for ‘Copy and Convert’ and was renamed to dd only because cc was reserved for the C compiler! This is the authentic information I got from the man pages of our Unix-V7 on our university PDP 11.


18

The difference between cat <input >output and dd if=input of=output is in the block size they will use. The default for dd if no bs parameter is given is 512 bytes (the size of a disk sector), whereas cat will use whatever blocksize it is coded to work with, which based on a quick inspection of the source code is either 32 KB or the st_blksize value ...


17

The dd command includes LOTS of options that cat is not able to accommodate. Perhaps in your usage cases cat is a workable substitute, but it is not a dd replacement. One example would be using dd to copy part of something but not the whole thing. Perhaps you want to rip out some of the bits from the middle of an iso image or the partition table from a hard ...


17

The result will be the same but in the first case dd will write two 1GB blocks while in the second one 2GB block. The difference is that dd keeps the copied block in memory. You will need 1GB of RAM in the first case and 2GB in the second. In my opinion there is no need to use such large blocks. You can do a couple of tests but in my case I achieve a ...


17

Did you compare their contents immediately after writing the duplicated contents? If yes, they should come out exactly the same. For example, # Duplicate dd bs=16M if=/dev/sdg of=/dev/sdk # Comparing should produce no output cmp /dev/sdg /dev/sdk # Compare, listing each byte difference; also no output cmp -l /dev/sdg /dev/sdk This is only true if the ...


16

dd dates from back when it was needed to translate old IBM mainframe tapes, and the block size had to match the one used to write the tape or data blocks would be skipped or truncated. (9-track tapes were finicky. Be glad they're long dead.) These days, the block size should be a multiple of the device sector size (usually 4KB, but on very recent disks ...


16

Wikipedia (dd) asserts it was named after IBM JCL command DD which stands for Data Description. I always thought it would mean data duplicate, though.


16

If you check Reading from /dev/urandom gives EOF after 33554431 bytes and follow the discussion, it points to another bug report where Ted Tso states... ...that commit 79a8468747c5 causes reads larger than 32MB results in a only 32MB to be returned by the read(2) system call. That is, it results in a short read. POSIX always allows for a short ...


15

dd was useful in the old days when people used tapes (when block sizes mattered) and when simpler tools such as cat might not be binary-safe. Nowadays, dd if=/dev/sdb of=/dev/sdc is a just complicated, error-prone, slow way of writing cat /dev/sdb >/dev/sdc. While dd still useful for some relatively rare tasks, it is a lot less useful than the number of ...


14

No one has yet mentioned that you can use dd to create sparse files, though truncate can also be used for the same purpose. dd if=/dev/zero of=sparse-file bs=1 count=1 seek=10GB This is almost instant and creates an arbitrary large file that can be used as a loopback file for instance: loop=`losetup --show -f sparse-file` mkfs.ext4 $loop mkdir myloop ...


14

You want dd_rescue. dd_rescue -a -b 8M /dev/sda1 /mount/external/backup/sda1.raw


14

Well, assuming you have stat and bash, you can get the file size with: stat -c %s your_file If you want to extract the last $amount bytes for that file with dd, you could: dd if=your_file of=extracted_part \ bs=1 count=$amount \ skip=$(( $(stat -c %s your_file) - $amount )) But the saner approach would be to use tail: tail -c $(( 1024*1024 )) ...


14

/dev/null is a special file, of type character device. The driver for that character device ignores whatever you try to write to the device and writes are always successful. If a write to /dev/null fails, it means that you've somehow managed to remove the proper /dev/null and replace it by a regular file. You might have accidentally removed /dev/null; then ...



Only top voted, non community-wiki answers of a minimum length are eligible