dd is a traditional utility for binary data copies

learn more… | top users | synonyms

47
votes
5answers
27k views

Is there a way to determine the optimal value for the bs parameter to dd?

On occasion I've seen comments online along the lines of "make sure you set 'bs=' because the default value will take too long," and my own extremely-unscientific experiences of, "well that seemed to ...
43
votes
3answers
8k views

When is dd suitable for copying data? (or, when are read() and write() partial)

Short version: In what circumstances is dd safe to use for copying data, safe meaning that there is no risk of corruption due to a partial read or write? Long version — preamble: dd is often used to ...
76
votes
7answers
15k views

dd vs cat — is dd still relevant these days?

I recently realized we can use cat as much as dd, and it's actually faster than dd I know that dd was useful in dealing with tapes where block size actually mattered in correctness, not just ...
21
votes
4answers
8k views

Why does dd from /dev/random give different file sizes?

I am running the following command on an ubuntu system: dd if=/dev/random of=rand bs=1K count=2 However, every time I run it, I end up with a file of a different size. Why is this? How can I ...
11
votes
3answers
2k views

saving data from a failing drive

An external 3½" HDD seems to be in danger of failing — it's making ticking sounds when idle. I've acquired a replacement drive, and want to know the best strategy to get the data off of the dubious ...
96
votes
12answers
78k views

How do I know if dd is still working?

I've not used dd all that much, but so far it's not failed me yet. Right now, I've had a dd going for over 12 hours - I'm writing an image back to the disk it came from - and I'm getting a little ...
11
votes
1answer
18k views

How do I mount an `img` created with /bin/dd of a hard drive?

I used dd to backup a 80GB drive dd if=/dev/sdb of=~/sdb.img Now I need to access some files on that drive, but I don't want to copy the ".img" back over the drive. mount ~/sdb.img /mnt/sdb doesn'...
11
votes
4answers
5k views

Create random data with dd and get “partial read warning”. Is the data after the warning now really random?

I create a 1TB file with random data with dd if=/dev/urandom of=file bs=1M count=1000000. Now I check with kill -SIGUSR1 <PID> the progress and get the following: 691581+0 Datensätze ein 691580+...
22
votes
5answers
12k views

Clone whole partition or hard drive to a sparse file

I like to clone a whole partition or a whole hard drive onto a larger external disk but like to create a sparse file. I often use dd for cloning, but it doesn't support sparse files. As a workaround I ...
5
votes
2answers
2k views

How to parallelize dd?

I'm currently having trouble with dd invoked with a sparse file as input (if) and a file as output (of) with conv=sparse. dd seems to be using one core of the CPU (Intel(R) Core(TM) i7-3632QM CPU @ 2....
2
votes
3answers
113 views

dd crashing - what to do about it

I have dd crashing after running a few hours. Typically with a message like Input/output error 68424+1 records in 68424+1 records out 342124883968 bytes (342 GB) copied, 5114.68 s, 66.9 MB/s Is ...
94
votes
19answers
16k views

What is the fastest way to send massive amounts of data between two computers?

This is a situation I am frequently in: I have a source server with a 320GB hard-drive inside of it, and 16GB of ram (exact specs available here, but as this is an issue I run into frequently on ...
42
votes
7answers
32k views

Best way to remove bytes from the start of a file?

Today I had to remove the first 1131 bytes from an 800MB mixed text / binary file, a filtered subversion dump I'm hacking for a new repository. What's the best way to do this? To start with I tried ...
12
votes
3answers
39k views

Full DD copy from hdd to hdd

UPDATE: 2 This seems a lot better than before (Thanks to Groxxda for the suggestions): sudo dd if=/dev/sdb bs=128K | pv -s 3000G | sudo dd of=/dev/sdc bs=128K ETA is about 9 hours for 3TB, whereas ...
25
votes
4answers
11k views

What does dd stand for?

I know very well what the command does, but man dd, info dd tell me: 'Convert and copy a file', as does GNU Coreutils. Google says its an abbreviation of everything between medicine and bad webchat ...
4
votes
1answer
1k views

Why is dd so slow with a bs of 100M [closed]

I just tried to overwrite a fast ssd using dd. Using the ubuntu boot image I typed in: dd if=/dev/zero of=/dev/sda bs=100M error writing '/dev/sda': No space left on device blah blah 256 GB copied, ...
5
votes
3answers
7k views

can 'dd' be used to clone to a smaller HDD, knowing that partitions will need editing?

I've used dd to clone disks like this: dd if=/dev/sdb of=/dev/sda bs=4096 conv=notrunc,noerror,sync And it's always worked fine. Any and all docs on 'dd' take pains to remind you that the target ...
4
votes
2answers
6k views

Check integrity of a dd backup

I just made a backup of an entire hard drive (50GB) over ssh via: dd if=/dev/hda | buffer -s 64k -S 10m | ssh myuser@myhost "cat > ~/image.img" What's now the best way to check the integrity of ...
41
votes
5answers
34k views

Is it possible to mount a gzip compressed dd image on the fly?

I like create an image backup for the first time I'm backing up a system. After this first time I use rsync to do incremental backups. My usual image backup is as follows: Mount and zero out the ...
11
votes
4answers
17k views

'seek' argument in command dd

Can some explain me what is happening in the following lines? dd if=/dev/urandom bs=4096 seek=7 count=2 of=file_with_holes especially seek part is not clear Man pages says : seek=BLOCKS ...
9
votes
5answers
7k views

Fast Way to Randomize HD?

I've read about how to make hard drives secure for encryption, and one of the steps is to write random bits to the drive, in order to make the encrypted data indistinguishable from the rest of the ...
10
votes
5answers
749 views

Is it safe to take the drive image of the current working drive?

I have to backup my hard disk. I want to use dd and put the image on an external hdd. Can I do this using dd from the OS that resides on the hdd itself or do I have to boot from another device, for ...
6
votes
2answers
6k views

How to check progress when cloning a disk using `dd`? [duplicate]

Possible Duplicate: dd: How do I know if it's still working? I'm using dd to clone a disk to an image file, and compress it on the fly: dd if=/dev/sda | gzip > /mnt/image_file How ...
13
votes
2answers
12k views

Use `dd` to cut file end part

There is probably simple trick to do this, but I can't figure from man page. How do I cut last 1MB from file with undetermined size for example, by using dd?
9
votes
3answers
345 views

How can `dd` be used to right-shift data blocks?

Consider a 100MB raw block device as a simple example. That is 204800 blocks of 512 bytes each for a total of 102760448 bytes. The challenge is to shift the first 98MB (200704 blocks) so there is a ...
8
votes
5answers
2k views

Why specify block size when copying devices of a finite size?

In online tutorials it is often suggested to use the following command to copy a CDROM to an iso image: $ dd if=/dev/dvd of=foobar.iso bs=2048 Why must the byte size be specified? I notice that in ...
5
votes
1answer
2k views

Writing raw images safely to USB sticks

Trying lots of different linux, on all kinds of hardware I find my self typing commands like this quite often: sudo dd if=xubuntu-13.10-desktop-amd64.iso of=/dev/sdc bs=10240 Needless to say, ...
14
votes
4answers
3k views

Read the middle of a large file

I have a 1 TB file. I would like to read from byte 12345678901 to byte 19876543212 and put that on standard output on a machine with 100 MB RAM. I can easily write a perl script that does this. ...
6
votes
6answers
2k views

How to write repeated free-form strings to a file, as fast as 'dd'?

dd can write repeating \0 bytes to a file very fast, but it can't write repeating arbitary strings. Is there a bash-shell method to write repeating arbitary strings equally as fast as 'dd' (including \...
5
votes
2answers
9k views

Unpack files and directories from an image created by dd?

I created an image of an NTFS partition using dd. I wonder if I can unload/unpack the files and directories in the image to any partition whose size is larger than the image size, so that I can ...
2
votes
3answers
404 views

How can I copy three partitions of my disk with a specific beginning and ending using dd?

Using the fdisk -l command I got the following answer: Device Boot Start End Blocks Id System /dev/sda1 * 2048 28266495 14132224 27 Hidden NTFS WinRE /dev/...
2
votes
1answer
91 views

Why dd reads from output device?

While dding a CentOS 7 x86_64 DVD image to a USB drive (sdb), I monitored the progress with iostat. It kept alternating between mostly writing to the USB drive: Device: tps kB_read/s ...
1
vote
2answers
8k views

How do I resize a disk image device?

I'm trying to reduce the size of a backup drive image. Original disk had these partitions: Model: ST916082 1A (scsi) Disk /dev/sde: 160GB Sector size (logical/physical): 512B/512B Partition Table: ...
1
vote
2answers
7k views

How to write a file at various offsets to a filesystem partition with dd command

Requirement I want to write a file at various offsets into the partition Partition /dev/part2 is mounted at /mypart I tried the command below: dd if=/dev/urandom of=/mypart/aaa bs=1024 seek=0 ...
6
votes
3answers
2k views

Linux/GParted can see partition table but dd bs=512 count=1 can't

I have an MBR formatted sd card and when connect to a Linux machine (xubuntu 12.04) it can mount a partition and parse the file system (as can GParted). However, when I try to read the MBR from the ...
5
votes
2answers
996 views

Is it better to use cat, dd, pv or another procedure to copy a CD/DVD?

Background I'm copying some data CDs/DVDs to ISO files to use them later without the need of them in the drive. I'm looking on the Net for procedures and I found a lot: Use of cat to copy a medium:...
4
votes
3answers
1k views

from a sparse file to a block device over the network

If I have a sparse file representing a block device, then how can I efficiently restore the file over a network? Ideally, I'd use rsync -S host:file /dev/sdb, but rsync won't write to block devices. ...
3
votes
2answers
3k views

How to use DD to clone a partition off a disk image?

I have a disk image, it's a "whole" disk image, e.g., contains multiple partitions, and I want to clone just one of them (not the first one..) onto a partition on an external drive with multiple ...
3
votes
3answers
512 views

How to inspect the contents of /dev/sdt?

I have what I hope is not a huge problem, but could end up being a very large problem. For some context, I mistakenly ran this command on my 27'' iMac: sudo dd if=ubuntu-rescue.img of=/dev/sdt bs=...
2
votes
1answer
63 views

Is there a way for a dd of a 80 GB hard drive to be minimal in size?

For backup reasons I need to use dd to copy a whole hard drive (including MBR, partitions,...). The hard drive is 80 GB in size where only 1.8 GB are used. The command I'm using is the following: dd ...
2
votes
3answers
2k views

How to clone linux installation onto an external harddisk?

I see a lot of dd related questions, but could not find answers to the following. If question/answers already exist, please point to them, then flag to close, delete or merge. Can a working linux ...
1
vote
3answers
102 views

The dd command isn't appear to accomplish anything [closed]

I'm currently trying to burn the Ubuntu operating system onto a USB flash drive of mine, so I can boot onto Ubuntu with my Chromebook. In order to do this, I executed the following dd command: sudo ...
1
vote
2answers
72 views

What's the POSIX way to read an exact number of bytes from a file?

Just hit this problem, and learned a lot from the chosen answer: Create random data with dd and get "partial read warning". Is the data after the warning now really random? Unfortunately ...
1
vote
0answers
59 views

how to copy a piece of a text file byte-by-byte to another text file? dd, head, or?

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, ...
0
votes
1answer
36 views

How to recover a node/content from a tarfile that's been overwritten

Disclaimer: I'm not very good at writing questions, and mine is a very specific scenario. So I'm just going to jump straight into the situation: I was browsing my filesystem, and somehow managed to ...
0
votes
1answer
79 views

Parallelize openssl as input to dd

I am using @tremby's great idea to fill a disk with random data. This involves piping openssl, which is encrypting a lot of zeros, to dd (bs=4M). I'm maxing out the single core on which this is ...
0
votes
1answer
50 views

How do I create a dd IMAGE from LVM for recovery purposes [duplicate]

How do I backup a LVM partition to an image for recovery purposes? I am trying to run dd on /dev/sda2, but it crashes after ca 8 hours, at around 380G sudo lvmdiskscan /dev/centos/swap [ 3....
0
votes
1answer
19 views

Restore unencrypted LVM2 logical volumes to LVM on LUKS setup?

I have an unencrypted Ubuntu 16.04 setup using LVM2. I'd like to 'port' this into an encrypted setup on the same computer and disk, which would save me the hassle of reinstalling all the apps and ...