Questions tagged [dd]

dd is a traditional utility for binary data copies

Filter by
Sorted by
Tagged with
68
votes
5answers
44k views

Is there a way to determine the optimal value for the bs parameter to dd?

On occasion I've seen comments online along the lines of "make sure you set 'bs=' because the default value will take too long," and my own extremely-unscientific experiences of, "well that seemed to ...
58
votes
2answers
13k views

When is dd suitable for copying data? (or, when are read() and write() partial)

Short version: In what circumstances is dd safe to use for copying data, safe meaning that there is no risk of corruption due to a partial read or write? Long version — preamble: dd is often used to ...
120
votes
8answers
31k views

dd vs cat — is dd still relevant these days?

I recently realized we can use cat as much as dd, and it's actually faster than dd I know that dd was useful in dealing with tapes where block size actually mattered in correctness, not just ...
24
votes
4answers
14k views

Why does dd from /dev/random give different file sizes?

I am running the following command on an ubuntu system: dd if=/dev/random of=rand bs=1K count=2 However, every time I run it, I end up with a file of a different size. Why is this? How can I ...
34
votes
5answers
120k views

Full DD copy from hdd to hdd

ORIGINAL QUESTION: If I have 2 identical hard drives with the following characteristics: SATA 6.0 Gb/s 5400 rpm 3TB How long should a full dd copy take to complete? So far it's been running for 5 ...
147
votes
13answers
150k views

How do I know if dd is still working?

I've not used dd all that much, but so far it's not failed me yet. Right now, I've had a dd going for over 12 hours - I'm writing an image back to the disk it came from - and I'm getting a little ...
5
votes
1answer
150 views

System lags when doing large R/W operations on external disks

I am having some issues with system-wide latency/lagging when doing large disk imaging operations on an Ubuntu 18.04 system. Here's the system specs: Processor: Intel Core i7 (never near capacity on ...
110
votes
20answers
25k views

What is the fastest way to send massive amounts of data between two computers? [closed]

This is a situation I am frequently in: I have a source server with a 320GB hard-drive inside of it, and 16GB of ram (exact specs available here, but as this is an issue I run into frequently on ...
12
votes
3answers
4k views

saving data from a failing drive

An external 3½" HDD seems to be in danger of failing — it's making ticking sounds when idle. I've acquired a replacement drive, and want to know the best strategy to get the data off of the dubious ...
66
votes
6answers
61k views

Is it possible to mount a gzip compressed dd image on the fly?

I like create an image backup for the first time I'm backing up a system. After this first time I use rsync to do incremental backups. My usual image backup is as follows: Mount and zero out the ...
42
votes
4answers
17k views

What does dd stand for?

I know very well what the command does, but man dd, info dd tell me: 'Convert and copy a file', as does GNU Coreutils. Google says its an abbreviation of everything between medicine and bad webchat ...
17
votes
3answers
15k views

Create random data with dd and get “partial read warning”. Is the data after the warning now really random?

I create a 1TB file with random data with dd if=/dev/urandom of=file bs=1M count=1000000. Now I check with kill -SIGUSR1 <PID> the progress and get the following: 691581+0 Datensätze ein 691580+...
19
votes
3answers
20k views

How do I mount an `img` created with /bin/dd of a hard drive?

I used dd to backup a 80GB drive dd if=/dev/sdb of=~/sdb.img Now I need to access some files on that drive, but I don't want to copy the ".img" back over the drive. mount ~/sdb.img /mnt/sdb doesn'...
5
votes
2answers
3k views

How to write file into another

I have an empty file (only zeroes are in it) of size 9,0KB and I need to write another file (with size 1,1KB) to it, but the first file must not lose its size or the rest of its contents. So if the ...
4
votes
3answers
838 views

Why does a gunzip to dd pipeline slow down at the end?

my command: gunzip -c serial2udp.image.gz | sudo dd of=/dev/mmcblk0 conv=fsync,notrunc status=progress bs=4M my output: 15930949632 bytes (16 GB, 15 GiB) copied, 1049 s, 15.2 MB/s 0+331128 ...
82
votes
3answers
129k views

How to dd a remote disk using SSH on local machine and save to a local disk

How can I create a backup of a remote disk using SSH on my local machine and save it to a local disk? I've tried the following: ssh hostname@my.ip.address "sudo dd if=/dev/sdX " | \ dd of=/home/...
35
votes
8answers
23k views

Clone whole partition or hard drive to a sparse file

I like to clone a whole partition or a whole hard drive onto a larger external disk but like to create a sparse file. I often use dd for cloning, but it doesn't support sparse files. As a workaround I ...
8
votes
2answers
8k views

How to parallelize dd?

I'm currently having trouble with dd invoked with a sparse file as input (if) and a file as output (of) with conv=sparse. dd seems to be using one core of the CPU (Intel(R) Core(TM) i7-3632QM CPU @ 2....
10
votes
3answers
15k views

How to check integrity of a dd backup?

I just made a backup of an entire hard drive (50GB) over ssh via: dd if=/dev/hda | buffer -s 64k -S 10m | ssh myuser@myhost "cat > ~/image.img" What's now the best way to check the integrity of ...
5
votes
1answer
2k views

Partially overwrote /dev/sda, can I recover anything?

I was messing around with my new hard drive, which only contains some personal files, and to check its write speed, I typed: sudo dd if=/dev/zero of=/dev/sda, /dev/sda/ being the new 2 TB connected to ...
5
votes
1answer
8k views

Why is dd so slow with a bs of 100M [closed]

I just tried to overwrite a fast ssd using dd. Using the ubuntu boot image I typed in: dd if=/dev/zero of=/dev/sda bs=100M error writing '/dev/sda': No space left on device blah blah 256 GB copied, ...
2
votes
3answers
1k views

dd crashing - what to do about it

I have dd crashing after running a few hours. Typically with a message like Input/output error 68424+1 records in 68424+1 records out 342124883968 bytes (342 GB) copied, 5114.68 s, 66.9 MB/s Is ...
61
votes
7answers
64k views

Best way to remove bytes from the start of a file?

Today I had to remove the first 1131 bytes from an 800MB mixed text / binary file, a filtered subversion dump I'm hacking for a new repository. What's the best way to do this? To start with I tried ...
23
votes
4answers
29k views

Benchmark ssd on linux: How to measure the same things as crystaldiskmark does in windows

I want to benchmark a ssd (possibly with encrypted filesystems) and compare it to benchmarks done by crystaldiskmark on windows. So how can I measure approximately the same things as crystaldiskmark ...
20
votes
2answers
10k views

Is it better to use cat, dd, pv or another procedure to copy a CD/DVD?

Background I'm copying some data CDs/DVDs to ISO files to use them later without the need of them in the drive. I'm looking on the Net for procedures and I found a lot: Use of cat to copy a medium: ...
12
votes
4answers
13k views

Why dd takes too long?

I need to copy one disk to another. I tried with the command below and it takes nearly a day to copy 1 TB of disk. dd if=/dev/sda of=/dev/sdb I have tried the same on a Unix system with the ...
19
votes
4answers
10k views

Read the middle of a large file

I have a 1 TB file. I would like to read from byte 12345678901 to byte 19876543212 and put that on standard output on a machine with 100 MB RAM. I can easily write a perl script that does this. ...
13
votes
5answers
1k views

Is it safe to take the drive image of the current working drive?

I have to backup my hard disk. I want to use dd and put the image on an external hdd. Can I do this using dd from the OS that resides on the hdd itself or do I have to boot from another device, for ...
13
votes
5answers
34k views

can 'dd' be used to clone to a smaller HDD, knowing that partitions will need editing?

I've used dd to clone disks like this: dd if=/dev/sdb of=/dev/sda bs=4096 conv=notrunc,noerror,sync And it's always worked fine. Any and all docs on 'dd' take pains to remind you that the target ...
11
votes
2answers
14k views

Why is sync so important when making a bootable linux usb stick?

From the Arch Linux Wiki: https://wiki.archlinux.org/index.php/USB_flash_installation_media # dd bs=4M if=/path/to/archlinux.iso of=/dev/sdx status=progress && sync [...] Do not miss ...
3
votes
4answers
19k views

How to Create Bootable Windows 10 image in Debian?

Pseudocode but originally developed for Windows 7 iso file but applied for Windows 8 in the thread How to create bootable Windows 8 iso image in Linux? but it does not work with Windows 10 iso # ...
6
votes
4answers
9k views

Sending the output from 'dd' to awk/sed/grep

I'm trying to approximate the computer's write speed using dd: dd if=/dev/urandom of=/dev/null bs=1K count=10000 which gives the following output 10000+0 records in 10000+0 records out 10240000 ...
5
votes
4answers
2k views

How to create a file system as a non root user?

I'd like to create a filesystem image as a non-root user, the problem I'm facing is that the way I'm currently doing it, it needs a mount and as a non-root can't do a mount. This is what I'm now ...
3
votes
2answers
9k views

Can I mount a partition from a full drive image? [duplicate]

Note: While I agree that this question basically is a duplicate of the above question, I feel @alienth's answer (below) is more concise, so I suggest you take a look at it before going to the other ...
6
votes
4answers
8k views

How to use DD to clone a partition off a disk image?

I have a disk image, it's a "whole" disk image, e.g., contains multiple partitions, and I want to clone just one of them (not the first one..) onto a partition on an external drive with multiple ...
5
votes
1answer
195 views

How to fill a device with zeros, without overwriting the bytes that are already zeros?

I have a USB flash drive usb 3, the reading speed is much more than the writing speed. Let's say that 99% of the flash memory is already full with zeros, and I would like to fill it with zeros until ...
5
votes
6answers
9k views

Check command success in bash

I'm currently writing a small script to backup bulks of floppy disks and format them afterward for later use. I use dd to copy the disk's image and cp to copy all the files on the disk. Here are the ...
4
votes
3answers
20k views

Terminal: create bootable USB from iso

How can I create a bootable USB stick from an iso image? I thought dd should do the work, but so far I were unsuccesful. This is what I've tried: umount /dev/sdx deleted every partition on sdx with ...
2
votes
2answers
12k views

How do I resize a disk image device?

I'm trying to reduce the size of a backup drive image. Original disk had these partitions: Model: ST916082 1A (scsi) Disk /dev/sde: 160GB Sector size (logical/physical): 512B/512B Partition Table: ...
5
votes
2answers
878 views

How to prevent dd's progress from being meaningless on Linux?

When running dd command to copy an ISO on Linux, I get a single progress print that stays open for a long time (many minutes). Then another at the end. The problem seems to be that a very large cache ...
4
votes
3answers
6k views

HDD image file checksum does not match with device checksum

I have a 660297728 byte HDD image with MD5 hash f5a9d398e974617108d26c1654fe7bcb: root@T42# ls -l image -rw-rw-r-- 1 noc noc 660297728 Sep 29 19:00 image root@T42# md5sum image ...
2
votes
3answers
2k views

What's the POSIX way to read an exact number of bytes from a file?

Just hit this problem, and learned a lot from the chosen answer: Create random data with dd and get "partial read warning". Is the data after the warning now really random? Unfortunately ...
19
votes
4answers
40k views

'seek' argument in command dd

Can some explain me what is happening in the following lines? dd if=/dev/urandom bs=4096 seek=7 count=2 of=file_with_holes especially seek part is not clear Man pages says : seek=BLOCKS ...
14
votes
6answers
14k views

Fast Way to Randomize HD?

I've read about how to make hard drives secure for encryption, and one of the steps is to write random bits to the drive, in order to make the encrypted data indistinguishable from the rest of the ...
10
votes
3answers
1k views

How can `dd` be used to right-shift data blocks?

Consider a 100MB raw block device as a simple example. That is 204800 blocks of 512 bytes each for a total of 102760448 bytes. The challenge is to shift the first 98MB (200704 blocks) so there is a ...
4
votes
1answer
10k views

What is the difference between `dd`, `cp` and `rsync`? [closed]

Besides the explanations from the man pages and --help information, in which way do the commands dd, cp and rsync differ when used to copy files? In which context is each of these superior to the ...
23
votes
5answers
4k views

On-the-fly stream compression that doesn't spill over into hardware resources?

I have 200 GB free disk space, 16 GB of RAM (of which ~1 GB is occupied by the desktop and kernel) and 6 GB of swap. I have a 240 GB external SSD, with 70 GB used1 and the rest free, which I need to ...
7
votes
1answer
3k views

Writing raw images safely to USB sticks

Trying lots of different Linux distributions on all kinds of hardware, I find myself typing commands like this quite often: sudo dd if=xubuntu-13.10-desktop-amd64.iso of=/dev/sdc bs=10240 Needless ...
7
votes
2answers
10k views

How to check progress when cloning a disk using `dd`? [duplicate]

Possible Duplicate: dd: How do I know if it's still working? I'm using dd to clone a disk to an image file, and compress it on the fly: dd if=/dev/sda | gzip > /mnt/image_file How can I ...
3
votes
1answer
5k views

How to shrink, clone an entire Linux disk and boot it?

I had an Ubuntu 14.04 running on a 1024 GB disk (disk A), which only used up to 130 GB spaces. I want to clone it to a 256 GB SSD disk ( disk B ). But failed. I used gparted to shrink the partitions ...