53
votes
Why does piping `tar` into `dd` not stop until the disk is full?
It's because you're doing it wrong.
You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read.
And then you have conv=sync ...
- 49.9k
48
votes
Accepted
Why dd takes too long?
dd has many (weird) options, see dd(1).
You should explicitly state the buffer size, so try
dd if=/dev/sda of=/dev/sdb bs=16M
IIRC, the default buffer size is only 512 bytes. The command above sets ...
- 10.6k
48
votes
Accepted
Why does my dd Full Disk Copy Keep failing at 8 GB?
The error is, dd: error reading '/dev/sda': Input/output error, which tells you that the problem is reading the source disk and not writing to the destination. You can replace the destination disk as ...
- 120k
47
votes
Benchmark ssd on linux: How to measure the same things as crystaldiskmark does in windows
I'd say fio would have no trouble producing those workloads. Note that despite its name CrystalDiskMark is actually a benchmark of a filesysystem on a particular disk - it can't do I/O raw to the disk ...
- 3,814
45
votes
Accepted
Why is sync so important when making a bootable linux usb stick?
The dd does not bypass the kernel disk caches when it writes to a device, so some part of data may be not written yet to the USB stick upon dd completion. If you unplug your USB stick at that moment, ...
- 8,651
40
votes
Accepted
Splitting a small file into 512 byte segments changes it, but splitting it in 1k segments doesn't
The order find processes the files is not deterministic. It may be just the same order as the underlying system call gives, which probably depends on the underlying filesystem structure and can be ...
- 142k
39
votes
Why can't I copy my DVD with dd?
People mention that opening the DVD with VLC (which displays the DVD menu) magically makes the data accessible to dd, but nobody has yet explained why that is and how VLC accomplishes this feat.
I ...
- 495
34
votes
Running dd. Why resource is busy?
Unmounting the sub partitions helped
/dev/disk2
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *8.0 GB ...
- 341
33
votes
Accepted
dd command indicates not enough disk space - trying to format sd card for raspberry pi
-rw-r--r-- 1 root root 10485760 Mar 3 22:04 /dev/sdb
/dev/sdb is a regular file, not a device. You must have run rm /dev/sdb at some point. It is created automatically when the device is inserted, ...
32
votes
Accepted
How do I find the first non-zero byte on a block device, with an optional offset?
You can do this using cmp, comparing to /dev/zero:
cmp /path/to/block-device /dev/zero
cmp will give you the offset of the first non-zero byte.
If you want to skip bytes, you can use GNU cmp’s -i ...
- 450k
31
votes
Got "No space left on device" when cloning 1TB disk to 1.2TB disk using dd
Compare this answer (note bs=1K is used there):
dd is an old, cranky program initially intended to operate on tape devices. When you tell it to read one block of 1kB, it attempts to read one block. ...
- 22.9k
31
votes
Accepted
Is "dd" a reliable tool to make an SSD image?
TLDR: Use ddrescue
It supports resume/continue capabilities, has automatic logs, and tons of other options. More at the ddrescue home page.
Example syntax:
ddrescue /dev/sde yourimagename.image sde....
- 2,352
30
votes
Accepted
Is dd if=/dev/urandom of=/dev/mem safe?
Don't try this at home! It can crash your system, and if you're really unlucky it could damage a peripheral or make your computer unbootable.
Actually, on most platforms, it just fails with an error, ...
30
votes
Accepted
How can I verify that my hard disk has been zeroed / wiped?
Has the disk been zeroed?
Yes. The output of your dd command shows that it has written 5000981077504 bytes. Your cmp command says that it's reached EOF (end of file) after 5000981077504 bytes, which ...
- 2,725
28
votes
Force dd not to cache or not to read from cache
Based on @sendmoreinfo's answer:
dd if=/dev/device iflag=direct bs=1M
It does not affect read-cache.
- 36k
27
votes
Why does `dd` use `=` in its parameters?
POSIX mentions the following in its rationale section for dd:
Certainly, many of the operands could have been designed to use the Utility Syntax Guidelines, which would have resulted in the classic ...
- 450k
26
votes
Accepted
compressing and decompressing dd image - zstd instead of gzip
Using dd like that (without any options) will make your life miserable. Just cut it out entirely. Or at the very least increase its block size and tell it not to object to short reads.
Without dd, ...
- 120k
25
votes
Benchmark ssd on linux: How to measure the same things as crystaldiskmark does in windows
Original Answer: https://unix.stackexchange.com/revisions/480191/12
I created a script that tries to replicate the behavior of crystaldiskmark 6 with fio, and I also added support for older tests (...
- 2,081
25
votes
Accepted
dd command oflag direct and sync flags
We can probably rule out conv=sync to start with. It does something rather different, which I expect you do not want :-).
pad every input block with NULs to ibs-size; when used with
block or ...
- 51.4k
24
votes
what is the difference between seek and skip in dd command?
skip (also known as iseek in some dd implementations) moves current pointer of the input stream while seek moves current pointer in the output stream.
Thus, by using skip you could ignore some data ...
- 8,651
24
votes
Why dd takes too long?
Years back in Unix-land dd was the required way to copy a block device. That has carried forward as cargo-cult knowledge even though (on Linux-based systems, at least) cat is almost always faster than ...
- 120k
24
votes
Accepted
What is the maximum value for the bs argument of dd?
The POSIX specifications for dd don’t specify a maximum explicitly, but there are some limits:
the datatype used to store the value given can be expected to be size_t, since that’s the type of the ...
- 450k
23
votes
Got "No space left on device" when cloning 1TB disk to 1.2TB disk using dd
As others have already identified, the problem is that you're using dd. Rule #1 of using dd is you do not use dd. Rule #2 of using dd is you do not use dd. Rule #3 of using dd (for experts only) is ...
22
votes
Benchmark ssd on linux: How to measure the same things as crystaldiskmark does in windows
Try KDiskMark, very similar to CrystalDiskmark.
The application is written in C++ with Qt and doesn't have any KDE dependencies.
Usage
To start, after select a folder (and optionally other params), ...
- 1,586
22
votes
Does dd do any kind of verification?
dd or any other application does not have “some kind of built in verification” in the sense you're probably thinking of: it doesn't read back the data from the storage medium to compare with what was ...
22
votes
Accepted
Will dd if=/dev/zero of=/dev/sda wipe out a pre-existing partition table?
Will dd if=/dev/zero of=/dev/sda wipe out a pre-existing partition table?
Yes, the partition table is in the first part of the drive, so writing over it will destroy it. That dd will write over the ...
- 142k
21
votes
How can I copy arbitrary part of binary file with reasonable speed?
GNU dd supports count_bytes, seek_bytes, skip_bytes flags. This allows you to use a performant blocksize choice with arbitrary offsets and sizes.
GNU dd since version 9.1 does this by default if you ...
- 49.9k
20
votes
On-the-fly stream compression that doesn't spill over into hardware resources?
dd reads and writes data one block at a time, and it only ever has one block outstanding. So
valgrind dd if=/dev/zero status=progress of=/dev/null bs=1M
shows that dd uses approximately 1MB of ...
- 450k
20
votes
Accepted
Why does writing random data using dd result in disk partitions?
Several possibilities:
Linux supports a lot of different partition table types, some of which use very few magic bytes, and then it's easy to mis-identify random data (*) [so it's possible to ...
- 49.9k
Only top scored, non community-wiki answers of a minimum length are eligible