Just looking to overwrite a small portion of a device with zeros or random bits. More precisely I would like to overwrite the first 1% of all the sectors or few MiB. Is there an easy way to do that ?
/dev/urandom is extremely slow and as such not suitable for overwriting large amounts of data (entire disk), it might do for small regions.
Example overwriting 8MiB:
dd bs=1M count=8 iflag=fullblock if=/dev/urandom of=/dev/destroyme
Alternatively you can use
shred -v -n 1 -s 8M /dev/destroyme
You can also use
losetup to create devices of specific size and offsets, and overwrite them with utilities that don't have their own size / offset options.
losetup --find --show --offset 0 --sizelimit $((8*1024*1024)) /dev/destroyme # will print /dev/loopX cat /dev/urandom > /dev/loopX losetup -d /dev/loopX
Same as with any file to overwrite the first 10 MiB with zeros:
head -c10M < /dev/zero 1<> /dev/sdax
For block device files, the
1<> to open without truncation is not even needed as there's no such thing as truncating a block device, so you can simply do:
head -c10M < /dev/zero > /dev/sdax
Not all head implementations support
-c, and when they do, not all support that
M suffix (and when they do
M may mean megabyte (1000000 bytes) like
head builtin or mebibyte (1048576 bytes) like GNU
head). In that case, you can do:
head -c "$((10 * 1024 * 1024)"
to make it explicit.
If we compare with
dd bs=1M count=10 < /dev/zero > /dev/sdax:
headis the command to read a specified number of bytes or lines from a file and write it to stdout
ddis a low level command to read and write data in the exact way you want it to.
headhere on the ground that it's the command designed for the task. I'd use
ddif I wanted to optimise for a specific use case, or use one of the specific features of
dd(see also the note on portability).
The read+write loop:
head -c10Mwill try and read the requested data no matter what and only fail with a non-zero exit status if an error is encountered.
dd bs=1M count=10, will do exactly 10 reads (as long as there's no error) and for each read that returns some data, do a corresponding write with the amount of data that was read. That only works as long as each read returns exactly the 1M requested. In practice, that's true for
/dev/zero, but on Linux (4.6 at least), for
/dev/urandom, I can't get more than 32MiB minus 1 byte in a single read (so still OK for 1MiB, though YMMV if you're using a different version of Linux), and for
/dev/random, only a few bytes (what's currently in the entropy pool). The GNU implementation of
iflag=fullblockto keep reading until the request buffer is full to behave like
head, but if you don't have GNU
dd, the only option is to do 1 byte reads at a time that would have dramatic impacts on the performance.
performance: For anything other than small amounts (less than a few hundred megabytes) where the data is written to buffers that will be flushed to disk later, or writing to
/dev/null, the process will be I/O bound. If reading to
/dev/randomthe bottle neck will be either the random number generation or the disk I/O. In those cases, you won't find much difference between dd and head. In any case
headis likely to have a higher CPU overhead (unnoticed when the performance is I/O bound).
headis a basic tool. Implementations will try to do the job as efficiently as possible while still being reliable for all types of input and output and not use too much resource. It will do the same read+write loop as
dd, the main difference in terms of performance is the size of reads and writes which will determine the number of system calls being made.
Those sizes will depend on the implementation and version of
headand possibly the system. With the latest version of GNU
headon my system, the reads are of size BUFSIZE (8KiB on GNU systems), and the writes of size 4KiB, though that can be changed with
stdbuf -o 1Mfor instance.
headbuiltin seems to do 64KiB reads and writes and doesn't use the libc's stdio, at least on my system.
headusing stdio also means extra overhead incurred by stdio (the implementation of which is system dependant).
Current versions of GNU
fadviseto tell the system that it will read the data sequentially so it can optimise caching accordingly. It's not impossible some implementations of
headdo that as well, or will do in the future.
ddbeing low-level, I would expect it would only do it if you told it to (I'm not aware of any
ddimplementation that has such support).
ddis very low level, it will call the
write()system calls directly. You can't get much more efficient than that other than by using specialised APIs like Linux'
It gives you much more control on the size of the
write(), so lets you optimise yourself based on the type of the input/output and the available resources. For instance, if you have a lot of memory available, you might as well read the whole data in one go (though in my tests while copying from /dev/zero to /dev/null, I can't see any significant improvement past a block size of 32KiB and performance even start to degrade after a blocksize of 1MiB).
conv=fullblockare portable. The only POSIX command to read a certain amount of data from a file is
dd, but to use it reliably (except on
/dev/zero), as discussed above, you need
bs=1which means dreadful performance.
consequence of write errors.
ddwill exit at once when trying to write past the end of the block device. If the disk has failing sectors, that will generally not be detected because the actual writing to disk is asynchronous. With the GNU implementation of
dd, you can force the write to be direct with
ddwill stop on the first error. You may want to use the default block size of 512 then if you want to write as much as possible before the first failing sector.
consequence of read errors
You shouldn't get any read error on /dev/zero, /dev/urandom or /dev/random. But more generally, both
ddwill exit with an error upon the first read error. With
dd, you can carry on upon errors with
conv=noerror. In that case, you'll probably want to add the
conv=noerror,sync) so that the failed blocks are padded with zeros.
headwill not give you any option to do that as it's not been designed for that.
pv -Ss 10M < /dev/zero > /dev/sdaxwill copy those 10M and give you a progress bar. By default the read/write size are 128KiB in my test. You can change it with the
-Boption, but in my tests, 128KiB gives the best results already.
-Eoption equivalent to
On Linux, it's also good for I/O on pipes as it uses the
splice()system call to optimise performances.
If you want to play with the
sendfile()system call, you can use
xfs_io -c 'sendfile -i src 0 10M' dst
sends 10M from src to dst. However, it only does one
sendfile()and the system call can't be used on
/dev/urandom. It can be used on sparse files though.
truncate -s 1T empty-file xfs_io -c 'sendfile -i empty-file 0 10M' /dev/sdax
would work but for large amounts (several Gibibytes), because that's one
sendfile()system call, that much memory must be allocated which means it's going to be less efficient than
dd bs=1M. Ideally, we'd want to do multiple
sendfile()s of only a few mebibytes at a time, but I'm not aware of any command that does that.
/dev/zeroinput, you don't really need to read the data for each block you write. After all, it's only zeros. It's easy enough to create a buffer with only zeros without having to read
/dev/zero, and we can reuse it between every write. For example:
PERLIO=:unix perl -e '$x = pack("x" . 1024*1024); print $x for 1..10000' > /dev/sdax
to write 10000 MiB would be a lot more efficient (despite the
perl overhead) than any solution that repeatedly reads
You can use
dd to do that using the
/dev/urandom device which will provide you with random data. An example :
dd if=/dev/urandom of=/dev/sdX bs=1M count=100
this will write 100mbyte of random data :
ifis the input file
ofis the output file
bs=1Mis the block size 1Mbyte
countis how many times should be this blocks written
You can as well use as input file
/dev/null or anything else which provides you data. This command will start writing the data at the beginning of the output file/device.