How to overwrite beginning of a device with random bits or zeros?

Question

Just looking to overwrite a small portion of a device with zeros or random bits. More precisely I would like to overwrite the first 1% of all the sectors or few MiB. Is there an easy way to do that ?

Answer 1

Although /dev/urandom is extremely slow and as such not suitable for overwriting large amounts of data (entire disk), it might do for small regions.

Example overwriting 8MiB:

dd bs=1M count=8 iflag=fullblock if=/dev/urandom of=/dev/destroyme

Alternatively you can use shred:

shred -v -n 1 -s 8M /dev/destroyme

You can also use losetup to create devices of specific size and offsets, and overwrite them with utilities that don't have their own size / offset options.

losetup --find --show --offset 0 --sizelimit $((8*1024*1024)) /dev/destroyme
# will print /dev/loopX
cat /dev/urandom > /dev/loopX
losetup -d /dev/loopX

Answer 2

Same as with any file to overwrite the first 10 MiB with zeros:

head -c10M < /dev/zero 1<> /dev/sdax

For block device files, the 1<> to open without truncation is not even needed as there's no such thing as truncating a block device, so you can simply do:

head -c10M < /dev/zero > /dev/sdax

Not all head implementations support -c, and when they do, not all support that M suffix (and when they do M may mean megabyte (1000000 bytes) like ksh93's head builtin or mebibyte (1048576 bytes) like GNU head). In that case, you can do:

head -c "$((10 * 1024 * 1024)"

to make it explicit.

If we compare with dd bs=1M count=10 < /dev/zero > /dev/sdax:

• conceptually

  • head is the command to read a specified number of bytes or lines from a file and write it to stdout
  • while dd is a low level command to read and write data in the exact way you want it to.

  I'd use head here on the ground that it's the command designed for the task. I'd use dd if I wanted to optimise for a specific use case, or use one of the specific features of dd (see also the note on portability).

• The read+write loop:

  • head -c10M will try and read the requested data no matter what and only fail with a non-zero exit status if an error is encountered.
  • dd bs=1M count=10, will do exactly 10 reads (as long as there's no error) and for each read that returns some data, do a corresponding write with the amount of data that was read. That only works as long as each read returns exactly the 1M requested. In practice, that's true for /dev/zero, but on Linux (4.6 at least), for /dev/urandom, I can't get more than 32MiB minus 1 byte in a single read (so still OK for 1MiB, though YMMV if you're using a different version of Linux), and for /dev/random, only a few bytes (what's currently in the entropy pool). The GNU implementation of dd has a iflag=fullblock to keep reading until the request buffer is full to behave like head, but if you don't have GNU dd, the only option is to do 1 byte reads at a time that would have dramatic impacts on the performance.

• performance: For anything other than small amounts (less than a few hundred megabytes) where the data is written to buffers that will be flushed to disk later, or writing to /dev/null, the process will be I/O bound. If reading to /dev/urandom or /dev/random the bottle neck will be either the random number generation or the disk I/O. In those cases, you won't find much difference between dd and head. In any case head is likely to have a higher CPU overhead (unnoticed when the performance is I/O bound).

  • head is a basic tool. Implementations will try to do the job as efficiently as possible while still being reliable for all types of input and output and not use too much resource. It will do the same read+write loop as dd, the main difference in terms of performance is the size of reads and writes which will determine the number of system calls being made.

    Those sizes will depend on the implementation and version of head and possibly the system. With the latest version of GNU head on my system, the reads are of size BUFSIZE (8KiB on GNU systems), and the writes of size 4KiB, though that can be changed with stdbuf -o 1M for instance.

    ksh93's head builtin seems to do 64KiB reads and writes and doesn't use the libc's stdio, at least on my system.

    GNU head using stdio also means extra overhead incurred by stdio (the implementation of which is system dependant).

    Current versions of GNU cat use fadvise to tell the system that it will read the data sequentially so it can optimise caching accordingly. It's not impossible some implementations of head do that as well, or will do in the future. dd being low-level, I would expect it would only do it if you told it to (I'm not aware of any dd implementation that has such support).

  • dd is very low level, it will call the read() and write() system calls directly. You can't get much more efficient than that other than by using specialised APIs like Linux' sendfile().

    It gives you much more control on the size of the read() and write(), so lets you optimise yourself based on the type of the input/output and the available resources. For instance, if you have a lot of memory available, you might as well read the whole data in one go (though in my tests while copying from /dev/zero to /dev/null, I can't see any significant improvement past a block size of 32KiB and performance even start to degrade after a blocksize of 1MiB).

• portability

  None of -c, bs=10M, conv=fullblock are portable. The only POSIX command to read a certain amount of data from a file is dd, but to use it reliably (except on /dev/zero), as discussed above, you need bs=1 which means dreadful performance.

• consequence of write errors.

  Both head and dd will exit at once when trying to write past the end of the block device. If the disk has failing sectors, that will generally not be detected because the actual writing to disk is asynchronous. With the GNU implementation of dd, you can force the write to be direct with oflag=direct which means dd will stop on the first error. You may want to use the default block size of 512 then if you want to write as much as possible before the first failing sector.

• consequence of read errors

  You shouldn't get any read error on /dev/zero, /dev/urandom or /dev/random. But more generally, both head and dd will exit with an error upon the first read error. With dd, you can carry on upon errors with conv=noerror. In that case, you'll probably want to add the sync option (conv=noerror,sync) so that the failed blocks are padded with zeros. head will not give you any option to do that as it's not been designed for that.

Alternatives.

• pv -Ss 10M < /dev/zero > /dev/sdax will copy those 10M and give you a progress bar. By default the read/write size are 128KiB in my test. You can change it with the -B option, but in my tests, 128KiB gives the best results already. pv has a -E option equivalent to dd's conv=noerror,sync.

  On Linux, it's also good for I/O on pipes as it uses the splice() system call to optimise performances.

• If you want to play with the sendfile() system call, you can use xfs_io.

  xfs_io -c 'sendfile -i src 0 10M' dst
  

  sends 10M from src to dst. However, it only does one sendfile() and the system call can't be used on /dev/zero, /dev/random nor /dev/urandom. It can be used on sparse files though.

   truncate -s 1T empty-file
   xfs_io -c 'sendfile -i empty-file 0 10M' /dev/sdax
  

  would work but for large amounts (several Gibibytes), because that's one sendfile() system call, that much memory must be allocated which means it's going to be less efficient than dd bs=1M. Ideally, we'd want to do multiple sendfile()s of only a few mebibytes at a time, but I'm not aware of any command that does that.

• For /dev/zero input, you don't really need to read the data for each block you write. After all, it's only zeros. It's easy enough to create a buffer with only zeros without having to read /dev/zero, and we can reuse it between every write. For example:

  PERLIO=:unix perl -e '$x = pack("x" . 1024*1024); print $x for 1..10000' > /dev/sdax
  

to write 10000 MiB would be a lot more efficient (despite the perl overhead) than any solution that repeatedly reads /dev/zero.

Answer 3

You can use dd to do that using the /dev/urandom device which will provide you with random data. An example :

dd if=/dev/urandom of=/dev/sdX bs=1M count=100

this will write 100mbyte of random data :

• if is the input file
• of is the output file
• bs=1M is the block size 1Mbyte
• count is how many times should be this blocks written

You can as well use as input file /dev/zero, /dev/null or anything else which provides you data. This command will start writing the data at the beginning of the output file/device.