7

Filling a drive with /dev/urandom seems to be very slow, so I created a file filled with FF:

dd if=/dev/zero ibs=1k count=1000 | tr "\000" "\377" >ff.bin

I'd like to fill the drive with copies of this file but the following command only writes once:

dd if=ff.bin of=/dev/sdb count=10000

How do I fill the drive with copies of the file, or is there a faster way to fill the drive with 1's?

  • 5
    Why not use zeros? 1-bits don't erase a disk better than 0-bits. – Gilles 'SO- stop being evil' Aug 19 '14 at 23:08
  • 3
    @Gilles I guess zero is special enough that the disk driver could cheat and not really write anything to disk, only marking blocks as empty. I think some virtual hard disk do so. But still, it depends on which reason he's filling a drive for. If it is security, neither 0 nor 1 are safe enough, and also filling with random wouldn't be good if the hard disk is SSD – pqnet Aug 21 '14 at 21:27
  • 1
    @pqnet Zero isn't special for physical storage. With a virtual hard disk, it might be, but filling with anything is unsafe. For SSD, there are specific issues with reallocated blocks, but writing with nonzero values doesn't help with that. – Gilles 'SO- stop being evil' Aug 21 '14 at 21:35
  • @Gilles well, the reason for which he want to write to his disk is not explained so I think it would be great if the question is clarified – pqnet Aug 21 '14 at 21:38
  • The venerable dd has a the option seek=N skip N obs-sized blocks at start of output, so you can write a loop to seek to the correct place k*M on the output block device before repeating a write of the M-sized file. – David Tonhofer Apr 20 at 13:50
10

Simply do:

tr '\0' '\377' < /dev/zero > /dev/sdb

It will abort with an error when the drive is full.

Using dd does not make sense here. You use dd to make sure reads and writes are made of a specific size. There's no reason to do it here. tr will do reads/writes of 4 or 8 kiB which should be good enough.

  • 1
    you might even get a boost with tr ... | tee - - - > /dev/disk but the likelihood that tr alone doesn't already meet and exceed the write bottleneck is pretty slim. – mikeserv Aug 19 '14 at 23:47
  • 1
    @mikeserv, I think the only way to increase performance from that is to reduce the number of system calls made. tee won't help for that. stdbuf -o1M tr... may help. – Stéphane Chazelas Aug 20 '14 at 5:41
  • Agreed. I sometimes use /dev/zero as a bucket with tr - like for newlines or whatever - and it always consumes a whole cpu. I think the pull from the kernel pumping out all of those bytes is pretty heavy. I was thinking that perhaps the file could just be duped. Maybe it was dumb. – mikeserv Aug 20 '14 at 5:52
  • 4
    @mikeserv, actually, using tee - - - does increase throughput significantly in my testing. That's going from 800MiB/s for tr (950MiB/s with stdbuf) to 2.3GiB/s with tee, like you say in any case way above the rate any current drive can sustain. – Stéphane Chazelas Aug 20 '14 at 6:01
  • Maybe true, but that 2.3g flood will still wind up trickling as soon your pipeline steps outside of job control. stdbuf is smart - I never use it and probably should. And this is the second time in as many days someone's brought that to my attention. I think the other thing is CentOS - I read somewhere about a 64kb kernel buffer. – mikeserv Aug 20 '14 at 6:09
4

For a faster /dev/urandom alternative, there is shred -v -n 1 (if pseudorandom is OK), or using cryptsetup with random key and zeroing that (for encrypted zeroes). Even without AES acceleration, it easily beats /dev/urandom speeds.

Not sure how fast tr is, otherwise you could just dd if= | tr | dd of=.

Using a file as a pattern source could be done like this:

(while [ 1 ]; do cat file; done) | dd of=...

Although the file should be reasonably large for that to be remotely efficient.

If the count= is important to you, add iflag=fullblock to the dd command. Partial reads are possible which would result in partial blocks to be counted as full blocks. This is especially when using larger blocksizes (like bs=1M), which you should if you want speed.

  • shred is fast, and it reports progress! Thanks! – linuxfix Aug 21 '14 at 16:09
2

This is a very fast and efficient way of writing 64 kB strings of 0xFF over and over, using awk, consuming < 8% of CPU and limited only by the speed of the drive you are writing to.

I have tried using tr as was suggested here and found it was painfully slow and consumed a lot of CPU translating each and every Byte. My method works in 64 kB blocks and is at least 3.5x faster than the single character-based tr (26 MB/s versus 7 MB/s on old PATA hw - finishing in 52 minutes in silence versus 3+ hours with loud cooling fan revs...). I like when people brush off basic computer science knowledge without even testing their opinion first.

I recommend building the following script using printf from the shell rather than attempting to write it in vi since most boot CD-ROMs will not give you sufficient tmp space for the buffer log to copy-and-paste 65,536 times...

1) Compose the script

$ printf "echo | awk '{\n\twhile (1) {\n\t\tprintf(\"%%s\", \"" >/tmp/writeones.sh
$ for i in `seq 1 65536`; do printf '\377' >>/tmp/writeones.sh; done
$ printf "\");\n}\n}'\n" >>/tmp/writeones.sh

the output script seems like: (Omit some characters 0xff in string)

echo | awk '{                                                                                                                           
    while (1) {
        printf("%s", "ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ...");
    }
}'

2) Test the script:

$ sh /tmp/writeones.sh | od -Ax -tx1
000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
*
Ctrl+C

3) Run the script and pipe to your drive with dd (double-check you have the right drive!!!):

# sh /tmp/writeones.sh | dd of=/dev/ada0 bs=65536 &

Why did I choose 64 kB? 1. Arg list limitation of AWK, and 2. for my hardware, this achieves the best speed for me. I also tried writing this in just shell using printf and found it was 40% slower and consumed 80% of the CPU because of all the forks.

0

To test the SSD speed, I wrote a little Perl program to write "zeros", "ones" or alphanumerics to a block device given on the command line. It doesn't invoke the dd golem, just does direct writes and syncs. May be of help here:

https://github.com/dtonhofer/wild_block_device_filler

Just run it as

wildly_fill_block_device.pl --dev=sdX1 --fillpat 1 --chunksize=1024P --sync

And it will write 0xFFs to /dev/sdX1 (unbuffered, using Perl's syswrite()) in chunks of (in this case) "1024 physical blocks" while synch-ing the data to disk after every write using fdatasync(). On my SSD, this runs at ~70 MiB/s.

It will ask you whether you are SURE before it proceeds to nuke the partition or disk.

0

You can do it pretty efficiently in Bash without external files, except the device to write to, or external commands, except dd.

  1. Use printf to generate a byte with all one bits, and put it in a Bash variable.
  2. Concatenate the Bash variable to itself multiple times.
  3. Use Bash process substitution to fake an infinitely long input file that repeatedly echoes the Bash variable with no trailing newline.
  4. Use that as the input file to dd.

If you have GNU dd, get a nice progress bar with:

sudo -i
bash
ones="$( printf '\xff' )"; for _ in {1..16}; do ones="$ones$ones"; done; dd status=progress bs=65536 if=<( while true; do echo -n "$ones"; done ) of=/dev/whatever

Remove the status=progress if it doesn't work for you. The 65536 is 216, because the initial byte with all one bits was duplicated 16 times.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.