18

I've read about how to make hard drives secure for encryption, and one of the steps is to write random bits to the drive, in order to make the encrypted data indistinguishable from the rest of the data on the hard drive.

However, when I tried using dd if=/dev/urandom of=/dev/sda in the past, the ETA was looking to be on the order of days. I saw something about using badblocks in lieu of urandom, but that didn't seem to help a whole lot. I would just like to know if there are any ways that might help me speed this up, such as options for dd or something else I may be missing, or if the speed is just a limitation of the HD.

5
  • Change your block size to something more friendly toward hard drives. dd bs=1M for example.
    – phemmer
    Apr 12, 2013 at 16:06
  • What speed were you getting? It takes a while to write an entire 3TB (for example) HDD. Also check iostat -kx 10 to see what busy % on the drive is.
    – derobert
    Apr 12, 2013 at 17:53
  • 7
    shred -v -n 1 /dev/overwritethis is fast. It's about the only case where shred is actually useful for something. Apr 12, 2013 at 19:14
  • @derobert: I can't really say for sure how fast it was, but I left for a few hours, came back, and it was around 10% complete for my 500G internal HD the first time I tried this. Thanks for the "iostat" tip btw
    – user22531
    Apr 12, 2013 at 20:47
  • @Patrick: I tried bs=4M sort of blindly, since I saw that on the guide for how to put the Arch CD on a usb. It helped slightly, but it was still pretty slow.
    – user22531
    Apr 12, 2013 at 20:49

6 Answers 6

16

dd if=/dev/urandom of=/dev/sda, or simply cat /dev/urandom >/dev/sda, isn't the fastest way to fill a disk with random data. Linux's /dev/urandom isn't the fastest cryptographic RNG around. Is there an alternative to /dev/urandom? has some suggestions. In particular, OpenSSL contains a faster cryptographic PRNG:

openssl rand $(</proc/partitions awk '$4=="sda" {print $3*1024}') >/dev/sda

Note that in the end, whether there is an improvement or not depends on which part is the bottleneck: the CPU or the disk.

The good news is that filling the disk with random data is mostly useless. First, to dispel a common myth, wiping with zeroes is just as good on today's hardware. With 1980s hard disk technology, overwriting a hard disk with zeroes left a small residual charge which could be recovered with somewhat expensive hardware; multiple passes of overwrite with random data (the “Gutmann wipe”) were necessary. Today even a single pass of overwriting with zeroes leaves data that cannot realistically be recovered even in laboratory conditions.

When you're encrypting a partition, filling the disk with random data is not necessary for the confidentiality of the encrypted data. It is only useful if you need to make space used by encrypted data indistinguishable from unused space. Building an encrypted volume on top of a non-randomized container reveals which disk blocks have ever been used by the encrypted volume. This gives a good hint as to the maximum size of the filesystem (though as time goes by it will become a worse and worse approximation), and little more.

10
  • 5
    Gutmann is a myth in its entirety, I don't think it's actually ever been done for a 1980s hard disk either. The irony is that with drives getting smarter, you actually should use random data nowadays to make sure the drive is forced to write, rather than free sector (trim) or compress data. Zeroes are only good if they're actually written. Apr 12, 2013 at 20:07
  • 1
    @mellowmaroon Yes, cat /dev/zero is almost always enough. It's only not enough if you want to hide how much space is free on the encrypted volume. Apr 13, 2013 at 9:02
  • 1
    @mellowmaroon It's pretty much useless. An eavesdropper will know that you have at most X MB of data (and possibly a lot less, because formerly-used-but-now-free space is indistinguishable from used space), so what? Also the location of the free space might reveal the filesystem type (superblock copies); that's rarely a concern (it's typically exposed in the cleartext /etc/fstab, unless you've encrypted the root partition, and even then there aren't such a large number of plausible options). Apr 14, 2013 at 8:56
  • 2
    @Gilles The problem I ran into in Ubuntu 13.10 is that openssl rand seems to have an upper limit on the number of bytes it will generate. If you exceed that limit, e.g. 'openssl rand 810000000000, then no random output is generated. Only a brief "help" text is printed. I'm trying to random (pretty much) fill a 3TB hard drive. Not sure if there is a way to get openssl` to produce that many random bytes. Jan 17, 2014 at 5:08
  • 1
    For irrational John's upper limit problem, 810 billion bytes comes out to around 754 GB. How about partitioning your disk into multiple 700-GB partitions, then running the openssl rand command on each partition in reverse starting from sda5 or whatever and working backwards to sda1 and finally sda?
    – jia103
    Dec 24, 2015 at 18:31
8

You can get OpenSSL to encrypt /dev/zero with a randomized password, giving decent pseudorandom data very fast (if your CPU supports accelerating it).

openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero | dd of=/dev/sda

You could pipe this through pv to get progress/ETA. The commands I'm running right now (in a root shell) are:

DISK="sda"
DISKSIZE=$(</proc/partitions awk '$4=="'"$DISK"'" {print sprintf("%.0f",$3*1024)}')
apt-get install pv
openssl enc -aes-256-ctr -nosalt \
  -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" \
  < /dev/zero |
  pv --progress --eta --rate --bytes --size "$DISKSIZE" |
  dd of=/dev/"$DISK" bs=2M

I got this idea from this answer, after having the same problem as irrational John, who commented on Gilles's answer above. This increased my wipe speed to my new RAID array from 11 MB/s to around 300 MB/s, taking what was going to take a week down to 10 hours.

I'll add that you should be able to use openssl rand #of_bytes rather than the more complicated openssl enc ... statement above, but there is a bug which will allow ssl to produce only 16 MB of output. (This bug has been filed, Jan 2016.)

And, as per the answer to this question, and continuing to assume that the CPU is the bottleneck, it may be possible to increase speed further by running multiple parallel openssl processes on separate cores, combining them using a FIFO.

2
  • Not sure the edit was necessary. Other answers to this question suggest openssl rand.
    – tremby
    Jan 11, 2016 at 7:35
  • 3
    Benchmark: dd if=/dev/urandom: 18MiB/s, openssl enc: ~180MiB/s, fio: 169MiB/s, openssl rand no support > 754GB. Note that if you also want auto-size calculation, use: openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt </dev/zero | pv --progress --eta --rate --bytes --size $(</proc/partitions awk '$4=="sda" {print sprintf("%.0f",$3*1024)}') | dd of=/dev/sda bs=2M. Beware, sda is present twice in this command.
    – KrisWebDev
    Mar 19, 2016 at 22:00
6

The openssl did not seem to work for me. I got "unknown options" and other issues with the provided solutions. So I ended up going with the program fio.

fio -name="fill" -ioengine=libaio -direct=1 -bs=512m -rw=write -iodepth=4 -size=100% -filename=/dev/md0

Which seems to be taking 3 hours to do 19TB across 24 HDDs. So roughly 1,800 MB/s

smp-016:~ # fdisk -l /dev/md0
Disk /dev/md0: 18890.1 GB, 18890060464128 bytes

smp-016:~ # fio -name="fill" -ioengine=libaio -direct=1 -bs=512m -rw=write -iodepth=4 -size=100% -filename=/dev/md0
fill: (g=0): rw=write, bs=512M-512M/512M-512M/512M-512M, ioengine=libaio, iodepth=4
fio-2.2.10
Starting 1 process
Jobs: 1 (f=1): [W(1)] [2.7% done] [0KB/1536MB/0KB /s] [0/3/0 iops] [eta 03h:01m:11s]

I hope this is actually random data. The man page says fio "Default: fill buffers with random data." http://linux.die.net/man/1/fio

I'm not doing it for secure/encryption purposes, just trying to be sure my later read tests are actual data and not just 0's. This same fio command could be used for SSD/NVMe preconditioning. As just using /dev/zero can lead to disk level compression "cheating" how much is actually written. Although I would add a -loops=2 flag to it, if it is a fresh SSD for benchmarking.

If you did want it to be secure you may be able to use the -randrepeat=bool option, as that will toggle "Seed the random number generator in a predictable way so results are repeatable across runs. Default: true.", but I'm still not certain how secure that would be.

Additionally some enterprise class HDDs out there are SED (Self Encrypting Drives) and will allow you to spin the encryption key to instantly and securely erasing all the data written.

Lastly, I have in the past used DBAN (aka Darik's Boot and Nuke), which has CD and USB bootable options and "is an open source project hosted on SourceForge. The program is designed to securely erase a hard disk until its data is permanently removed and no longer recoverable"

2
  • 1
    The size=100% didn't work for me, but fill_device=1 (or fill_fs=1) did.
    – d.jamison
    Dec 7, 2016 at 19:55
  • I think thats would be dependent if your giving it a block level device or a filesystem to fill up. With the block level device it knows how big it is and the size is a percent of the device size. Where as the fill_device goes until it gets a device full error. I think the fill_device flag may not overwrite any files, just unused space. I haven't put that to the test, as I usually avoid filesystems unless necessary when I do device testing. Dec 9, 2016 at 22:54
4

I was trying to fill a 4TB external USB HDD with dd if=/dev/urandom of=/dev/sdX status=progress which was way too slow (irrespective of bs= settings), and it seems that openssl has a cap on how much random data it will output (at least for version 1.0.2p). The best option I found was the comment from frostschutz to use shred, as in:

shred -v -n 1 /dev/sdX

Make sure to use the -n 1 otherwise it will default to writing the device 3 times over (plus the -v which shows progress). I don't think the quality of the pseudo-random numbers is that high, but it's enough for me to prepare a large capacity portable HDD for encryption.

0

Completing Marco's answer, what you need is faster random number generator.

You use a simple program that echos random numbers from a good library like boost::random and use that one in dd.

If you choose boost, you can use this example, changing the experiment function to your needs.

3
  • How much faster is the boost solution on your system? A quick non-scientific benchmark on my machine yields exactly the same speed as /dev/urandom.
    – Marco
    Apr 12, 2013 at 17:10
  • boost::random doesn't offer a crypto RNG, does it? If you're going to use a non-crypto RNG, you might as well use zeroes: at least you won't have an illusion of security. Apr 12, 2013 at 18:37
  • I can't be specific about how much faster boost::random generators are, the only way to know for sure is measure their fastest algorithm against /dev/urandom
    – RSFalcon7
    Apr 12, 2013 at 21:48
0

The bottleneck if neither the block size nor the hard drive, but the slow generation of pseudo-random numbers. /dev/urandom is by magnitudes faster compared to /dev/random since it's not blocking on a low entropy pool.

You can confirm this by measuring the raw output of your pseudo-random numbers:

pv /dev/urandom >/dev/null

This rate will be much slower than the write rate of your hard drive. A correct solution totally depends on your required level of security. If you require high security either use a fast hardware random generator or accept the slow speed. If your security needs are not that high, you could capture a few dozen MiB of data and write that string repeatedly to the drive. Or maybe even writing zeros from /dev/zero is an option.

Summary

/dev/random - secure, very slow
/dev/urandom - less secure¹, slow
hardware RNG - secure, fast, very expensive
(/dev/zero - not random at all, very fast)

¹According to Is a rand from /dev/urandom secure for a login key? /dev/urandom is as secure as /dev/random. Thanks to Gilles for pointing this out.

4

You must log in to answer this question.