How can I speed up secure erasing of a disk?

Question

I need to securely erase harddisks from time to time and have used a variety of tools to do this:

• cat /dev/zero > /dev/disk
• cat /dev/urandom > /dev/disk
• shred
• badblocks -w
• DBAN

All of these have in common that they take ages to run.

In one case cat /dev/urandom > /dev/disk killed the disk, apparently overheating it.

Is there a "good enough" approach to achieve that any data on the disk is made unusable in a timely fashion? Overwriting superblocks and a couple of strategically important blocks or somesuch?

The disks (both, spinning and ssd) come from donated computers and will be used to install Linux-Desktops on them afterwards, handed out to people who can't afford to buy a computer, but need one.

The disks of the donated computers will usually not have been encrypted. And sometimes donors don't even think of deleting files beforehand.

Update:

From the answers that have come in so far, it seems there is no cutting corners. My best bet is probably setting up a lab-computer to erase multiple disks at once. One more reason to ask big companies for donations :-)

Thanks everyone!

Answer 1

Overwriting the superblock or partition table just makes it inconvenient to reconstruct the data, which is obviously still there if you just do a hex dump.

Hard disks have a built-in erasing feature: ATA Secure Erase, which you can activate using hdparm:

1. Pick a password (any password):

   hdparm --user-master u --security-set-pass hunter1 /dev/sdX

2. Initiate erasure:

   hdparm --user-master u --security-erase hunter1 /dev/sdX

Since this is a built-in feature, it is unlikely that you'll find a faster method that actually offers real erasure. (It's up to you, though, to determine whether it meets your level of paranoia.)

Alternatively, use the disk with full-disk encryption, then just throw away the key when you want to dispose of the data.

Answer 2

Potentially unpopular answer: Put up with the time to completely wipe the disks securely.

You've accepted donation hardware from various sources, and made promises that their data will be securely erased. So you have to do as you promised and securely erase the disks.

If I was a donor and I found that you weren't keeping your promises, I'd not give you any more hardware in the future.

And don't short the DBAN runtime either thinking that clearing the first half or first 10% is sufficient, when it is not.

---

To make your DBAN runs more convenient, do you generally remove drives from hosts? If so, have a fairly low-power desktop aside with multiple SATA ports. When you have a couple of drives ready for wiping, set them all up on this one machine and wipe them with a bootable DBAN ISO or a netboot DBAN session. Then leave the machine running until the last one is complete.

You could use a SATA-USB disk dock, but these tend to be really slow compared to native SATA connections.

Second, do you have a workflow of storing donations then processing them? Or do you accept the hardware in, test it, wipe it and then store the items for use? I'd suggest you make a pipeline that works for you, and make DBAN a stage in that pipeline. So when you pull a machine from the store, its already wiped as well as known-good.

Answer 3

dd with big blocks could be faster than cat:

dd if=/dev/zero of=/dev/disk bs=16M

Killing a disk in the process is goodness, it was likely marginal anyway.

If you LUKS-encrypt, this can be done after setting-up the encryption (makes it harder to spot used/unused space on the disk).

You can't avoid the I/O, at 50MB/s (which is a good speed for old disks), 250GB takes 5000 seconds (about 1.5 hours) to write. The only question is how close to that you can get.

Answer 4

blkdiscard - discard sectors on a device

I haven't tried this, but from a quick read, you could try blkdiscard.

This will just send simple commands to the drive and should be very fast, but I have not found any guarantee that the data in the blocks will be erased. There are some hints though.

• From the manpage:

WARNING: All data in the discarded region on the device will be lost!

• From wikipedia:

TRIM tells the SSD to mark a LBA region as invalid and subsequent reads on the region will not return any meaningful data. For a very brief time, the data could still reside on the flash internally. However, after the TRIM command is issued and garbage collection has taken place, it's highly unlikely that even a forensic scientist would be able to recover the data.

I would obviously not trust this for serious business, but since your requirement is "good enough", it might be a viable option. Presumably it only works on SSDs as well, which may be a more serious hurdle if you're dealing with old donated computers.

Answer 5

From security (paranoic) point of view, it's probably not possible.

There is one scenario when you can rewrite an only a small part of the disk, and most people will consider it save. If a drive was encrypted overwriting the encryption header (a place where the encryption key is stored) should make it impossible to recover.

But there is another problem - the SSD can potentially keep header data (the random data that was used to overwrite header can be saved to other cells).

Answer 6

Most companies that care about data security already encrypt their drives, would (should) not give you drive encryption keys, so drive data is already effectively 'erased' anyway.

Parallel erasure would resolve the biggest speed issue. SATA I/O speed is not your bottleneck, HDD physical write speed is the limit. Modern motherboards have at least 4-8 SATA drive connections. Set up one (or more) lab computer(s) to boot from SSD (fast). Connect 4-6 drives to be erased, then use preferred method to erase from provided approaches.

Suppose you have 4 drives (b,c,d,e)... then a simple script would erase all drives in parallel. And the lab computers enable you to log the drive erasures.

ts=$(date +%Y%m%d%H%M%S)
mkdir -p erase-${ts}
# figure out where your linux mounts the drives (avoid trashing your SSD)
for drive in b c d e; do
    target="sd$drive"
    dd if=/dev/zero of=/dev/${target} bs=16M 2>&1 |tee erase-${ts}/${target} &
done

Although, I would run each drive erasure in a separate terminal, so I could watch for errors on specific drives. Drives with more than a few errors should be replaced. The SMART info from drives will give you an idea of drive age.

If you have 2-4 lab computers configured, the challenge would be keeping enough drives loaded in the lab computers to keep them busy erasing drives. Suppose you have 5 drives per lab computer X 3 lab computers = 15 drives being erased in parallel. You are finishing a batch every 30 (ish) minutes

All bets are off with SSD, they work differently. Most have much faster write speed. And none of the above erases sectors that have been mapped out of use by bad-block detection (some data may not be erased anyway).

There have been several studies done, notably a rather large study by Google, that found that after the first two years, the probability of hard drive failure reached 5-6% (per year) and increased significantly in subsequent years. Any disk over 5 years old has a substantial failure probability. Consider just retiring (destroy, recycle) hard drives over five years old.

My advice would be to only keep drives three or fewer years old, Or replace all the drives, and retire (destroy or recyle).

Answer 7

Frame challenge:

Stop disassembling the PCs

I've disassembled a lot of PCs in my life. You're complaining about time, but you're wasting time disassembling PCs for imaging -- every PC has to be benched twice, then re-tested.

And your burn-in period is weak tea. You're only putting the system together long enough to configure and test your newly imaged drive. So you'll be shipping defective PCs to customers (many are given to you for a reason; niggling glitches e.g. thermal problems are one such.)

Change your assembly line. Rack the PC under power, attach a KVM, and use the PC to wipe the PC. That will also give it a handy "ran it for awhile" burn-in; and if you use /dev/urandom a stout CPU/thermal burn-in as well.

You can either boot up under their Windows environment and delete all but the system files and applications, or you can boot off a CD, or if the system is capable of it, booting off a USB stick. Or, you can image the system first and wipe the freespace after your new image is installed.

If running natively under the legacy OS, there is no need to delete and wipe Windows system files and applications, e.g. World of Warcraft is 40GB of data that doesn't need to be seecure-wiped. So \Windows, \Program Files and other system directories can be excluded from the security wipe. (If they put data there, they can't fairly expect a secure wipe of it). There's a highly portable version of perl 4 that would make easy work out of this, and perl 4 is plenty powerful enough for this kind of thing.

If I was using the native OS, I'd do it in 2 passes: First delete all non-system non-app files; then fill the disk with zeroes (or copies of a system file if you're worried about CIA-tier disk forensics).

Or wipe it last, after imaging

Same thing; still use the PC itself to image, e.g. booting from a CD and pulling the image across the ethernet or whatever.

Now, build the image with an action-on-first-bootup, where it wipes the freespace, then disables itself.

All data is overwritten either by a) the files of the new image, or b) your overwriting of freespace.

If you want this function to be "check-point-able", i.e. not have to start over after a power failure, then have it simply write immense files with random names into /tmp/ until the disk is full. On confirming the disk is full, have it remove the /tmp/ files, then remove itself.

Once the disk is 100% occupied with new data, the old data must have been overwritten.

Answer 8

Even on a device where the filesystem has been trashed, it is often possible to recover, say, JPG images solely by reading the device.

To make that harder maybe you can overwrite all starts of files with data.

Something like:

mount /dev/disk /mnt
find /mnt -type f |
  parallel --pipe --block 8k --roundrobin -q perl -ne 'chomp;open(A,"+<",$_);print A "0"x8192'

If there are only few files, this would be faster than overwriting the full drive.

Answer 9

Update for those searching for an opensource DBAN replacement / alternative

I would also recommend to go the real wiping procedure of the entire disk instead of just zero filling. If someday I'd have to trust someone to destroy my data, I'd like that person to be trustworthy about doing the job right.

Anyway, since DBAN has not seen any developpment since 2015 and it's commercial successor is not open source, I was searching for a good solution that would do the job.

Found nwipe, a fork of DBAN's included dwipe program, that just happens to stay open source.

A full featured bootable OS called ShredOS2020 that has the same look and feel than DBAN, based on nwipe can be downloaded from ShredOS github

Hope this will save searching time for some of you guys ;)

Answer 10

/dev/zero is fast, /dev/urandom is slow and cpu-heavy.

Both of them are better used with dd using a large block (bs=1M) than with cat. cat has no understanding of blocks, it moves one character at a time, using a lot of cpu cycles.

shred is the tool optimized for that purpouse, doing basically what dd does in an organized way. It uses less "random" data (compared to getting everything from /dev/urandom) when overwriting the disk with random data - still random enough to push an SSD out of spare blocks and make HDD magnetic analisys hard by cross-character interference, but less cpu-hungry to generate.

I never used DBAN, but from what I read it seems basically a shred-like tool in a bootable wrapper.

Then again, there is ATA SECURITY_ERASE command (and even SECURITY_ERASE_ENHANCED command), accessible in hdparm. They simply write 0's to the disk without the need to push them in the interface. On HDD, they are as fast as dd (hours). On SSD, they are blkdiscard on the whole disk. The enhanced variety tries to zero also the blocks marked as bad.

blkdiscard - works on SSD only. Marks blocks as free so they appear as 0's, there is no guarantee when they will be erased internally.

Answer 11

Don't install an OS, image it

If you are going to put GNU/Linux on it anyway, you could do both in one step: take a drive (or image) with a fresh install, and clone it to the new disk.

If your OS takes e.g. 8GB, then that's 8GB of space you don't need to blank out first. Not a lot compared to 1TB I admit, but it might save you some install time as well. The more you preinstall the better ;)

If the source image is too short to cover the whole disk (better too short than too long), you can use dd to blank out the rest of the disk. And then resize the partition after cloning, if you like.

If you want to use an image file rather than cloning a disk, you can use a useful feature called "sparse files" to cut down the file size wherever the blocks are all zeros in the image.

Answer 12

The disks (both, spinning and ssd) come from donated computers and will be used to install Linux-Desktops on them afterwards, handed out to people who can't afford to buy a computer, but need one.

Okay, so your long-term solution is this (it's a variation on DBAN)

• prepare a Linux install self-bootable CD (using e.g. Anaconda)
• write a script that will:
  • run memtest86 and other simple diagnostics
  • recognize the hardware (just the CPU, whether it's 32 or 64 bits)
  • zero the whole hard disk using dd
  • partition the hard disk into system, swap, and data
  • install the appropriate Linux distribution version onto system
  • eject the CD

Now say three PCs come in the lab, you turn them on, place a copy of the CD in the tray, boot from the CD and continue doing whatever you were doing before, until they warn you that the work is done. Years ago I did something like that: the script would play three long beeps immediately upon starting (so I knew that the speaker worked, they were desktops and I didn't want to connect a monitor unless I really had to), then other sounds depending on the exit status.

Basically, every computer took about one minute of work - forty seconds to boot it up, twenty seconds to shut it down when it started playing either "Reveille" or "Taps". I had no privacy issues there, so I never added the dd step; but it should be easy to implement. Even if my original proposal, as @PeterCordes noticed, was flawed.

Answer 13

Set up a Raspberry Pi to automatically erase any USB storage plugged into it, and light up a LED when it's done. Then clone the SD card and buy a bunch of Raspberries to work in parallel. If you choose to use the ATA secure erase, with 4 devices you will be able to erase drives as fast as you can plug them. If you choose to overwrite the entire device, it will still be slow, but still 4 times as fast compared to erasing one drive at a time.

The same can be done with regular laptops if you have them in your donation box as well, saving you the cost of hardware.

Answer 14

From the answers that have come in so far, it seems there is no cutting corners.

Here's an answer that cuts corners.

Does the incoming computer have an SSD? Find software that supports "ATA Secure Erase" and run it on the drive. What this does is tell the SSD to wipe it's internal encryption key, making the contents completely unreadable. Takes 30 seconds and is good enough for anything less than nuclear secrets.

Does the incoming computer have a spinning drive? Remove it, drill a couple of holes through the case and platter and send it to the recycle center. Get an SSD + mounting bracket from Amazon for about $50. This is far less than your time to process the old drives and sill provide much better performance. "Donated computers" implies an education environment, 200GB is plenty big enough.

And if you haven't got one already, get a 2-port hard drive dock that supports cloning. Plug in the master disk and a new blank one, press the copy button. It does a block copy so is completely OS independent. Also helps bulk-erasing the incoming SSDs.

Answer 15

You may want to parallelize it. For instance in a 8 core processor you may use the following command line:

for i in {1..8}; do dd bs=16M if=/dev/zero of=$i & done

Note: Check your HDD specs sheet and adjust buffer accordingly.

In my case my HDD has a 128MB buffer size, hence a 128MB/8 core=16MB buffer.