32

I need to securely erase harddisks from time to time and have used a variety of tools to do this:

  • cat /dev/zero > /dev/disk
  • cat /dev/urandom > /dev/disk
  • shred
  • badblocks -w
  • DBAN

All of these have in common that they take ages to run.

In one case cat /dev/urandom > /dev/disk killed the disk, apparently overheating it.

Is there a "good enough" approach to achieve that any data on the disk is made unusable in a timely fashion? Overwriting superblocks and a couple of strategically important blocks or somesuch?

The disks (both, spinning and ssd) come from donated computers and will be used to install Linux-Desktops on them afterwards, handed out to people who can't afford to buy a computer, but need one.

The disks of the donated computers will usually not have been encrypted. And sometimes donors don't even think of deleting files beforehand.

Update:

From the answers that have come in so far, it seems there is no cutting corners. My best bet is probably setting up a lab-computer to erase multiple disks at once. One more reason to ask big companies for donations :-)

Thanks everyone!

  • 2
    alternatively, encrypting the full disk adds a layer of security (one can store the key on an other disk, be the key itself encrypted or not) – A.B Nov 20 at 13:06
  • 1
    True, but that still require the disk to be securely erased before. I'm handing these out to people I don't know. – markgraf Nov 20 at 13:53
  • 2
    Please clarify... You say hard disks - are they spinning platters or SSDs ? – Criggie Nov 21 at 0:38
  • 23
    If dd killed the disk, then that disk was on its last legs anyway and it's a good thing that it died during wiping instead of when you were relying on it for data. – SurpriseDog Nov 21 at 2:48
  • 6
    If I had a hammer, / I'd securely erase in morning, / I'd securely erase in the evening... – RonJohn Nov 21 at 16:01

14 Answers 14

40

Overwriting the superblock or partition table just makes it inconvenient to reconstruct the data, which is obviously still there if you just do a hex dump.

Hard disks have a built-in erasing feature: ATA Secure Erase, which you can activate using hdparm:

  1. Pick a password (any password):

    hdparm --user-master u --security-set-pass hunter1 /dev/sdX

  2. Initiate erasure:

    hdparm --user-master u --security-erase hunter1 /dev/sdX

Since this is a built-in feature, it is unlikely that you'll find a faster method that actually offers real erasure. (It's up to you, though, to determine whether it meets your level of paranoia.)

Alternatively, use the disk with full-disk encryption, then just throw away the key when you want to dispose of the data.

  • 15
    It's important to note that secure erase can only run when the hard drive has a password set, and secure erasing the drive also erases the password that you just set. I should also note that some early SATA drives (from the 2000s) didn't implement it correctly and only erased part of the disk, or bricked themselves. I don't think that's a problem today, but if you really want to be sure, you should probably read out the entire drive afterward. For erasing a modern SSD this is the only thing you should be doing. – Michael Hampton Nov 21 at 4:28
  • 4
    You got the password wrong! It's supposed to be hunter2! /s – Colonel Thirty Two Nov 21 at 15:47
  • 4
    @johndoe Nothing more is needed than to zero out all bits on the drive, unless you are trying to erase ancient MFM/RLL drives and early IDE drives. But these don't have secure erase anyway. – Michael Hampton Nov 21 at 19:04
  • 2
    @cartographer It seems you're referring to SSDs, while this question and most of the discussion has been about spinning rust. I'd be interested in hearing about more modern rotational drives which don't implement secure erase properly. – Michael Hampton Nov 21 at 20:28
  • 2
    Secure erase still fails to securely erase in a number of scenarios. Some drives, for example, will stop the secure erase the second they hit a bad block, leaving the drive partially erased. The only responsible thing to do is to use a program that wipes and verifies the wipe. – Mr. T Nov 23 at 11:12
17

Potentially unpopular answer: Put up with the time to completely wipe the disks securely.

You've accepted donation hardware from various sources, and made promises that their data will be securely erased. So you have to do as you promised and securely erase the disks.

If I was a donor and I found that you weren't keeping your promises, I'd not give you any more hardware in the future.

And don't short the DBAN runtime either thinking that clearing the first half or first 10% is sufficient, when it is not.


To make your DBAN runs more convenient, do you generally remove drives from hosts? If so, have a fairly low-power desktop aside with multiple SATA ports. When you have a couple of drives ready for wiping, set them all up on this one machine and wipe them with a bootable DBAN ISO or a netboot DBAN session. Then leave the machine running until the last one is complete.

You could use a SATA-USB disk dock, but these tend to be really slow compared to native SATA connections.

Second, do you have a workflow of storing donations then processing them? Or do you accept the hardware in, test it, wipe it and then store the items for use? I'd suggest you make a pipeline that works for you, and make DBAN a stage in that pipeline. So when you pull a machine from the store, its already wiped as well as known-good.

  • 2
    Given the context I think this ^^^ is the correct answer. – Ole Tange Nov 21 at 8:57
  • 1
    Workflow is: Dis-assemble, wipe the disks, then put things back together, install, test. As it looks right now, I'm going to stick with completely overwriting... – markgraf Nov 21 at 10:37
  • 1
    @markgraf wth? stop doing that. You say time is a factor, yet here you are disassembling PCs for no reason. Look, you need to do a PC burn-in anyway. So rack the PC near a KVM, and use the PC to wipe the PC. You don't need to clobber system files, obviously; that'll happen when you image on the new OS. – Harper - Reinstate Monica Nov 23 at 16:58
12

dd with big blocks could be faster than cat:

dd if=/dev/zero of=/dev/disk bs=16M

Killing a disk in the process is goodness, it was likely marginal anyway.

If you LUKS-encrypt, this can be done after setting-up the encryption (makes it harder to spot used/unused space on the disk).

You can't avoid the I/O, at 50MB/s (which is a good speed for old disks), 250GB takes 5000 seconds (about 1.5 hours) to write. The only question is how close to that you can get.

  • 2
    If the disk dies you don't know whether the data is recoverable (it probably is with special equipment or a controller transplant) so you'll need to physically destroy it. – user253751 Nov 20 at 23:02
  • 5
    I don't think the OP is worrying about that... And in any case destroying the disk is a matter of seconds with an electric drill. – xenoid Nov 20 at 23:11
7

blkdiscard - discard sectors on a device

I haven't tried this, but from a quick read, you could try blkdiscard.

This will just send simple commands to the drive and should be very fast, but I have not found any guarantee that the data in the blocks will be erased. There are some hints though.

  • From the manpage:

WARNING: All data in the discarded region on the device will be lost!

TRIM tells the SSD to mark a LBA region as invalid and subsequent reads on the region will not return any meaningful data. For a very brief time, the data could still reside on the flash internally. However, after the TRIM command is issued and garbage collection has taken place, it's highly unlikely that even a forensic scientist would be able to recover the data.

I would obviously not trust this for serious business, but since your requirement is "good enough", it might be a viable option. Presumably it only works on SSDs as well, which may be a more serious hurdle if you're dealing with old donated computers.

6

From security (paranoic) point of view, it's probably not possible.

There is one scenario when you can rewrite an only a small part of the disk, and most people will consider it save. If a drive was encrypted overwriting the encryption header (a place where the encryption key is stored) should make it impossible to recover.

But there is another problem - the SSD can potentially keep header data (the random data that was used to overwrite header can be saved to other cells).

  • 3
    Actually in a LUKS-encrypted disk you can tell LUKS to destroy all keys, no need to even overwrite the header. – xenoid Nov 20 at 13:15
  • 3
    SSD definitely are a problem in that regard. So no way around DBAN there, apparently. – markgraf Nov 20 at 13:29
  • 4
    @markgraf DBAN is an exceedingly bad idea on SSDs due to the way their controllers and wear leveling (and reserved storage and so on) works. For SSDs you have to rely on tools provided by the manufacturer (and hope they don't contain bugs, so yeah if it's really important you'll have to shred it) – Voo Nov 22 at 20:40
5

Most companies that care about data security already encrypt their drives, would (should) not give you drive encryption keys, so drive data is already effectively 'erased' anyway.

Parallel erasure would resolve the biggest speed issue. SATA I/O speed is not your bottleneck, HDD physical write speed is the limit. Modern motherboards have at least 4-8 SATA drive connections. Set up one (or more) lab computer(s) to boot from SSD (fast). Connect 4-6 drives to be erased, then use preferred method to erase from provided approaches.

Suppose you have 4 drives (b,c,d,e)... then a simple script would erase all drives in parallel. And the lab computers enable you to log the drive erasures.

ts=$(date +%Y%m%d%H%M%S)
mkdir -p erase-${ts}
# figure out where your linux mounts the drives (avoid trashing your SSD)
for drive in b c d e; do
    target="sd$drive"
    dd if=/dev/zero of=/dev/${target} bs=16M 2>&1 |tee erase-${ts}/${target} &
done

Although, I would run each drive erasure in a separate terminal, so I could watch for errors on specific drives. Drives with more than a few errors should be replaced. The SMART info from drives will give you an idea of drive age.

If you have 2-4 lab computers configured, the challenge would be keeping enough drives loaded in the lab computers to keep them busy erasing drives. Suppose you have 5 drives per lab computer X 3 lab computers = 15 drives being erased in parallel. You are finishing a batch every 30 (ish) minutes

All bets are off with SSD, they work differently. Most have much faster write speed. And none of the above erases sectors that have been mapped out of use by bad-block detection (some data may not be erased anyway).

There have been several studies done, notably a rather large study by Google, that found that after the first two years, the probability of hard drive failure reached 5-6% (per year) and increased significantly in subsequent years. Any disk over 5 years old has a substantial failure probability. Consider just retiring (destroy, recycle) hard drives over five years old.

My advice would be to only keep drives three or fewer years old, Or replace all the drives, and retire (destroy or recyle).

  • I may be wrong about this, but my understanding was that data on an SSD is heavily fragmented, because fragmentation doesn't matter with super-low latency seeks (contra spinning disks). As such, to securely erase an SSD, you really just need to wipe the file table. – Xophmeister Nov 21 at 12:06
  • 1
    @Xophmeister That doesn't erase the data then, that just turns it into a jigsaw puzzle. And a credit card number or a session cookie could fit on one piece. (blocks are probably 4096 bytes) – user253751 Nov 21 at 12:41
  • @Xophmeister, SSD's are interesting and different from spinning disks. But data in blocks (even in bad-blocks) still exists, and a dedicated hacker may be able to access the data (think, reprogram the micro-controller) – ChuckCottrill Nov 22 at 21:49
3

Frame challenge:

Stop disassembling the PCs

I've disassembled a lot of PCs in my life. You're complaining about time, but you're wasting time disassembling PCs for imaging -- every PC has to be benched twice, then re-tested.

And your burn-in period is weak tea. You're only putting the system together long enough to configure and test your newly imaged drive. So you'll be shipping defective PCs to customers (many are given to you for a reason; niggling glitches e.g. thermal problems are one such.)

Change your assembly line. Rack the PC under power, attach a KVM, and use the PC to wipe the PC. That will also give it a handy "ran it for awhile" burn-in; and if you use /dev/urandom a stout CPU/thermal burn-in as well.

You can either boot up under their Windows environment and delete all but the system files and applications, or you can boot off a CD, or if the system is capable of it, booting off a USB stick. Or, you can image the system first and wipe the freespace after your new image is installed.

If running natively under the legacy OS, there is no need to delete and wipe Windows system files and applications, e.g. World of Warcraft is 40GB of data that doesn't need to be seecure-wiped. So \Windows, \Program Files and other system directories can be excluded from the security wipe. (If they put data there, they can't fairly expect a secure wipe of it). There's a highly portable version of perl 4 that would make easy work out of this, and perl 4 is plenty powerful enough for this kind of thing.

If I was using the native OS, I'd do it in 2 passes: First delete all non-system non-app files; then fill the disk with zeroes (or copies of a system file if you're worried about CIA-tier disk forensics).

Or wipe it last, after imaging

Same thing; still use the PC itself to image, e.g. booting from a CD and pulling the image across the ethernet or whatever.

Now, build the image with an action-on-first-bootup, where it wipes the freespace, then disables itself.

All data is overwritten either by a) the files of the new image, or b) your overwriting of freespace.

If you want this function to be "check-point-able", i.e. not have to start over after a power failure, then have it simply write immense files with random names into /tmp/ until the disk is full. On confirming the disk is full, have it remove the /tmp/ files, then remove itself.

Once the disk is 100% occupied with new data, the old data must have been overwritten.

  • 1
    Use /dev/urandom. /dev/random blocks for "quality" entropy, of which there isn't much on a system without rdrand or other HW RNG. But yes, beyond opening up the case to look for obvious problems like dust-clogged fans, don't disassemble. – Peter Cordes Nov 23 at 18:32
  • Can't do without the disassembling, unfortunately. The components need to be dusted off (literally). And sometimes parts need to be mixed and matched. "Wiping last" is a new approach, though. I'll bear that in mind. Thanks! – markgraf Nov 25 at 14:20
2

Even on a device where the filesystem has been trashed, it is often possible to recover, say, JPG images solely by reading the device.

To make that harder maybe you can overwrite all starts of files with data.

Something like:

mount /dev/disk /mnt
find /mnt -type f |
  parallel --pipe --block 8k --roundrobin -q perl -ne 'chomp;open(A,"+<",$_);print A "0"x8192'

If there are only few files, this would be faster than overwriting the full drive.

  • 1
    Wouldn't erase anything that the original user has deleted. – pipe Nov 20 at 21:12
  • 1
    True. By no means perfect. – Ole Tange Nov 20 at 21:25
1

/dev/zero is fast, /dev/urandom is slow and cpu-heavy.

Both of them are better used with dd using a large block (bs=1M) than with cat. cat has no understanding of blocks, it moves one character at a time, using a lot of cpu cycles.

shred is the tool optimized for that purpouse, doing basically what dd does in an organized way. It uses less "random" data (compared to getting everything from /dev/urandom) when overwriting the disk with random data - still random enough to push an SSD out of spare blocks and make HDD magnetic analisys hard by cross-character interference, but less cpu-hungry to generate.

I never used DBAN, but from what I read it seems basically a shred-like tool in a bootable wrapper.

Then again, there is ATA SECURITY_ERASE command (and even SECURITY_ERASE_ENHANCED command), accessible in hdparm. They simply write 0's to the disk without the need to push them in the interface. On HDD, they are as fast as dd (hours). On SSD, they are blkdiscard on the whole disk. The enhanced variety tries to zero also the blocks marked as bad.

blkdiscard - works on SSD only. Marks blocks as free so they appear as 0's, there is no guarantee when they will be erased internally.

  • 1
    "cat has no understanding of blocks, it moves one character at a time, using a lot of cpu cycles." My last impression when reading the source code of cat years ago was different. – glglgl Nov 21 at 13:49
  • 1
    cat should be using reasonable blocksize like 64-128K (if in doubt, check with strace). urandom has improved a lot, it's fast enough to overwrite one HDD at full speed. but yes, zero, shred, or producing random data by encrypting zero (with AESNI), is much faster than urandom. – frostschutz Nov 21 at 14:41
  • There is more than one cat in the first place. How about busybox? – fraxinus Nov 21 at 15:40
  • busybox cat seems to be using 4K blocks – frostschutz Nov 21 at 18:20
  • Do you have any references that HDD magnetic analysis is at all possible? I've never seen reputable sources for it, whereas I have seen plenty of reputable sources arguing that a simple zero write is sufficient to ensure complete overwriting of at least a mechanical HDD. – Hashim Nov 22 at 21:56
1

Don't install an OS, image it

If you are going to put GNU/Linux on it anyway, you could do both in one step: take a drive (or image) with a fresh install, and clone it to the new disk.

If your OS takes e.g. 8GB, then that's 8GB of space you don't need to blank out first. Not a lot compared to 1TB I admit, but it might save you some install time as well. The more you preinstall the better ;)

If the source image is too short to cover the whole disk (better too short than too long), you can use dd to blank out the rest of the disk. And then resize the partition after cloning, if you like.

If you want to use an image file rather than cloning a disk, you can use a useful feature called "sparse files" to cut down the file size wherever the blocks are all zeros in the image.

1

The disks (both, spinning and ssd) come from donated computers and will be used to install Linux-Desktops on them afterwards, handed out to people who can't afford to buy a computer, but need one.

Okay, so your long-term solution is this (it's a variation on DBAN)

  • prepare a Linux install self-bootable CD (using e.g. Anaconda)
  • write a script that will:
    • run memtest86 and other simple diagnostics
    • recognize the hardware (just the CPU, whether it's 32 or 64 bits)
    • zero the whole hard disk using dd
    • partition the hard disk into system, swap, and data
    • install the appropriate Linux distribution version onto system
    • eject the CD

Now say three PCs come in the lab, you turn them on, place a copy of the CD in the tray, boot from the CD and continue doing whatever you were doing before, until they warn you that the work is done. Years ago I did something like that: the script would play three long beeps immediately upon starting (so I knew that the speaker worked, they were desktops and I didn't want to connect a monitor unless I really had to), then other sounds depending on the exit status.

Basically, every computer took about one minute of work - forty seconds to boot it up, twenty seconds to shut it down when it started playing either "Reveille" or "Taps". I had no privacy issues there, so I never added the dd step; but it should be easy to implement. Even if my original proposal, as @PeterCordes noticed, was flawed.

  • Partition the HDD, install Linux, then zero the partition? – spuck Nov 22 at 23:13
  • @spuck yes -- the data partition, not the system partition, of course! – LSerni Nov 22 at 23:14
  • 1
    This fails to zero unused space on the system partition. – Peter Cordes Nov 23 at 15:51
  • @PeterCordes well spotted. My old script actually didn't need to securely erase anything, and the dd stage I added to the recipe was incorrect. I think I fixed that. – LSerni Nov 23 at 17:06
0

Set up a Raspberry Pi to automatically erase any USB storage plugged into it, and light up a LED when it's done. Then clone the SD card and buy a bunch of Raspberries to work in parallel. If you choose to use the ATA secure erase, with 4 devices you will be able to erase drives as fast as you can plug them. If you choose to overwrite the entire device, it will still be slow, but still 4 times as fast compared to erasing one drive at a time.

The same can be done with regular laptops if you have them in your donation box as well, saving you the cost of hardware.

-1

From the answers that have come in so far, it seems there is no cutting corners.

Here's an answer that cuts corners.

Does the incoming computer have an SSD? Find software that supports "ATA Secure Erase" and run it on the drive. What this does is tell the SSD to wipe it's internal encryption key, making the contents completely unreadable. Takes 30 seconds and is good enough for anything less than nuclear secrets.

Does the incoming computer have a spinning drive? Remove it, drill a couple of holes through the case and platter and send it to the recycle center. Get an SSD + mounting bracket from Amazon for about $50. This is far less than your time to process the old drives and sill provide much better performance. "Donated computers" implies an education environment, 200GB is plenty big enough.

And if you haven't got one already, get a 2-port hard drive dock that supports cloning. Plug in the master disk and a new blank one, press the copy button. It does a block copy so is completely OS independent. Also helps bulk-erasing the incoming SSDs.

  • Destroying working equipment is massive overkill and a big waste of money when you have no budget (I'm assuming volunteers) and are dealing with donated computers. If ATA Secure Erase is sufficient for SSDs, overwriting with zeros (or zero + random + zero or something) is also fine. – Peter Cordes Nov 23 at 15:50
  • I don't like the part of the answer that calls for using SSD secure erase features unlikely to be preseent on a donated computer. However I agree with the "replace with SSD" simply becuase it involves an upgrade to SSD. Quite agree, 128GB is plenty for a free PC. – Harper - Reinstate Monica Nov 23 at 19:47
-4

You may want to parallelize it. For instance in a 8 core processor you may use the following command line:

for i in {1..8}; do dd bs=16M if=/dev/zero of=$i & done

Note: Check your HDD specs sheet and adjust buffer accordingly.

In my case my HDD has a 128MB buffer size, hence a 128MB/8 core=16MB buffer.

  • 11
    In this case parallelization is much more likely to slow things down due to conflicting head moves... //-ization gives you more CPU, not more I/O bandwidth, and such operations are not CPU-bound... – xenoid Nov 20 at 15:25
  • Perhaps having several drives being erased in parallel is an alternative. Avoids the heads problem. Anyway, clever calculation could have all heads writing in parallel, by using carefully selected offsets... – ChuckCottrill Nov 21 at 1:44
  • @ChuckCottrill: you assume that the exact physical layout of "sectors" on the disk is known. With modern drives, only the manufacturer knows that. – virolino Nov 22 at 8:38
  • 1
    Can’t understand the down votes. Did any of you at least tested it? I own a Seagate Barracuda ST5000DM000 5TB 3.5, SATA, 5900RPM, 128 Mb Buffer and I can attest that when compared to a single process with the same buffer size you’ll get at least a two fold speed increase. Just test it and see by yourself. Please just don't assume everyone's still stuck in the 90's. – mjoao Nov 22 at 11:51
  • 2
    You're testing this on files inside the filesystem. IDK why that would be slower, but I still doubt this will help on a /dev/sdx block device. It might be that your filesystem is interleaving blocks of those 8 files or at least putting them near each other, so seek distances are small. But if you want to do 8x dd using different seek= offsets on /dev/sdx then the seek distances won't be small. It's also unlikely that # of CPU cores would make a difference on a magnetic HD. We're talking about 200MB/s, vs. memory bandwidth of ~20 to 30GB/s. – Peter Cordes Nov 23 at 15:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.