6

I bought cheep 2 TB HDDs (60 € each) and want to check whether they return the data they were fed when reading before using them. I checked some cheep thumb drives drives by copying large files I had lying around to them and checking the hashes of the data they gave back (and found ones which just throw data away after their actual storage capacity is exhausted). Unfortunately, I don't have any 2 TB files lying around.

I now want to generate 2 TB of pseudorandom data, write it to the disks, and take a hash of the disks. I then want to write the same data directly to the hash function and get the hash it should produce this way. The pseudorandom function doesn't have to be cryptographically secure in any way, it just needs to produce data with high entropy fast.

If I write a script which just hashes a variable containing a number, prints the hash to stdout, increments the variable, and repeats, the data rate is way too slow, even on when using a fast CPU. Like 5 orders of magnitude too slow (not even 60 kByte/s).

Now, I could attempt to do this with tee but that seems like a really bad idea and I can't just reproduce the same data over and over again.

Ideally, I'd pass some short argument (a number, a string, I don't care) to the program and get an arbitrarily large amount of data out at its stdout, and that data is the same on each call.

  • 1
    Why do you need to do it with random data? Why not output increasing numbers continuously? If you absolutely need it to be random, most computer-based random number generators use a seed which makes it reproducible. Since performance is an issue, you should do it in C, not in a shell script. – Julie Pelletier Mar 18 '17 at 23:20
  • It doesn't have to be random for this but it'd be nice and it'd be nice to have this. Are there any pseudorandom generators which take their seeds as an argument? Writing a C program for this would not be hard (I could just use srand() and rand().) but if there is a standard tool to get this done, knowing about it would be nice. – UTF-8 Mar 18 '17 at 23:24
  • 1
    If you know how to program it in 5 minutes, go ahead and do it. It's rather unlikely that someone would choose that over basic utilities like badblocks which is also included in fsck. Your concept will not result in a better verification. If no bad blocks are found, then your drive looks good. If you find some then it's starting to get less reliable. Of course there are other factors that can cause a drive to fail but these won't be noticeable in either tests until it fails. – Julie Pelletier Mar 19 '17 at 2:37
6

Well, most people just go with badblocks...

Otherwise, just encrypt zeroes. Encryption does exactly what you want. Encrypted zeroes look like random data. Decrypting random data turns it back into zeroes. It's deterministic, reversible so as long as you know the key.

cryptsetup open --type plain --cipher aes-xts-plain64 /dev/yourdisk cryptodisk
shred -n 0 -z -v /dev/mapper/cryptodisk # overwrites everything
cmp /dev/zero /dev/mapper/cryptodisk    # byte-by-byte comparison

This should utilize full disk speed on a modern system with AES-NI.


Also kind of works for just piping (without backed by real storage)

cd /dev/shm # tmpfs
truncate -s 1E exabyte_of_zero
losetup --find --show --read-only exabyte_of_zero
cryptsetup open --type plain --cipher aes-xts-plain64 --readonly /dev/loop4
cat /dev/mapper/loopcrypt | something_that_wanted_random_data

or if we're still writing to a disk and comparing

cat /dev/mapper/loopcrypt > /dev/sdx
# overwrites until no space left on device
cmp /dev/mapper/loopcrypt /dev/sdx
# compares until EOF on /dev/sdx OR loopcrypt and sdx differ byte X.

Unlike PRNG this can also be used to start comparing data somewhere in the middle of the file. With a traditional PRNG you have to re-generate it all over again to reach back to whatever position you were interested in. Of course, you could just make a random seed based on offset or something...

  • I want this to be very flexible, that's why I'm asking about data streams. Cryptsetup doesn't allow me to pass its output to stdout. Furthermore, it has to be run as root to work. If hardware acceleration isn't available, encryption is probably way slower than generating random data by other means. – UTF-8 Mar 18 '17 at 23:38
  • @UTF-8 stdout/pipe kind of also works. In terms of speed: well, it was considerably faster than /dev/urandom last time I tried it on a NAS with old / slow / unaccelerated CPU. But urandom implementation changed a lot since then... you can go with PRNG but that is already what badblocks does with -t random. Not sure about your "has to be run as root to work", you have to be root to write to raw disk in first place. – frostschutz Mar 18 '17 at 23:54
  • Yes, I know that I have to be root to write to disks but I also have to be root when I don't want to write to disks. I had the problem of needing reproducible pseudorandom data in the past and always cheated my way around it but now I'm looking for an actual solution. Do you know about a PRNG tool for large data for the terminal? I only know of rand but it only produces relatively small number represented as decimal numbers encoded in UTF-8. Plus, it doesn't typically come with distros (which is another downside, it's not a requirement that it's a standard tool (combi) but it'd be nice). – UTF-8 Mar 19 '17 at 0:03
  • 1
    @UTF-8 A more convenient way to encrypt zeros is the following: openssl enc -e -aes-128-ctr -k "key" -in /dev/zero -out /dev/stdout (substitute /dev/stdout with anything you want) This generates at 1.1 GB/s, massively outstripping almost any drive's throughput. To reverse this, use -d rather than -e, change the -in argument and check that what you read back is just zeros. – Iwillnotexist Idonotexist Mar 19 '17 at 4:13
  • 1
    badblocks really is a much better choice for checking whether a storage medium is a fraud. I checked a fraudulent thumb drive with sudo badblocks -wsv /dev/sdb and it found bad blocks within a few minutes. – UTF-8 Mar 19 '17 at 14:19
0

If you are so afraid of data integrity, then just use ex.: ZFS, it has built-in integrity checking, since a new HW could be ok, but later years pass it could became bad.

https://changelog.complete.org/archives/9769-silent-data-corruption-is-real

Here’s something you never want to see:

ZFS has detected a checksum error:

   eid: 138
 class: checksum
  host: alexandria
  time: 2017-01-29 18:08:10-0600
 vtype: disk

This means there was a data error on the drive. But it’s worse than a typical data error — this is an error that was not detected by the hardware. Unlike most filesystems, ZFS and btrfs write a checksum with every block of data (both data and metadata) written to the drive, and the checksum is verified at read time. Most filesystems don’t do this, because theoretically the hardware should detect all errors. But in practice, it doesn’t always, which can lead to silent data corruption. That’s why I use ZFS wherever I possibly can.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.