Is it possible to simultaneously write a tar archive to two tape drives using a single command, by directing the same output to both drives?

I want to save time by performing the operation in parallel. Also I want to ensure that both archives are binary identical so I only need to store one checksum. (I don't know about tar, but some archive formats may store a timestamp in the header that is generated when it is created, thus causing the checksums to be different, when in fact the files within both archives are identical)

  • 2
    if your tape drives accept a piped data stream or dd with a fixed blocksize, you can use tee and FIFOs/process substitution to send one input stream (tar) to multiple output (dd, sha*sum, ...). However the slowest drive will slow everything else down (adding some buffering might help avoid frequent start-stops). – frostschutz May 30 '20 at 8:36
  • @frostschutz. Thanks for the info. If you can post it as an answer I will accept it. Would also be super helpful if you could give me a sample command, including buffering that writes a tar to multiple drives and also outputs a checksum. – swami May 31 '20 at 3:50

Unfortunately I don't have any tape drives to play with, so this answer is theoretical in nature.

If your tape drive accepts a piped data stream or dd with a fixed blocksize, you can do it with tee and also have additional data streams for the checksum calculation.

With FIFOs:

# make fifos
mkfifo tape.0.fifo tape.1.fifo tape.sha384sum.fifo tape.sha512sum.fifo

# start fifo readers
dd iflag=fullblock bs=256K if=tape.0.fifo of=/dev/tape0 &
dd iflag=fullblock bs=256K if=tape.1.fifo of=/dev/tape1 &
sha384sum < tape.sha384sum.fifo > tape.sha384sum &
sha512sum < tape.sha512sum.fifo > tape.sha512sum &

# start fifo writer
tar -cz some/stuff/ | tee > *.fifo

With bash process substitution:

tar -cz some/stuff/ | \
    tee > >(dd iflag=fullblock bs=256K of=/dev/tape0) \
          >(dd iflag=fullblock bs=256K of=/dev/tape1) \
          >(sha384sum > tape.sha384sum) \
          >(sha512sum > tape.sha512sum)

Note that the extra > is there because tee writes to both stdout and files; the > causes one of the arguments to be the stdout redirection, while the others are file parameters for tee to write to directly. The alternative would discard stdout as > /dev/null.

Additional buffering could be implemented with any pipe buffering program of your choice, or with dd if you don't have any available. For example:

dd iflag=fullblock bs=128M | dd iflag=fullblock bs=128M obs=256K of=/dev/tape0

The dd | dd construct allows the first dd to receive a chunk of data directly, passing it on to the second dd. That leaves the first dd free to buffer the next block of data, while the second dd is busy writing it.

This works around the single-process nature of dd (doesn't read the next block until it's done writing the previous block), a hoop you might not have to jump through with a dedicated buffering program.

Effectively this is a 256M buffer with a 128M min-fill characteristic.

Whether such buffering has any positive effect depends on the speed and behavior of the tape drives. It's also possible to have no effect (drive fast enough to not make a difference), or a negative effect (drive stopping while waiting for buffer to fill up or conflicting with the drive's own buffering mechanisms).

  • 1
    brilliant answer – swami May 31 '20 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.