We are building a data archiving system where we want to have the option to store redundant copies of data on different storage types: tape/disk/cloud, and also with the option of using different archive formats: tar, zip etc
Many of the files are fairly large (50GB+), and once data is archived it is never modified. We have found that tape is ideal for this use case. We store the block offsets for each archived file in our database, and once a tape is almost full we "finalize" it by writing an index of all the file block offsets to the end (so it is self describing).
I would like to know if it would be possible to do the same thing using a hard disk by using it unformatted (i.e. without any file system), and reading/writing to it as a block device. We would append archives to the disk starting from the first block and write one archive immediately after the other until the disk is almost full. In this way we would avoid any file fragmentation issues, make full use of the disk capacity, increase read/write speeds, and it would be much easier to do data recovery if ever needed.
Tapes allow us to easily seek to the end of the written data, whereas with disk I guess we would have to record the number of the last block that was used, and ensure we started writing the next archive starting from the next block. I would like to know how we could calculate that in a rigorous way, where we could be sure we would not be overwriting any previously written data.
Using dd, I think this would be fairly straightforward using the seek option. However we would like to use the archiving tools (tar, zip etc) to write the data directly to disk (like we do with tape), to avoid the time taken to copy the files into an intermediate archive file (on our staging disk), and then using dd to write this file to the archive disk. However tar, zip don't have any option to seek like dd does. I suppose they would just open the block device and start writing from the beginning.
Would like to know if anybody has done anything like this before, or has any other inputs/thoughts regarding this idea - especially potential pitfalls to be aware of. Also, if I should fill the drive with zeros before writing anything to it?