3

I'm working on big data and I need to archive a directory that is larger than 64 terabytes. I cannot create such large file (archive) on my file system. Unluckily, all proposed solutions for creating a multiple-parts archive on Linux suggest creating an archive first and then splitting it into smaller files with split command.

I know that it is possible with f.e. 7zip, but unluckily I'm quite forced to use tools built in RedHat 6 - tar, gzip, bzip2...

I was wondering about creating a script that would ask user for the maximum volume size. It would archive every single file with gzip, split those files, that are too big and then manually merge them into many tars with the chosen size. Is that a good idea?

Is there any other possibility to achieve big archive division with basic Linux commands?

UPDATE:

I've tested the solution on the filesystem with the restricted maximum file size and it worked. The pipe that redirects the tar output directly into split command has worked as intended:

tar -czf - HugeDirectory | split --bytes=100GB - MyArchive.tgz.

The created files are already small and when merging them together no supersized files are created:

cat MyArchive.tgz* | tar -xzf -

  • I am confused: are you trying to compress a single 64+ TB file into .tar.gz? Why do you feel you need .tar in the picture then? .gz should be perfectly fine, and then you can man split if you need multiple files. – ajeh Jun 28 '18 at 17:13
  • You have a single directory with 64 TB of files in it? Where is that stored and where do you plan to store the archive? tar specifically deals (it was written for it) with tapes of limited size and expanding a given archive over multiple tapes. – Patrick Mevzek Jun 28 '18 at 17:13
  • Yes, files in directory are over 64 TB. The problem is that I cannot use gzip command on the whole folder, because the output archive is larger then the filesystem allows. Let's say that I have 100 TB of data, and compressed archive would be 70 TB, which is bigger than allowed.Therefore I cannot split, because I'll receive an error when compressing. – JoshThunar Jun 29 '18 at 7:57
2

If you have enough space to store the compressed archive, then the archive could be created and split in one go (assuming GNU split):

tar -c -vz -f - directory | split --additional-suffix=.gz.part -b 1G

This would create files called xaa.gz.part, xab.gz.part etc., each file being a 1G compressed bit of the tar archive.

To extract the archive:

cat x*.gz.part | tar -x -vz -f -

If the filesystem can not store the compressed archive, the archive parts needs to be written to another filesystem, alternative to some remote location.

On that remote location, for example:

ssh user@serverwithfiles tar -c -vz -f - directory | split --additional-suffix=.gz.part -b 1G

This would transfer the compressed archive over ssh from the machine with the big directory to the local machine and split it.

  • Yes, this is the most popular solution on the Internet. Unfortunately the compressed file is too big for the filesystem. – JoshThunar Jun 29 '18 at 7:51
  • @JoshThunar Your solution would then have to involve writing the parts to another filesystem, or to some remote location. The alternative, to delete the original while creating the archive, would be unsafe. – Kusalananda Jun 29 '18 at 7:59
  • So why not using the method I mentioned in my answer? It permits you to capture the parts inside single small files that you may compress while star is either writing the next one or while staris waiting for the media change confirmation from the user- – schily Jun 29 '18 at 8:43
  • @schily The reason I don't use star is twofold: 1) You already mentioned in your answer. 2) I'm unfamiliar with it. – Kusalananda Jun 29 '18 at 8:47
  • star is the solution of your choice if you like to have a reliable feature enhanced tar implementation. It has less deviations fro tar than gtar and it is easier to learn. I've got feedback from various sysadmins. They told me that they needed a day to understand how to use it but then never would use something else. – schily Jun 29 '18 at 9:06
0

With basic UNIX command, I see no way except than splitting an existing file.

But how about installing star and using the multi-volume feature from star?

This has the advantage that there is a check whether the parts are supplied in the right order.

  • Thank you for your answer. However I'm providing the solution for my client's machine so this is why installing anything is very constrained. – JoshThunar Jun 29 '18 at 8:49
  • RedHat provides star packages since approx. 20 years and the documentation is better than the documentation for gtar: schilytools.sourceforge.net/man/man1/star.1.html BTW: Do never try to use the multi volume features from gtar, since they create files that cannot be read back by gtarwith a proability of approx. 5%. – schily Jun 29 '18 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.