I'm working on big data and I need to archive a directory that is larger than 64 terabytes. I cannot create such large file (archive) on my file system. Unluckily, all proposed solutions for creating a multiple-parts archive on Linux suggest creating an archive first and then splitting it into smaller files with
I know that it is possible with f.e. 7zip, but unluckily I'm quite forced to use tools built in RedHat 6 - tar, gzip, bzip2...
I was wondering about creating a script that would ask user for the maximum volume size. It would archive every single file with gzip, split those files, that are too big and then manually merge them into many tars with the chosen size. Is that a good idea?
Is there any other possibility to achieve big archive division with basic Linux commands?
I've tested the solution on the filesystem with the restricted maximum file size and it worked. The pipe that redirects the
tar output directly into
split command has worked as intended:
tar -czf - HugeDirectory | split --bytes=100GB - MyArchive.tgz.
The created files are already small and when merging them together no supersized files are created:
cat MyArchive.tgz* | tar -xzf -