0

I have 28,000 files of size 6.7 GB in the directory aaa/input.

I need create TAR files in the directory aaa/output, such that each TAR file cannot exceed 1.9 GB, so I have to tar the 28,000 files into 4 TAR files (6.7/1.9) as

Log_(current date)_01.tar
Log_(current date)_02.tar
...
etc.

such that each tarball is less than 1.9 GB.

Is there a way to do this using a Unix script?

6
  • I won't probably perfectly answer your question, but a GUI archiver will do that easily: PeaZip (available on Flatpak/FlatHub). – X.LINK Feb 17 at 12:53
  • You could tar everything and then split the tar file into pieces. You would then have to joint it again before extracting, however. Would this be an acceptable solution? – terdon Feb 17 at 12:58
  • I would probably make 4 subdirectories, hard-link (ln) 7,000 files into each of those, and tar each subdir. The link commands can be made by a find on the main directory, split the names list 4 ways, and then pass each list into an xargs ln -t command. – Paul_Pedant Feb 17 at 13:24
  • As discussed lower, we would like to ask for clarification - which OS and tar version you are using. (OS can be deducted from e. g. uname -a output, tar usually shows version with tar --version, beware that tar is not standardized by POSIX, therefore it may vary - that's the reason for this clarification request) – d.c. Feb 17 at 18:20
  • 6
    Does this answer your question? How to create split tar archive in multiple stages to save space? – Greenonline Feb 17 at 21:48
1

You can use tar in combination with split to achieve this:

tar cf - aaa/input | split --bytes=1900m --filter='gzip > $FILE.gz' --numeric-suffixes - ./Log-

This command creates a tar archive to stdout and pipe the output to the split command which will split on every 1900MB, filters the data through gzip, uses stdin as input (the -) and places the output in the current directory with prefix Log-

The output will look like:

./Log-00.gz
./Log-01.gz
..

The files will be less in size because split will split in 1900M of input which will be less after compressing.

Alternatively you might use

printf 'n Log-%02d.tar\n' {2..4}|tar cf Log-01.tar -ML1900m aaa/input

To create the archives.

the first n in the printf command will tell tar to use a new name for the next archive. That's also the reason to start the range at 2.

Normally when tar prompts for a new archive the following is displayed:

Prepare volume #2 for '<my tar file>' and hit return:

When you press ? a brief menu is displayed:

 n name        Give a new file name for the next (and subsequent) volume(s)
 q             Abort tar
 y or newline  Continue operation
 !             Spawn a subshell
 ?             Print this list
6
  • 1. Long options of split are not portable. 2. The parts are useless without other parts --> you could simply split final tar.gz and have the command more effective. 3. There might be an extension to tar for multi-volume archives on the platform or might not, we don't know the platform... – d.c. Feb 17 at 13:54
  • @d.c., 1. OP does not ask for portable options explicitly. 2. Simply split the final file requires to have twice the amount of space, my option does not require this and might be less effective in terms of performance but is more effective if you don't have much space. 3. Yes, tar can use multiple volumes using printf 'n Log-%02d.tar\n' {2..4}|tar cf Log-01.tar -ML1900m aaa/input – Lambert Feb 17 at 14:03
  • 1. On "Unix" server should be portable solution the default one. We shall be glad to have a standard. I dealt with the man yesterday on serverfault and he didn't have even some of common extensions to POSIX, but you couldn't kno... 2. No space difference as both shell be done in pipe. 3. Which tar? I am working daily with at least 3 different implementations (BTW tar itself is out of portable specification - POSIX). – d.c. Feb 17 at 14:11
  • @d.c., ;-) Indeed I don't know the OPs platform. The term 'unix script' often leaves to the interpretation. I agree with you on the second part. For number 3, I use GNU tar on Solaris 11 and on Linux Mint – Lambert Feb 17 at 15:27
  • Hi ,im getting errors printf: {2..4} expected numeric value, tar: -ML1900m: No such file or directory, tries changing small m to capital M , still same error,what to do? – Subramanian Feb 18 at 16:02
1

If you can run GNU tar, the options --tape-length=N and --new-volume-script could be used.

Your command to create the volumes would be something like this:

tar cvf volume /path/to/files --tape-length=1945M --new-volume-script=/path/to/script.sh

And to extract everything from them:

tar xvf volume --new-volume-script=/path/to/script.sh

For the new volume script you can use the following one (from tar documentation at gnu.org). It works for volume creation, listing, comparing and extracting. Or you can modify it to change the name of the volumes, or write your own from scratch. Check the man page for the environment values available to it.

#! /bin/bash
# For this script it's advisable to use a shell, such as Bash,
# that supports a TAR_FD value greater than 9.

echo Preparing volume $TAR_VOLUME of $TAR_ARCHIVE.

name=`expr $TAR_ARCHIVE : '\(.*\)-.*'`
case $TAR_SUBCOMMAND in
-c)       ;;
-d|-x|-t) test -r ${name:-$TAR_ARCHIVE}-$TAR_VOLUME || exit 1
          ;;
*)        exit 1
esac

echo ${name:-$TAR_ARCHIVE}-$TAR_VOLUME >&$TAR_FD

Not the answer you're looking for? Browse other questions tagged or ask your own question.