Tar big files in different directories

Question

I have ~20 files, each about 10 GB in size, in different folders that I need to compress into a single tar. It is not practical to copy them all to one location. More precisely, the files are organized something like this:

index1_index2/subfolder/bigfile.dat

Here index1 and index2 each vary over a few values.

What is a practical way to get a single tarball containing all these files, say with bash or something?

Answer 1

If they are on the same filesystem, you could hard-link those to a common directory and tar that directory.

Alternatively, if you are using GNU tar, you could (with a little more flexibility) soft-link those to a common directory and using its -h option, tar the files which the soft-links point to.

The manual page for the latter shows:

-h, --dereference

follow symlinks; archive and dump the files they point to

FreeBSD tar supports equivalent options, but names them like pax (see below).

There is of course no POSIX tar to use for comparison. If you can use pax, it has a similar -L option:

-L
If a symbolic link referencing a file of type directory is specified on the command line or encountered during the traversal of a file hierarchy, pax shall archive the file hierarchy rooted in the file referenced by the link, using the name of the link as the root of the file hierarchy. Otherwise, if a symbolic link referencing a file of any other file type which pax can normally archive is specified on the command line or encountered during the traversal of a file hierarchy, pax shall archive the file referenced by the link, using the name of the link. The default behavior, when neither -H or -L are specified, shall be to archive the symbolic link itself.

Whether hard-linking, or soft-linking, the result is that you do not have to move your existing files. Hard-linking changes the ctime (timestamp) of your files, while soft-linking does not. But soft-linking (though it reduces the necessity of being on the same filesystem) is not universally supported by tar implementations using identical options.

OP's original statement appeared to indicate that the reason why it was inconvenient to put all files together in one directory was because of some limitation by the application which generates them. Clarification makes it apparent that the problem is that they all have the same filename. While linking into one directory (and for instance encoding their original directory name into the common location) is certainly doable, there are other options for simply collecting the files into a single archive, retaining their existing directory names. One drawback to doing this is that it limits the ability to restore into arbitrary locations. However: the simplest way of passing some ~20 names to tar would be on the command-line, e.g.,

`tar czf myoutput.tar.gz $(find . -type f -name bigfile.dat)`

For only 20 files that is not likely to be a problem with command-line length. If it were a large number of files (or very long pathnames), that would make it harder, since tar implementations as a rule lack the ability to pass a list of pathnames other than as separate command-line parameters (see Solaris for instance). For those, one might try to work around the limitation by creating an archive in steps — but then that would not work with compression. Some implementations provide options for reading a list of filenames from a file. The GNU tar -T (also FreeBSD) option does this. Other programs may provide a third alternative, reading the list of filenames from the standard input (as done by pax, which was influenced by cpio), but generally tar does not: it may read the file content from standard input.

Answer 2

The most straightforward way is:

tar cfz bigtar.tar.gz /path/to/file1 /path/to/file2 ... /path/to/file20

If there are matchable similarities between the target files, you could use find to create the files list and GNU tar's -T (--files-from).

e.g. if they're all .mp4 files:

find /path -type -f -iname '*.mp4' | tar cf bigtar.tar -T -

Otherwise, if there's no easy pattern for selecting your filenames, you could use find to create an initial list in a tmpfile, edit that manually to remove filenames you don't want to include in the tar, and then use the edited tmpfile's name with tar's -T option.