0

I have ~20 files, each about 10 GB in size, in different folders that I need to compress into a single tar. It is not practical to copy them all to one location. More precisely, the files are organized something like this:

index1_index2/subfolder/bigfile.dat

Here index1 and index2 each vary over a few values.

What is a practical way to get a single tarball containing all these files, say with bash or something?

5
  • Do you want the archive members to be index1_index2/subfolder/bigfile.dat, or subfolder/bigfile.dat, or bigfile.dat? Commented Nov 15, 2015 at 21:08
  • Is the filename always bigfile.dat? Are there other files that are bigger? I would try it with find options | xargs tar czf bigtar. The options depend on your answer.
    – ott--
    Commented Nov 15, 2015 at 21:37
  • Yes it is always the same filename. I didn't realize you could pipe things into tar, that makes things easier Commented Nov 16, 2015 at 5:52
  • Piping as suggested by @ott would result in multiple invocations of tar, which would create one file containing only as many files as the last time tar was run. Perhaps not what was anticipated. Commented Nov 16, 2015 at 9:53
  • @ThomasDickey Piping to xargs results in exact one invocation of tar. tar c creates a new archive or overwrites an existing one, no matter what happended on an earlier run. Have you overlooked that xargs?
    – ott--
    Commented Nov 16, 2015 at 18:44

2 Answers 2

4

If they are on the same filesystem, you could hard-link those to a common directory and tar that directory.

Alternatively, if you are using GNU tar, you could (with a little more flexibility) soft-link those to a common directory and using its -h option, tar the files which the soft-links point to.

The manual page for the latter shows:

-h, --dereference

follow symlinks; archive and dump the files they point to

FreeBSD tar supports equivalent options, but names them like pax (see below).

There is of course no POSIX tar to use for comparison. If you can use pax, it has a similar -L option:

-L
If a symbolic link referencing a file of type directory is specified on the command line or encountered during the traversal of a file hierarchy, pax shall archive the file hierarchy rooted in the file referenced by the link, using the name of the link as the root of the file hierarchy. Otherwise, if a symbolic link referencing a file of any other file type which pax can normally archive is specified on the command line or encountered during the traversal of a file hierarchy, pax shall archive the file referenced by the link, using the name of the link. The default behavior, when neither -H or -L are specified, shall be to archive the symbolic link itself.

Whether hard-linking, or soft-linking, the result is that you do not have to move your existing files. Hard-linking changes the ctime (timestamp) of your files, while soft-linking does not. But soft-linking (though it reduces the necessity of being on the same filesystem) is not universally supported by tar implementations using identical options.

OP's original statement appeared to indicate that the reason why it was inconvenient to put all files together in one directory was because of some limitation by the application which generates them. Clarification makes it apparent that the problem is that they all have the same filename. While linking into one directory (and for instance encoding their original directory name into the common location) is certainly doable, there are other options for simply collecting the files into a single archive, retaining their existing directory names. One drawback to doing this is that it limits the ability to restore into arbitrary locations. However: the simplest way of passing some ~20 names to tar would be on the command-line, e.g.,

`tar czf myoutput.tar.gz $(find . -type f -name bigfile.dat)`

For only 20 files that is not likely to be a problem with command-line length. If it were a large number of files (or very long pathnames), that would make it harder, since tar implementations as a rule lack the ability to pass a list of pathnames other than as separate command-line parameters (see Solaris for instance). For those, one might try to work around the limitation by creating an archive in steps — but then that would not work with compression. Some implementations provide options for reading a list of filenames from a file. The GNU tar -T (also FreeBSD) option does this. Other programs may provide a third alternative, reading the list of filenames from the standard input (as done by pax, which was influenced by cpio), but generally tar does not: it may read the file content from standard input.

2
  • pax is tar and cpio (at least - most paxs will also do zip and so on, but tar is the default archive format).
    – mikeserv
    Commented Nov 16, 2015 at 6:05
  • I'm aware of that, but depending on the system, OP may not even have pax installed. Commented Nov 16, 2015 at 8:56
0

The most straightforward way is:

tar cfz bigtar.tar.gz /path/to/file1 /path/to/file2 ... /path/to/file20

If there are matchable similarities between the target files, you could use find to create the files list and GNU tar's -T (--files-from).

e.g. if they're all .mp4 files:

find /path -type -f -iname '*.mp4' | tar cf bigtar.tar -T -

Otherwise, if there's no easy pattern for selecting your filenames, you could use find to create an initial list in a tmpfile, edit that manually to remove filenames you don't want to include in the tar, and then use the edited tmpfile's name with tar's -T option.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .