I am running many tasks on a Linux cluster. Each task creates many output files. When all tasks are finished, I run something like
tar cf foo.tar output_files/to create a
tar archive. This is a very slow process since there are many thousands of files and directories.
Is there any way to do this in parallel as the output files are being created?
Is it possible to have multiple
tar processes, spread across multiple machines, all adding their files to the same archive at once?
The cluster has a shared filesystem.
I am not interested in compression since it slows things down even more, and because all of the input files are themselves already compressed. Ideally the output would be a
tar file, but I would consider other archive formats as well.