I have 5 million files which take up about 1TB of storage space. I need to transfer these files to a third party.
What's the best way to do this? I have tried reducing the size using .tar.gz, but even though my computer has 8GB RAM, I get an "out of system memory" error.
Is the best solution to snail-mail the files over?
Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file.
GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI software is trying to collect a list of all the filenames in memory. It's unnecessary to do that in order to create an archive. The dedicated tools tar and gzip are defintely designed to work with streaming input and output which means that they can deal with input and output a lot bigger than memory.
If you avoid the GUI program, you can most likely generate this archive using a completely normal everyday tar invocation like this:
tar czf foo.tar.gz foo
where foo is the directory that contains all your 5 million files.
The other answers to this question give you a couple of additional alternative tar commands to try in case you want to split the result into multiple pieces, etc...
"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync:
rsync -alPEmivvz /source/dir remote.host.tld:/base/dir
If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try:
7z a archive.7z /source/dir
Or if you don't feel comfortable with 7z at least try making a .tar.xz archive:
tar cJv archive.tar.xz /source/dir
(it should be noted, that older versions of tar don't create .tar.xz archives, but .tar.lzma archives, when using the J switch. Even yet older versions of tar, don't support the J flag altogether.)
Since you're using a GUI program to create those files, I'm assuming you're feeling a bit uncomfortable using a command line interface.
To facilitate creation, management and extraction of archives from the command line interface, there's the small utility called atool. It is available for practically every common distro I've seen, and works pretty much every single archive I've stumbled upon, unless the hopelessly obscure ones.
Check whether your distro has atool in their repos, or ask your admin to install it, when it's in a workplace environment.
atool installs a bunch of symlinks to itself, so packing and unpacking becomes a breeze:
apack archive.tar.xz <files and/or directories>
Creates an archive.
aunpack archive.7z
Expands the archive.
als archive.rar
Lists file contents.
What kind of archive is created, atool discerns that by the filename extension of your archive in the command line.
rsync is that if (when) the connection breaks, rsync can pick up where it left off.
Commented
Jun 12, 2015 at 15:13
Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party.
The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. If your third party only has DVD you need 125:1 (roughly).
If you cannot match those compression numbers just use a normal disc, copy and snail mail that to the third party. In that case shipping something smaller than a 1Tb drive that would need compression is madness.
You just have to compare that to using ssh -C (standard compression) or preferably rsync with compression to copy the files over the network, no need to compress and tar up front. 1Tb is not impossible to move over the net, but it is going to take a while.
Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer:
You didn't tell which OS you were using, but as you're speaking about tar.gz compression, I'll assume you're using some GNU/Linux-like OS. For that I'll suggest Transmission. It's an open-source torrent software that runs on Mac and Linux. I like it because the developers put an effort into making it native to every GUI clients they support: no cross-platform language.
You could combine this method with compression, however you'll lose the ability to prioritize parts of the transfer.
7z would be my choice. It allows auto-splitting of archives and supports multi-threaded compression. No, xz doesn't, despite what the help message says. Try with:
7za a -v100m -m0=lzma2 -mx=9 -ms=on -mmt=$THREADS archive.7z directory/
The output is split in 100MB blocks (change it with the -v switch).
The only real downside is that 7z does not retain unix metadata (e.g. permissions and owner). If you need that, pipe tar output into 7za instead (see man 7za for some examples).
split on a .tar.gz file, and get to keep the metadata.
tar approach.
Commented
Jun 12, 2015 at 17:00
I right clicked the folder and clicked "create archive" and selected the .tar.gz option. The directory structure is deep, over 500,000 directories
Yeah, good luck getting that to package up. And the GUI tool will try to do that on the same volume, which means a) you need another 1Tb of free space and b) the head thrashing of reading one file and appending it to the archive will not be too healthy for the drive.
I see two possibilities:
1) rsync from your computer to the 3rd party. One or the other will need a global IP address that doesn't change often. The huge advantage of rsync is if you get interrupted for any reason, it can pick up where it left off. Copying a 1Tb tarball is either going to succeed or fail
2) purchase a 1Tb external drive, copy things to it, and have the drive delivered. High latency, but really good bandwidth when it arrives. Various online backup services to this for the initial backup.
I suggest you to use something like :
tar -c -L 104857600 -f file1.tar -f file2.tar -f file3.tar -f file4.tar -f file5.tar -f file6.tar -f file7.tar -f file8.tar -f file9.tar -f file10.tar
gzip file*.tar
which will create 10 files which will be at most 100GB big. But that wouldn't answer your "OOM" issue since tar is supposed to stream it's read/write so the bottleneck is obivously gzip.
How about that:
find /path | tar -T - -czf - | ssh remotehost "cd /target/dir/; tar xzf -"
find searches lists the directory treetar -T - reads the list from stdin-czf - creates an archive and writes to stdout, z gzips the streamssh remotehost login to remotehost using sshcd /target/dir/ change to the target directorytar xzf - extract the incoming stream from stdinWith fexsend you can send files of any number and size to any recipient on-the-fly, see:
http://fex.rus.uni-stuttgart.de:8080/usecases/downunder.html
Of course, you need a F*EX server, but it is free:
.tar.gzor COPYING the resulting compressed file? Either way, something is weird, because neither operation should consume more memory just because the files are big. That is, both operations should be streaming. Please include more information about exactly what commands are failing.tarshould just archive files incrementally as it lists them, never building up a list in memory. But again, please show the exact command you are using. Also, are all the files in the same directory or is the directory structure very deep?