Tag Info

New answers tagged

1

Of course you can. Firstly tar in most cases does not include compression, it hands that off to a helper (if it supports it at all). At first that helper was compress (.tar.Z). I have also seen gzip (.tar.gz or .tgz), bzip2 (.tar.bz2), infozip's zip and unzip tools (.tar.zip), and some I cant rember the official names for (.tar.lha, .tar.lhz, .tar.zoo, ...


0

Here is a overview of gzip http://www.gzip.org/algorithm.txt Short answer is that it will not improve significantly after the initial data needed for the hashes is taken into account.


3

Not much. The "distance" covered by the DEFLATE algorithm which gzip uses is limited to 32 KB. Wikipedia link -> DEFLATE It is worth benchmarking against the various gzip compression levels and also considering bzip2 and xz.


8

It does up to a certain point and this evens out. The compression algorithms have a restriction on the size of the blocks they look at (bzip2) and/or on the tables they keep with information on previous patterns (gzip). In the case of gzip, once a table is full old entries get pushed out, and compression no further improves. Depending on the your ...


2

Use zless utility from that same gzip suite, it combines functionality of gzip -d and less into one


2

This is what pipes were made for: gzip -dc | less


2

As already indicated: Using random files is not good since they already contain maximum "information entropy", therefore won't compress; You need to pack a lot of files for a fair comparison. A better test case might be this: cd /var/tmp tar -zcf test1.tar /usr tar -cf test2.tar /usr gzip test2.tar ls -h (Note: Hoping there are no mounts under ...


2

The random file content you chose is not a good example - the compressed tarfiles will be bigger than the originals. You'll see the same with files in already compressing formats (many image/audio/video formats, for example). But tar-ing together multiple files with compressible content would typically produce smaller total tarfile size than when tar-ing ...


9

You're up against the "block size" of the compressor. Most compression programs break the input into blocks and compress each block. It appears the bzip block size only goes up to 900K, so it won't see any pattern that takes longer than 900K bytes to repeat. http://www.bzip.org/1.0.3/html/memory-management.html gzip appears to use 32K blocks. With xz ...


5

$> ls -la *csv -rw-rw-r-- 1 user at 0 Jun 23 15:22 1.csv -rw-rw-r-- 1 user at 0 Jun 23 15:22 2.csv -rw-rw-r-- 1 user at 0 Jun 23 15:22 3.csv $> gzip *csv $> ls -la *csv* -rw-rw-r-- 1 user at 26 Jun 23 15:22 1.csv.gz -rw-rw-r-- 1 user at 26 Jun 23 15:22 2.csv.gz -rw-rw-r-- 1 user at 26 Jun 23 15:22 3.csv.gz $>


2

I right clicked the folder and clicked "create archive" and selected the .tar.gz option. The directory structure is deep, over 500,000 directories Yeah, good luck getting that to package up. And the GUI tool will try to do that on the same volume, which means a) you need another 1Tb of free space and b) the head thrashing of reading one file and ...


2

7z would be my choice. It allows auto-splitting of archives and supports multi-threaded compression. No, xz doesn't, despite what the help message says. Try with: 7za a -v100m -m0=lzma2 -mx=9 -ms=on -mmt=$THREADS archive.7z directory/ The output is split in 100MB blocks (change it with the -v switch). The only real downside is that 7z does not retain ...


4

Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer: At least as fast as other internet transfers: your upload speed will determine the transfer speed No data corruption Choose which files to transfer first No extra local/cloud storage space needed Free You didn't tell which OS you were using, but as you're ...


27

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


12

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


7

Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party. The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. ...


0

I suggest you to use something like : tar -c -L 104857600 -f file1.tar -f file2.tar -f file3.tar -f file4.tar -f file5.tar -f file6.tar -f file7.tar -f file8.tar -f file9.tar -f file10.tar gzip file*.tar which will create 10 files which will be at most 100GB big. But that wouldn't answer your "OOM" issue since tar is supposed to stream it's read/write so ...


0

How about that: find /path | tar -T - -czf - | ssh remotehost "cd /target/dir/; tar xzf -" find searches lists the directory tree tar -T - reads the list from stdin -czf - creates an archive and writes to stdout, z gzips the stream ssh remotehost login to remotehost using ssh cd /target/dir/ change to the target directory tar xzf - extract the incoming ...



Top 50 recent answers are included