Tag Info

Hot answers tagged

27

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


12

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


9

You're up against the "block size" of the compressor. Most compression programs break the input into blocks and compress each block. It appears the bzip block size only goes up to 900K, so it won't see any pattern that takes longer than 900K bytes to repeat. http://www.bzip.org/1.0.3/html/memory-management.html gzip appears to use 32K blocks. With xz ...


7

Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party. The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. ...


4

As long as stdout and stderr are line-buffered and the lines of one of the two are always spottable, mixing them is not a problem: consider the output of a program where both stdout and stderr are line-buffered, and where stderr is easily spottable: $ cat file xxxxxxxxxx tar: ---------- yyyyyyyyyy tar: ---------- zzzzzzzzzz tar: ---------- Extracting ...


4

Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer: At least as fast as other internet transfers: your upload speed will determine the transfer speed No data corruption Choose which files to transfer first No extra local/cloud storage space needed Free You didn't tell which OS you were using, but as you're ...


3

It's probably a GNU specific option, but you could use the -O or --to-stdout to extract files to standard output ~/test$ mkdir foo ~/test$ cd foo ~/test/foo$ cat >bar hello world ~/test/foo$ cd .. ~/test$ tar -acvf file.tgz foo foo/ foo/bar ~/test$ ls file.tgz foo ~/test$ rm -rf foo ~/test$ tar -axf file.tgz foo/bar -O hello world ~/test$ ls file.tgz


3

I suspect you're on AIX, where you're not allowed to mix old-style options without a minus (xvf) and new-style options with a minus (-C). Put a minus in front of all your options, and it should work. tar -xvf BACKUP.tar -C test If that still doesn't work, also put each option separately, as per the help message tar -x -v -f BACKUP.tar -C test See ...


3

You can suppress the warnings while preserving the timestamps: tar xzf myarchive.tar --warning=no-timestamp


2

This is simple as less a.tar:./x/y.txt This magic trick works if you have lesspipe installed and if the env variable LESSOPEN is defined to be | /usr/bin/lesspipe.sh %s which is expected if you have lesspipe installed correctly.


2

I right clicked the folder and clicked "create archive" and selected the .tar.gz option. The directory structure is deep, over 500,000 directories Yeah, good luck getting that to package up. And the GUI tool will try to do that on the same volume, which means a) you need another 1Tb of free space and b) the head thrashing of reading one file and ...


2

7z would be my choice. It allows auto-splitting of archives and supports multi-threaded compression. No, xz doesn't, despite what the help message says. Try with: 7za a -v100m -m0=lzma2 -mx=9 -ms=on -mmt=$THREADS archive.7z directory/ The output is split in 100MB blocks (change it with the -v switch). The only real downside is that 7z does not retain ...


2

This prints contents of ./x/y.txt from a.tar to STDOUT. tar xfO a.tar ./x/y.txt


2

As already indicated: Using random files is not good since they already contain maximum "information entropy", therefore won't compress; You need to pack a lot of files for a fair comparison. A better test case might be this: cd /var/tmp tar -zcf test1.tar /usr tar -cf test2.tar /usr gzip test2.tar ls -h (Note: Hoping there are no mounts under ...


2

The random file content you chose is not a good example - the compressed tarfiles will be bigger than the originals. You'll see the same with files in already compressing formats (many image/audio/video formats, for example). But tar-ing together multiple files with compressible content would typically produce smaller total tarfile size than when tar-ing ...


1

The -b option in tar was used to control the block size tar writes to a device, so that is exactly what you want. But -b 512 regarding the manual page tar(1) means a block size of 512*512 = 262144. All block sizes are valid that your device, you write the tar output to, can handle. In history this was needed for different tape drives the tar command was ...


1

Generally, no. It would be possible to design a filesystem that provides this kind of deduplication, but it would be very costly, for very little practical benefit, so I doubt that it's been done. The issue is that deduplication only looks at aligned extents. Deduplicating filesystems generally work at a block level. When the filesystem driver is about to ...


1

Oh, but this is a question about the contents of a file within a tar file. And actually, in some cases this isn't so hard. The thing is, a tar file is just a blocked out stream file - each file within the archive is found after the one before it, and each file gets a metadata header based on a specified format. Based on that format, I once wrote shitar - ...



Only top voted, non community-wiki answers of a minimum length are eligible