Tag Info

New answers tagged

0

Here’s a small snippet that converts a ZIP archive to a matching TAR.GZ archive OnTheFly. Convert ZIP archive to TAR archive on the fly # File: zip2tar.py # # Convert ZIP archive to TAR.GZ archive. # # Written by Fredrik Lundh, March 2005. # helpers (tweak as necessary) def getuser(): # return user name and user id return "anonymous", 1000 def ...


1

Of course you can. Firstly tar in most cases does not include compression, it hands that off to a helper (if it supports it at all). At first that helper was compress (.tar.Z). I have also seen gzip (.tar.gz or .tgz), bzip2 (.tar.bz2), infozip's zip and unzip tools (.tar.zip), and some I cant rember the official names for (.tar.lha, .tar.lhz, .tar.zoo, ...


0

I don't know how to do it conveniently with BSD or GNU tar. (Inconveniently: don't exclude the directory itself but exclude every file in the directory except the one you want to include.) You can do it relatively easily with pax, which is POSIX's replacement for cpio and tar. Pax should be available everywhere, though some Linux distributions don't include ...


0

Run tar twice? First exclude Texas and then just extract the file?


4

As long as stdout and stderr are line-buffered and the lines of one of the two are always spottable, mixing them is not a problem: consider the output of a program where both stdout and stderr are line-buffered, and where stderr is easily spottable: $ cat file xxxxxxxxxx tar: ---------- yyyyyyyyyy tar: ---------- zzzzzzzzzz tar: ---------- Extracting ...


0

For the benefit of those running into this post here is the solution that works - at least on Ubuntu 14.04. For the folders structure I have shown above issue tar -zcf pack.tgz --exclude-from=/path/to/excludes.list pack excludes.list contains, in my case rest/index.php rest/info.log rest/filetoexclude.ext ...essentially a list of files to exclude. ...


0

Just use "another/index.php" as the pattern (or even "rest/another/index.php"): $ mkdir -p pack/rest/vendor $ mkdir -p pack/rest/another $ touch pack/rest/{vendor,another}/index.php $ tar --create --verbose --file xx.tar --exclude "another/index.php" pack/ pack/ pack/rest/ pack/rest/another/ pack/rest/vendor/ pack/rest/vendor/index.php I found that for ...


2

As already indicated: Using random files is not good since they already contain maximum "information entropy", therefore won't compress; You need to pack a lot of files for a fair comparison. A better test case might be this: cd /var/tmp tar -zcf test1.tar /usr tar -cf test2.tar /usr gzip test2.tar ls -h (Note: Hoping there are no mounts under ...


2

The random file content you chose is not a good example - the compressed tarfiles will be bigger than the originals. You'll see the same with files in already compressing formats (many image/audio/video formats, for example). But tar-ing together multiple files with compressible content would typically produce smaller total tarfile size than when tar-ing ...


9

You're up against the "block size" of the compressor. Most compression programs break the input into blocks and compress each block. It appears the bzip block size only goes up to 900K, so it won't see any pattern that takes longer than 900K bytes to repeat. http://www.bzip.org/1.0.3/html/memory-management.html gzip appears to use 32K blocks. With xz ...


1

The -b option in tar was used to control the block size tar writes to a device, so that is exactly what you want. But -b 512 regarding the manual page tar(1) means a block size of 512*512 = 262144. All block sizes are valid that your device, you write the tar output to, can handle. In history this was needed for different tape drives the tar command was ...


2

I right clicked the folder and clicked "create archive" and selected the .tar.gz option. The directory structure is deep, over 500,000 directories Yeah, good luck getting that to package up. And the GUI tool will try to do that on the same volume, which means a) you need another 1Tb of free space and b) the head thrashing of reading one file and ...


2

7z would be my choice. It allows auto-splitting of archives and supports multi-threaded compression. No, xz doesn't, despite what the help message says. Try with: 7za a -v100m -m0=lzma2 -mx=9 -ms=on -mmt=$THREADS archive.7z directory/ The output is split in 100MB blocks (change it with the -v switch). The only real downside is that 7z does not retain ...


4

Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer: At least as fast as other internet transfers: your upload speed will determine the transfer speed No data corruption Choose which files to transfer first No extra local/cloud storage space needed Free You didn't tell which OS you were using, but as you're ...


27

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


12

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


7

Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party. The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. ...


0

I suggest you to use something like : tar -c -L 104857600 -f file1.tar -f file2.tar -f file3.tar -f file4.tar -f file5.tar -f file6.tar -f file7.tar -f file8.tar -f file9.tar -f file10.tar gzip file*.tar which will create 10 files which will be at most 100GB big. But that wouldn't answer your "OOM" issue since tar is supposed to stream it's read/write so ...


0

How about that: find /path | tar -T - -czf - | ssh remotehost "cd /target/dir/; tar xzf -" find searches lists the directory tree tar -T - reads the list from stdin -czf - creates an archive and writes to stdout, z gzips the stream ssh remotehost login to remotehost using ssh cd /target/dir/ change to the target directory tar xzf - extract the incoming ...


1

Generally, no. It would be possible to design a filesystem that provides this kind of deduplication, but it would be very costly, for very little practical benefit, so I doubt that it's been done. The issue is that deduplication only looks at aligned extents. Deduplicating filesystems generally work at a block level. When the filesystem driver is about to ...


2

This is simple as less a.tar:./x/y.txt This magic trick works if you have lesspipe installed and if the env variable LESSOPEN is defined to be | /usr/bin/lesspipe.sh %s which is expected if you have lesspipe installed correctly.


1

Oh, but this is a question about the contents of a file within a tar file. And actually, in some cases this isn't so hard. The thing is, a tar file is just a blocked out stream file - each file within the archive is found after the one before it, and each file gets a metadata header based on a specified format. Based on that format, I once wrote shitar - ...


2

This prints contents of ./x/y.txt from a.tar to STDOUT. tar xfO a.tar ./x/y.txt


0

You can use this line tar -axf a.tar -O


3

It's probably a GNU specific option, but you could use the -O or --to-stdout to extract files to standard output ~/test$ mkdir foo ~/test$ cd foo ~/test/foo$ cat >bar hello world ~/test/foo$ cd .. ~/test$ tar -acvf file.tgz foo foo/ foo/bar ~/test$ ls file.tgz foo ~/test$ rm -rf foo ~/test$ tar -axf file.tgz foo/bar -O hello world ~/test$ ls file.tgz



Top 50 recent answers are included