Hot answers tagged

158

Advantages of using .tar.gz instead of .gz are that tar stores more meta-data (UNIX permissions etc.) than gzip. the setup can more easily be expanded to store multiple files .tar.gz files are very common, only-gzipped files may puzzle some users. (cf. MelBurslans comment) The overhead of using tar is also very small. If not really needed, I still do ...


84

With a recent GNU tar on bash or derived shell: XZ_OPT=-9 tar cJf tarfile.tar.xz directory tar's lowercase j switch uses bzip, uppercase J switch uses xz. The XZ_OPT environment variable lets you set xz options that cannot be passed via calling applications such as tar. This is now maximal. See man xz for other options you can set (-e/--extreme ...


60

You are actually asking only half of the question. The other question being, "Why would I compress a tar file with gzip?". And the answer is not just that gzip makes the file smaller (in most cases): tar: stores filename and other metadata: mode, owner ID, group ID, filesize, modification time stores a checksum (for the header only) gzip: can store ...


55

It is also possible to decompress it using standard shell script + gzip. The trick is to prepend the gzip magic number and compress method (see http://www.onicos.com/staff/iz/formats/gzip.html) to the actual data: printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" |cat - zlib.raw |gzip -dc


44

See this answer. Quoted below for convenience: Calculate the bitrate you need by dividing 1 GB by the video length in seconds. So, for a video of length 16:40 (1000 seconds), use a bitrate of 1000000 bytes/sec: ffmpeg -i input.mp4 -b 1000000 output.mp4 Additional options that might be worth considering is setting the Constant Rate Factor, which ...


40

Assuming xz honors the standard set of commandline flags - including compression level flags, you could try: tar -cf - foo/ | xz -9 -c - > foo.tar.xz


40

Such an utility is zerofree. From its description: Zerofree finds the unallocated, non-zeroed blocks in an ext2 or ext3 file-system and fills them with zeroes. This is useful if the device on which this file-system resides is a disk image. In this case, depending on the type of disk image, a secondary utility may be able to reduce the size of the disk ...


37

It depends on whether the disk image is a full disk image, or just a partition. Washing the partition(s) If the disk is in good working condition, you will get better compression if you wash the empty space on the disk with zeros. If the disk is failing, skip this step. If you're imaging an entire disk then you will want to wash each of the partitions on ...


36

You can do that using unzip -Zt zipname which prints a summary directly about the archive content, with total size. Here is an example on its output: unzip -Zt a.zip 1 file, 14956 bytes uncompressed, 3524 bytes compressed: 76.4% Then, using awk, you can extract the number of bytes: unzip -Zt a.zip | awk '{print $3}' 14956 Finally, put it in a for loop ...


34

Install zip and use zip -r foo.zip . You can use the flags -0 (none) to -9 (best) to change compressionrate Excluding files can be done via the -x flag. From the man-page: -x files --exclude files Explicitly exclude the specified files, as in: zip -r foo foo -x \*.o which will include the contents of foo in ...


34

Tar is an archiving tool (Tape ARchive), it only collects files and their metadata together and produces one file. If you want to compress that file later you can use gzip/bzip2/xz. For convenience, tar provides arguments to compress the archive automatically for you. Checkout the tar man page for more details.


33

I have found a solution (one of the possible ones), it's using openssl: $ openssl zlib -d < /tmp/data or $ openssl zlib -d -in /tmp/data *NOTE: zlib functionality is apparently available in recent openssl versions >=1.0.0 (OpenSSL has to be configured/built with zlib or zlib-dynamic option, the latter is default)


32

Compression ratio is very dependent of what you're compressing. The reason text compresses down so well is because it doesn't even begin to fully utilize the full range of numbers representable in the same binary space. So formats that do (e.g compressed files) can store the same information in less space just by virtue of using all those binary numbers that ...


32

If you want to grep recursively in all .eml.gz files, you can use: find -name \*.eml.gz -print0 | xargs -0 zgrep "STRING" You have to escape the first '*' so that the shell does not interpret it. "-print0" tells find to print a null character after each file it finds; "xargs -0" reads from standard input and runs the command after it for each file; ...


31

It seems that the original bzip was pulled circa 1998 due to patent issues with the arithmetic compression used in. A bit of digging (really only reading Wikipedia) turns up an archived link to the bzip2 website from around this time. Here is the relevant section detail this and other differences: How does it relate to your previous offering (bzip-0.21) ...


29

There is a quite big advantage to using only-gzipped text files - the contents can be directly accessed with command-line tools like less, zgrep, zcat.


28

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


27

lrzip is what you're really looking for, especially if you're compressing source code! Quoting the README: This is a compression program optimised for large files. The larger the file and the more memory you have, the better the compression advantage this will provide, especially once the files are larger than 100MB. The advantage can be chosen to ...


21

I would say it's likely that the people just don't realise they can use gzip/bzip2/xz without tar. Possibly because they come from a DOS/Windows background where it is normal for compression and archiving to be integrated in a single format (ZIP, RAR, etc). While there may be slight advantages to using tar in some situations due to the storage of metadata ...


19

Most movie formats/encodings (and image formats too) are already compressed. You can't compress them much further by adding a second compression layer - same as if you try to zip a bzip2-compressed archive (or the other way around). You won't find a lossless compression algorithm that will compress these files much further. In fact, you might even end up ...


19

If you type unzip -l <zipfile>, it prints a listing of files within the zip, with their uncompressed sizes, then the total uncompressed size of all of them. This is human-readable output, but you can get a machine-readable number using unzip -l <zipfile> | tail -n1 | awk '{ print $1 }'. To get a total size: total=0 for file in *.zip; do # or ...


18

You can configure the key bindings and set many settings for less in a file called ~/.lesskey. Once you've created the file, run the lesskey command; it generates a file called ~/.less which less reads when it starts. The setting you want is LESSOPEN. It's an input formatter for less. The less package comes with a sample formatter in /bin/lesspipe; it ...


18

7zip is more a compactor (like PKZIP) than a compressor. It's available for Linux, but it can only create compressed archives in regular files, it's not able to compress a stream for instance. It's not able to store most of Unix file attributes like ownership, ACLs, extended attributes, hard links... On Linux, as a compressor, you've got xz that uses the ...


18

There's a lot of confusion here because there isn't just one zgrep. I have two versions on my system, zgrep from gzip and zgrep from zutils. The former is just a wrapper script that calls gzip -cdfq. It doesn't support the -r, --recursive switch.1 The latter is a c++ program and it supports the -r, --recursive option. Running zgrep --version | head -n 1 will ...


17

You can create a 10MB gzip file like this: head -c 10M /dev/urandom | gzip -1 >10m.gz This uses urandom to get a high-entropy stream of bytes: since this is incompressible, the gzipped version will be about the same size as the input. You can then catenate copies of your gzip file together: cat $(perl -e "print '10m.gz ' x 30") >300m.gz Thirty ...


17

There is an important difference that could make using tar important under some circumstances: Besides the "metadata" that @jofel mentioned in his answer, tar records the filename in the archive. When you extract it, you get the original filename regardless of what the archive is called. In your case the tar archive and the file it contains have the related ...


15

AFAIK there is no limit of size for gzip - at least not 30GB. Of course, you need the space for the zipped file on your disc, both versions will be there simultanously while compressing. bzip2 compresses files (not only big ones :-) better, but it is (sometimes a lot) slower.


15

unzip -l lists the size of each file and prints a final line with their sum. So you can loop through the zip files and add up the output of unzip -l "$zip" | awk 'END {print $1}' or of unzip -Zt "$zip" | awk 'END {print $3}'. For a shell loop, unzip -Zt may be a little faster: total=0 for z in *.zip; do set $(unzip -Zt -- "$z") total=$((total + $3)) ...


13

You compression attempts failed because your data is already highly compressed and there's not much more to gain, see the other answers for more detailed explanations. However, if you can agree on lossy compression, in contrast to lossless like you tried before, you can compress the images significantly. But since data is cut away, it can not be undone. ...



Only top voted, non community-wiki answers of a minimum length are eligible