290

See this answer. Quoted below for convenience: Calculate the bitrate you need by dividing 1 GB by the video length in seconds. So, for a video of length 16:40 (1000 seconds), use a bitrate of 1000000 bytes/sec: ffmpeg -i input.mp4 -b 1000000 output.mp4 Additional options that might be worth considering is setting the Constant Rate Factor, which ...


164

Advantages of using .tar.gz instead of .gz are that tar stores more meta-data (UNIX permissions etc.) than gzip. the setup can more easily be expanded to store multiple files .tar.gz files are very common, only-gzipped files may puzzle some users. (cf. MelBurslans comment) The overhead of using tar is also very small. If not really needed, I still do not ...


150

With a recent GNU tar on bash or derived shell: XZ_OPT=-9 tar cJf tarfile.tar.xz directory tar's lowercase j switch uses bzip, uppercase J switch uses xz. The XZ_OPT environment variable lets you set xz options that cannot be passed via calling applications such as tar. This is now maximal. See man xz for other options you can set (-e/--extreme ...


144

If you want to grep recursively in all .eml.gz files in the current directory, you can use: find . -name \*.eml.gz -print0 | xargs -0 zgrep "STRING" You have to escape the first * so that the shell does not interpret it. -print0 tells find to print a null character after each file it finds; xargs -0 reads from standard input and runs the command after it ...


141

It is also possible to decompress it using standard shell-script + gzip, if you don't have, or want to use openssl or other tools.The trick is to prepend the gzip magic number and compress method to the actual data from zlib.compress: printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" |cat - /tmp/data |gzip -dc >/tmp/out Edits: @d0sboots commented: For RAW ...


108

Install zip and use zip -r foo.zip . You can use the flags -0 (none) to -9 (best) to change compressionrate Excluding files can be done via the -x flag. From the man-page: -x files --exclude files Explicitly exclude the specified files, as in: zip -r foo foo -x \*.o which will include the contents of foo in foo....


83

Assuming xz honors the standard set of commandline flags - including compression level flags, you could try: tar -cf - foo/ | xz -9 -c - > foo.tar.xz


78

Such an utility is zerofree. From its description: Zerofree finds the unallocated, non-zeroed blocks in an ext2 or ext3 file-system and fills them with zeroes. This is useful if the device on which this file-system resides is a disk image. In this case, depending on the type of disk image, a secondary utility may be able to reduce the size of the disk ...


74

User @tino commented below the OpenSSL answer but I think this should be separate: zlib-flate -uncompress < FILE I tried this and it worked for me. zlib-flate can be found in package qpdf (in Debian Squeeze and Fedora 23, according to comments in other answers)


69

There's a lot of confusion here because there isn't just one zgrep. I have two versions on my system, zgrep from gzip and zgrep from zutils. The former is just a wrapper script that calls gzip -cdfq. It doesn't support the -r, --recursive switch.1 The latter is a c++ program and it supports the -r, --recursive option. Running zgrep --version | head -n 1 will ...


66

It depends on whether the disk image is a full disk image, or just a partition. Washing the partition(s) If the disk is in good working condition, you will get better compression if you wash the empty space on the disk with zeros. If the disk is failing, skip this step. If you're imaging an entire disk then you will want to wash each of the partitions on ...


65

lrzip is what you're really looking for, especially if you're compressing source code! Quoting the README: This is a compression program optimised for large files. The larger the file and the more memory you have, the better the compression advantage this will provide, especially once the files are larger than 100MB. The advantage can be chosen to ...


63

You are actually asking only half of the question. The other question being, "Why would I compress a tar file with gzip?". And the answer is not just that gzip makes the file smaller (in most cases): tar: stores filename and other metadata: mode, owner ID, group ID, filesize, modification time stores a checksum (for the header only) gzip: can store the ...


61

I have found a solution (one of the possible ones), it's using openssl: $ openssl zlib -d < /tmp/data or $ openssl zlib -d -in /tmp/data *NOTE: zlib functionality is apparently available in recent openssl versions >=1.0.0 (OpenSSL has to be configured/built with zlib or zlib-dynamic option, the latter is default)


49

Tar is an archiving tool (Tape ARchive), it only collects files and their metadata together and produces one file. If you want to compress that file later you can use gzip/bzip2/xz. For convenience, tar provides arguments to compress the archive automatically for you. Checkout the tar man page for more details.


38

You can do that using unzip -Zt zipname which prints a summary directly about the archive content, with total size. Here is an example on its output: unzip -Zt a.zip 1 file, 14956 bytes uncompressed, 3524 bytes compressed: 76.4% Then, using awk, you can extract the number of bytes: unzip -Zt a.zip | awk '{print $3}' 14956 Finally, put it in a for loop ...


37

7zip is more a compactor (like PKZIP) than a compressor. It's available for Linux, but it can only create compressed archives in regular files, it's not able to compress a stream for instance. It's not able to store most of Unix file attributes like ownership, ACLs, extended attributes, hard links... On Linux, as a compressor, you've got xz that uses the ...


33

Unless you're looking for a specific bitrate, I'd recommend the -crf option. This is the most commonly used for x264 encoding: http://slhck.info/articles/crf In short: a CRF of 23 would make "DVD" quality movie (~700MB-1GB) and lower CRF values would be higher quality (larger files).


33

Summary of the methods (as mentioned in this question and elsewhere) to clear unused space on ext2/ext3/ext4: Zeroing unused space File system is not mounted If the "disk" your filesystem is on is thin provisioned (e.g. a modern SSD supporting TRIM, a VM file whose format supports sparseness etc.) and your kernel says the block device understands it, you ...


32

Compression ratio is very dependent of what you're compressing. The reason text compresses down so well is because it doesn't even begin to fully utilize the full range of numbers representable in the same binary space. So formats that do (e.g compressed files) can store the same information in less space just by virtue of using all those binary numbers that ...


32

mkimage -l uImage Will dump the information in the header. tail -c+65 < uImage > out Will get the content. tail -c+65 < uImage | gunzip > out will get it uncompressed if it was gzip-compressed. If that was an initramfs, you can do cpio -t < out or pax < out to list the content. If it's a ramdisk image, you can try and mount it with: ...


32

It seems that the original bzip was pulled circa 1998 due to patent issues with the arithmetic compression used in. A bit of digging (really only reading Wikipedia) turns up an archived link to the bzip2 website from around this time. Here is the relevant section detail this and other differences: How does it relate to your previous offering (bzip-0.21) ?...


30

You mentioned wanting to reduce filesize to fit more videos on a mobile device, which is my usecase as well. All the answers here are for reducing the compression quality but nobody has mentioned reducing video frame size. It's a lot quicker, from about 3 to 5 times quicker than recompressing in my experience. See the ffmpeg docs on scaling for more info. ...


30

You can use the command file to determine the type of compression that was used for a particular file. Examples $ file file.zip file.zip: Zip archive data, at least v1.0 to extract To uncompress this file, use unzip. $ file file.rar file.rar: RAR archive data, v1d, os: Win32 To uncompress this file, use unrar. $ file file.7z file.7z: 7-zip archive ...


30

There is a quite big advantage to using only-gzipped text files - the contents can be directly accessed with command-line tools like less, zgrep, zcat.


29

How is that? Shouldn't gzip manage to compress all those zeros? Yes, if they were zeroes. Unused disk space does not mean it contains zeros; it means it is unused, and may contain anything. There are programs that wipe unused disk space to zeroes. I suggest you use those before making the disk image. (I don't recall any offhand; in Linux, I'd just use dd ...


28

I recommend pigz from Mark Adler, co-author of the zlib compression library. Execute pigz to see the available flags. You will notice: -z --zlib Compress to zlib (.zz) instead of gzip format. You can uncompress using the -d flag: -d --decompress --uncompress Decompress the compressed input. Assuming a file named 'test': pigz -z test - ...


27

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


22

I tested most of the other proposed answers to this question. The test data conclusions are below. These are the proposed answers that I tested: (BR) Modify the bitrate, using: ffmpeg -i $infile -b $bitrate $newoutfile (CR) Vary the Constant Rate Factor, using: ffmpeg -i $infile -vcodec libx264 -crf 23 $outfile (SZ) Change the video screen-size (for ...


22

AFAIK there is no limit of size for gzip - at least not 30GB. Of course, you need the space for the zipped file on your disc, both versions will be there simultanously while compressing. bzip2 compresses files (not only big ones :-) better, but it is (sometimes a lot) slower.


Only top voted, non community-wiki answers of a minimum length are eligible