Tag Info

Hot answers tagged

56

With a recent GNU tar on bash or derived shell: XZ_OPT=-9 tar cJf tarfile.tar.xz directory tar's lowercase j switch uses bzip, uppercase J switch uses xz. The XZ_OPT environment variable lets you set xz options that cannot be passed via calling applications such as tar. This is now maximal. See man xz for other options you can set (-e/--extreme ...


33

Tar is an archiving tool (Tape ARchive), it only collects files and their metadata together and produces one file. If you want to compress that file later you can use gzip/bzip2/xz. For convenience, tar provides arguments to compress the archive automatically for you. Checkout the tar man page for more details.


32

Compression ratio is very dependent of what you're compressing. The reason text compresses down so well is because it doesn't even begin to fully utilize the full range of numbers representable in the same binary space. So formats that do (e.g compressed files) can store the same information in less space just by virtue of using all those binary numbers that ...


31

It is also possible to decompress it using standard shell script + gzip. The trick is to prepend the gzip magic number and compress method (see http://www.onicos.com/staff/iz/formats/gzip.html) to the actual data: printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" |cat - zlib.raw |gzip -dc


31

It seems that the original bzip was pulled circa 1998 due to patent issues with the arithmetic compression used in. A bit of digging (really only reading Wikipedia) turns up an archived link to the bzip2 website from around this time. Here is the relevant section detail this and other differences: How does it relate to your previous offering (bzip-0.21) ...


27

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


26

Such an utility is zerofree. From its description: Zerofree finds the unallocated, non-zeroed blocks in an ext2 or ext3 file-system and fills them with zeroes. This is useful if the device on which this file-system resides is a disk image. In this case, depending on the type of disk image, a secondary utility may be able to reduce the size of the disk ...


23

It depends on whether the disk image is a full disk image, or just a partition. Washing the partition(s) If the disk is in good working condition, you will get better compression if you wash the empty space on the disk with zeros. If the disk is failing, skip this step. If you're imaging an entire disk then you will want to wash each of the partitions on ...


22

Assuming xz honors the standard set of commandline flags - including compression level flags, you could try: tar -cf - foo/ | xz -9 -c - > foo.tar.xz


20

Install zip and use zip -r foo.zip . You can use the flags -0 (none) to -9 (best) to change compressionrate Excluding files can be done via the -x flag. From the man-page: -x files --exclude files Explicitly exclude the specified files, as in: zip -r foo foo -x \*.o which will include the contents of foo in ...


19

I have found a solution (one of the possible ones), it's using openssl: $ openssl zlib -d < /tmp/data ... *NOTE: zlib functionality is apparently available in recent openssl versions >=1.0.0 (OpenSSL has to be configured/built with zlib or zlib-dynamic option, the latter is default)


18

Most movie formats/encodings (and image formats too) are already compressed. You can't compress them much further by adding a second compression layer - same as if you try to zip a bzip2-compressed archive (or the other way around). You won't find a lossless compression algorithm that will compress these files much further. In fact, you might even end up ...


17

You can create a 10MB gzip file like this: head -c 10M /dev/urandom | gzip -1 >10m.gz This uses urandom to get a high-entropy stream of bytes: since this is incompressible, the gzipped version will be about the same size as the input. You can then catenate copies of your gzip file together: cat $(perl -e "print '10m.gz ' x 30") >300m.gz Thirty ...


16

You can configure the key bindings and set many settings for less in a file called ~/.lesskey. Once you've created the file, run the lesskey command; it generates a file called ~/.less which less reads when it starts. The setting you want is LESSOPEN. It's an input formatter for less. The less package comes with a sample formatter in /bin/lesspipe; it ...


16

See this answer. Quoted below for convenience: Calculate the bitrate you need by dividing 1 GB by the video length in seconds. So, for a video of length 16:40 (1000 seconds), use a bitrate of 1000000 bytes/sec: ffmpeg -i input.mp4 -b 1000000 output.mp4 Additional options that might be worth considering is setting the Constant Rate Factor, which ...


14

lrzip is what you're really looking for, especially if you're compressing source code! Quoting the README: This is a compression program optimised for large files. The larger the file and the more memory you have, the better the compression advantage this will provide, especially once the files are larger than 100MB. The advantage can be chosen to ...


13

AFAIK there is no limit of size for gzip - at least not 30GB. Of course, you need the space for the zipped file on your disc, both versions will be there simultanously while compressing. bzip2 compresses files (not only big ones :-) better, but it is (sometimes a lot) slower.


13

You compression attempts failed because your data is already highly compressed and there's not much more to gain, see the other answers for more detailed explanations. However, if you can agree on lossy compression, in contrast to lossless like you tried before, you can compress the images significantly. But since data is cut away, it can not be undone. ...


12

.Z files are compressed with the older compress utility while .gz are compressed with gzip. Some ancient systems might be missing gzip/gunzip so will use uncompress and .Z files.


12

Use tar: tar -cf my_big_folder.tar /my/big/folder Restore the archive with tar -xf my_big_folder.tar -C / -C will change to the root directory to restore your archive since the archive created above contains absolute paths. EDIT: Due to the relatively big size of the archive, it'd be best to send it [directly] to its final location, using SSH or a mount ...


12

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


11

tar produces archives; compression is a separate functionality. However tar alone can reduce space usage when used on a large number of small files that are smaller than the filesystem's cluster size. If a filesystem uses 1kb clusters, even a file that contains a single byte will consume 1kb (plus an inode). A tar archive does not have this overhead. BTW, ...


10

Most common image formats are already compressed (like jpg, png, gif), so you don't get much savings. 1% sounds about right. Adding more compression can actually make the result (slightly) larger, because the compression algorithm has no benefit on compressed data, and then the format (eg. gzip) has to add header and/or structure information to the output. ...


10

I use atool. It does the job. It works with many, though not all formats: tar, gzip, bzip2, bzip, lzip, lzop, lzma, zip, rar, lha, arj, arc, p7zip etc. These compression tools are still needed, though as atool is simply a front end for them. I particularly like the als command it provides which lists the contents of any supported archive format. The main ...


10

7zip is more a compactor (like PKZIP) than a compressor. It's available for Linux, but it can only create compressed archives in regular files, it's not able to compress a stream for instance. It's not able to store most of Unix file attributes like ownership, ACLs, extended attributes, hard links... On Linux, as a compressor, you've got xz that uses the ...


10

gzip or bzip2 will compress the file and remove the non-compressed one automatically (this is their default behaviour). However, keep in mind that while the compressing process, both files will exists. If you want to compress log files (ie: files containing text), you may prefer bzip2, since it has a better ratio for text files. bzip2 -9 myfile # ...


9

zlib implements the compression used by gzip, but not the file format. Instead, you should use the gzip module, which itself uses zlib. import gzip s = '...' with gzip.open('/tmp/data', 'w') as f: f.write(s)


9

use the --exclude=directory flag. Works like a charm.


8

You can use the unzip utility with the -v flag: unzip -v files.zip Archive: files.zip Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 0 Stored 0 0% 11-23-2011 15:02 00000000 file1 0 Stored 0 0% 11-23-2011 15:02 00000000 file2 -------- ...



Only top voted, non community-wiki answers of a minimum length are eligible