Shrinking (compressing) and restoring (decompressing) of data.

learn more… | top users | synonyms

91
votes
8answers
10k views

Why would I tar a single file?

At my company, we download a local development database snapshot as a db.dump.tar.gz file. The compression makes sense, but the tarball only contains a single file (db.dump). Is there any point to ...
63
votes
7answers
45k views

How to XZ a directory with TAR using maximum compression?

So I need to compress a directory with max compression. How can I do it with xz? I mean I will need tar too because I can't compress a directory with only xz. Is there a oneliner to produce e.g. ...
42
votes
1answer
76k views

How can I reduce a video's size with ffmpeg?

How can I use ffmpeg to reduce the size of a video by lowering the quality (as minimally as possible, naturally, but I need it to run on a mobile device that doesn't have much available space)? I ...
40
votes
5answers
33k views

Is it possible to mount a gzip compressed dd image on the fly?

I like create an image backup for the first time I'm backing up a system. After this first time I use rsync to do incremental backups. My usual image backup is as follows: Mount and zero out the ...
40
votes
8answers
55k views

How do I grep recursively through .gz files?

I am using a script to regularly download my gmail messages that compresses the raw .eml into .gz files. The script creates a folder for each day, and then compresses every message into its own file. ...
38
votes
1answer
2k views

What happened to bzip1?

bzip2 had been a de facto standard for quite strong compression throughout many years already. I myself had typed the bzip2 command thousands of times already, which makes me wonder - what happened to ...
34
votes
5answers
40k views

How to uncompress zlib data in UNIX?

I have created zlib-compressed data in Python, like this: import zlib s = '...' z = zlib.compress(s) with open('/tmp/data', 'w') as f: f.write(z) (or one-liner in shell: echo -n '...' | python2 ...
32
votes
5answers
35k views

Clear unused space with zeros (ext3,ext4)

How to clear unused space with zeros ? (ext3,ext4) I'm looking for something smarter than cat /dev/zero > /mnt/X/big_zero ; sync; rm /mnt/X/big_zero Like FSArchiver is looking for "used space" ...
28
votes
2answers
29k views

Zip everything in current directory

I'd like to compress and package everything, including files and folders in current directory, into a single ZIP file on Ubuntu. What would be the most convenient command for this (and name of the ...
24
votes
5answers
29k views

Which file compression software for linux offers the highest size reduction? [closed]

I do a ton of file compression. Most of the stuff I am compressing is just code, so I need to use loss less compression. I was wondering if there was anything that offers a better size reduction ...
22
votes
4answers
2k views

How to know how much space an uncompressed zip will take

Given a (really long) list of zip files, how can you tell the size of them once uncompressed?
19
votes
4answers
8k views

Does tar actually compress files, or just group them together?

I usually assumed that tar was a compression utility, but I am unsure, does it actually compress files, or is it just like an ISO file, a file to hold files?
17
votes
3answers
3k views

Fastest way of working out uncompressed size of large GZIPPED file

Once a file is gzipped, is there a way of quickly querying it to say what the uncompressed file size is (without decompressing it), especially in cases where the uncompressed file is > 4GB in size. ...
13
votes
9answers
4k views

Memory problems when compressing and transferring a large number of small files (1TB in total)

I have 5 million files which take up about 1TB of storage space. I need to transfer these files to a third party. What's the best way to do this? I have tried reducing the size using .tar.gz, but ...
12
votes
3answers
2k views

Universal extractors

With an increasing number of archive/compression file formats, is there a single free/open-source command line tool to rule them all? Perhaps something with a consistent / unified set of flags? (note ...
11
votes
3answers
8k views

How can I view gzipped files in less without having to type zless?

I am using Ubuntu, and I would like to be able to type less compressed_text_file.gz and page the contents of the text file in uncompressed form. Is there a way to do this?
11
votes
3answers
8k views

How do I compress files in-place?

I have a machine with 90% hard-disk usage. I want to compress its 500+ log files into a smaller new file. However, the hard disk is too small to keep both the original files and the compressed ones. ...
11
votes
3answers
720 views

What does the `-C` flag exactly do in `scp`?

I always use either rsync or scp in order to copy file from/to a remote machine. Recently, I discovered in the manual of scp (man scp) the flag -C -C Compression enable. Passes the -C flag to ...
10
votes
2answers
10k views

How to create a tar.gz file from a folder excluding a folder

How do I exclude a file or folder while creating a tar.gz file with tar command?
9
votes
4answers
22k views

Is it possible to compress a very large file (~30 GB) using gzip?

Is it possible to compress a very large file (~30 GB) using gzip? If so, what commands, switches, and options should I use? Or is there another program (preferably one commonly available on Ubuntu ...
9
votes
1answer
10k views

How to extract files from uImage?

Buildroot is generating Images for an embedded device where they should run on. This is working very well. In those Images the rootfs is included. Due to some research I'd like to look into that ...
8
votes
2answers
564 views

Create a gzip file of a predetermined size

I want to test an upload of files of content type application/x-gzip of various sizes ranging from 100 MB to 999 MB. How can go I about creating .gz files of these predetermined sizes? If I do dd ...
8
votes
6answers
280 views

Is there any program to provide a consistent interface across multiple archive types?

At the moment, if I download a compressed file, it could be any of a .tar.gz archive, a tar.bz2 arhive, a .zip archive or a .gz archive. And each time I do so, I have to remember what the command line ...
8
votes
3answers
542 views

Can tarring files improve compression?

Can tarring a bunch of files together improve compression with the standard tools, e.g. gzip, bzip2, xz? I've long thought this to be the case but never tested it out. If we have 2 copies of the same ...
8
votes
3answers
15k views

speed up gzip compression

Is it possible to speed up the gzip process? I'm using mysqldump "$database_name" | gzip > $BACKUP_DIR/$database_name.sql.gz to backup a database into a directory, $BACKUP_DIR. the manpage ...
8
votes
1answer
5k views

On the Fly Compression for a Directory

I have a directory that is always going to be storing text files that are rarely (think weekly) used. Naturally this is a great place to use compression. However, rather than having to use tar ever ...
8
votes
2answers
8k views

What's the difference between gunzip and unzip?

What is the difference between gunzip and unzip? Tried to search but couldn't find anything relevant with this.
8
votes
1answer
1k views

How should I combine many compressed files into one archive?

I have a few hundred .tar.xz files which are almost identical (they are daily database dumps, and the database changes slowly). I believe that due to the similarities in the uncompressed files, they ...
7
votes
3answers
3k views

Is there a command to list the compressed file sizes for files within a .zip file?

I'd like to see the absolute size in bytes of each file that has been compressed into single zip file. Having read the zip man page, I'm not sure that that utility can do it. This is on Mac OS X. ...
7
votes
3answers
5k views

Unpack file compressed in .txz and .tar with one command

I have a file compressed in *.txz. After unpacking it I received a *.tar file. Is there any way to unpack it twice with one command? I mean unpack file (*.tar).txz with one command? For know I'm do ...
7
votes
2answers
1k views

How to store a large folder in a single file without compression

Sorry if this is a stupid question, but I want to take a 78gb folder and store it in a single file (for upload into a cloud service), as if I am compressing it in an archive, but I don't want any ...
7
votes
5answers
22k views

How to read a .war file?

I am trying to view the content of a .war file. I first did set its permissions with chmod 777 then when I try to accces it using: cd /usr/local/standalone/deployments/Sample.war/WEB-INF/classes/ ...
7
votes
3answers
994 views

How to recompress 2 million gzip files without storing them twice?

I have about 2 million (60GiB) of gzipped small files and I would like to create a compressed archive containing all of them in an uncompressed version. Unfortunately, I cannot just uncompress them ...
7
votes
2answers
1k views

Why use cpio for initramfs?

I am making my own initramfs following the Gentoo wiki. Instead of the familiar tar and gzip, the page is telling me to use cpio and gzip. Wikipedia says that cpio is used by the 2.6 kernel's ...
6
votes
2answers
4k views

What's the difference between tar.gz and tar.Z?

The downloads on this page, e.g. of NetCDF C/C++/Fortran 4.1.3, show tar.Z and tar.gz formats. The tar.Z doesn't even have a configure file in it. What is it used for?
6
votes
3answers
399 views

Does gz compression ratio improve over time?

I have some process that creates a stream of millions of highly similar lines. I'm piping this to gz. Does the compression ratio improve over time in such a setup? I.e. is the compression ratio better ...
6
votes
1answer
364 views

How to specify additional parameters for GNU sort's --compress-program option?

sort --compress-program=/bin/gzip works fine, but how can I pass options down to the compression programm (e.g. --best or --fast)? sort --compress-program=/bin/gzip\ --best failes with the error ...
6
votes
1answer
860 views

Is there any way in BTRFS to set compression permanently?

I have a USB HDD. I want to mount it with compression enabled. I can do in the fstab of my system or even using udev rules. The problem is that I won't mount my USB HDD on my computer only. Up to ...
6
votes
2answers
169 views

How do I layer compression on top of encryption on top of BtrFS, per-file?

BtrFS should support this really, but it doesn't. So ECryptFS looks like it will fill the gap. The only thing is, how would I layer compression on top of this? Reasons for compression on top of ...
5
votes
2answers
15k views

Having trouble uncompressing a few files [duplicate]

So I have to uncompress 4 different files. When I do file file1 file file2 file file3 file file4 for example, different messages are displayed. For example one starts like "gzip compressed data, ...
5
votes
3answers
12k views

Determine whether a particular file is compressed

How to know in Linux whether a particular file is compressed or not using a command?
5
votes
1answer
220 views

Get compressed size of piped output with gzip?

To get uncompressed size of already compressed file, I can use -l option in gzip utility: gzip -l compressedfile.gz However is there a way to get size of compressed file if I am piping the output? ...
5
votes
1answer
1k views

Appending checksum information to file

I'd like to embed checksum information into file, that I transfer. It's tar.gz or tar.xz file and I can only transfer one file to remote side. How do you recommend I embed checksum information? I ...
5
votes
2answers
2k views

rsync-like tool that supports compression and maintains hardlinks?

I use an rsync-based backup scheme to maintain daily (and soon weekly and monthly as well) "snapshots" of my file server, using rsync's link-dest feature to store hardlinks between the snapshots for ...
5
votes
2answers
725 views

Best compression of similar files?

I've got a few full old backups of things like binary database dumps. Obviously, they don't differ much so doing full backups is not the smartest idea here. For now, I'm looking for a compression ...
5
votes
1answer
267 views

What do you do to scan with 600 dpi while maintaing a small output file?

I just installed xsane which I already successfully tested with my Epson Perfection V30. When scanning with these settings the output file is remarkably large. Is there a way to decrease the size, ...
5
votes
1answer
492 views

Command-line incremental backup tools

Which command line backup tool(s) has the best compression ratio? I want to backup my entire system, including media files, text files etc. I found this list on the Arch wiki site but I don't know ...
4
votes
2answers
1k views

Compressed filesystem inside a file in Linux

I have a flash drive which is FAT32 formatted. I want to put a linux filesystem on the drive inside a file. I know I can do this by creating a file and formatting is with ext3 (or any other file ...
4
votes
3answers
530 views

Compression / archiving that keeps permissions and file owner?

I know that similar questions have been asked about permissions, but which form of compression or archiving keeps the permissions and file owners of each file and directory? I was considering the ...
4
votes
1answer
1k views

Does ssh compress connections in a tunnel individually?

I have one compressed SSH tunnel with one tunnel and multiple connections over it. Data in every connection tunneled is rather homogeneous, but compressing an interleaved stream will obviously make ...