Questions tagged [compression]

Shrinking (compressing) and restoring (decompressing) of data.

Filter by
Sorted by
Tagged with
204
votes
9answers
287k views

How can I reduce a video's size with ffmpeg?

How can I use ffmpeg to reduce the size of a video by lowering the quality (as minimally as possible, naturally, but I need it to run on a mobile device that doesn't have much available space)? I ...
135
votes
6answers
300k views

How do I grep recursively through .gz files?

I am using a script to regularly download my gmail messages that compresses the raw .eml into .gz files. The script creates a folder for each day, and then compresses every message into its own file. ...
116
votes
12answers
100k views

How to XZ a directory with TAR using maximum compression?

So I need to compress a directory with max compression. How can I do it with xz? I mean I will need tar too because I can't compress a directory with only xz. Is there a oneliner to produce e.g. foo....
106
votes
10answers
157k views

How to uncompress zlib data in UNIX?

I have created zlib-compressed data in Python, like this: import zlib s = '...' z = zlib.compress(s) with open('/tmp/data', 'w') as f: f.write(z) (or one-liner in shell: echo -n '...' | python2 -...
101
votes
8answers
18k views

Why would I tar a single file?

At my company, we download a local development database snapshot as a db.dump.tar.gz file. The compression makes sense, but the tarball only contains a single file (db.dump). Is there any point to ...
84
votes
2answers
120k views

Zip everything in current directory

I'd like to compress and package everything, including files and folders in current directory, into a single ZIP file on Ubuntu. What would be the most convenient command for this (and name of the ...
73
votes
6answers
89k views

Clear unused space with zeros (ext3,ext4)

How to clear unused space with zeros ? (ext3,ext4) I'm looking for something smarter than cat /dev/zero > /mnt/X/big_zero ; sync; rm /mnt/X/big_zero Like FSArchiver is looking for "used space" ...
66
votes
6answers
63k views

Is it possible to mount a gzip compressed dd image on the fly?

I like create an image backup for the first time I'm backing up a system. After this first time I use rsync to do incremental backups. My usual image backup is as follows: Mount and zero out the ...
51
votes
5answers
72k views

Which file compression software for linux offers the highest size reduction? [closed]

I do a ton of file compression. Most of the stuff I am compressing is just code, so I need to use loss less compression. I was wondering if there was anything that offers a better size reduction ...
40
votes
1answer
2k views

What happened to bzip1?

bzip2 had been a de facto standard for quite strong compression throughout many years already. I myself had typed the bzip2 command thousands of times so far, which makes me wonder - what happened to ...
37
votes
4answers
26k views

Does tar actually compress files, or just group them together?

I usually assumed that tar was a compression utility, but I am unsure, does it actually compress files, or is it just like an ISO file, a file to hold files?
33
votes
3answers
25k views

What does the `-C` flag exactly do in `scp`?

I always use either rsync or scp in order to copy file from/to a remote machine. Recently, I discovered in the manual of scp (man scp) the flag -C -C Compression enable. Passes the -C flag to ...
30
votes
3answers
33k views

How can I view gzipped files in less without having to type zless?

I am using Ubuntu, and I would like to be able to type less compressed_text_file.gz and page the contents of the text file in uncompressed form. Is there a way to do this?
24
votes
4answers
12k views

Fastest way of working out uncompressed size of large GZIPPED file

Once a file is gzipped, is there a way of quickly querying it to say what the uncompressed file size is (without decompressing it), especially in cases where the uncompressed file is > 4GB in size. ...
22
votes
4answers
3k views

How to know how much space an uncompressed zip will take

Given a (really long) list of zip files, how can you tell the size of them once uncompressed?
22
votes
5answers
5k views

On-the-fly stream compression that doesn't spill over into hardware resources?

I have 200 GB free disk space, 16 GB of RAM (of which ~1 GB is occupied by the desktop and kernel) and 6 GB of swap. I have a 240 GB external SSD, with 70 GB used1 and the rest free, which I need to ...
22
votes
4answers
9k views

How to decompress jsonlz4 files (Firefox bookmark backups) using the command line?

There seems to be various JavaScript+browser specific ways of decompressing this, but isn't there some way to transform jsonlz4 files to something unlz4 will read?
21
votes
3answers
45k views

How to extract files from uImage?

Buildroot is generating images for an embedded device where they should run. This is working very well. In those images, the rootfs is included. Due to some research, I'd like to look into that ...
20
votes
2answers
3k views

Zipped hard drive image very big

I just made an image of a freshly installed dual boot (Ubuntu and Windows) using this command (which I've been using for a while for smaller images): dd if=/dev/sda | gzip > /mnt/drive.img.gz On ...
20
votes
4answers
29k views

How do I compress files in-place?

I have a machine with 90% hard-disk usage. I want to compress its 500+ log files into a smaller new file. However, the hard disk is too small to keep both the original files and the compressed ones. ...
20
votes
2answers
62k views

Having trouble uncompressing a few files [duplicate]

So I have to uncompress 4 different files. When I do file file1 file file2 file file3 file file4 for example, different messages are displayed. For example one starts like "gzip compressed data, …, ...
19
votes
3answers
3k views

Universal extractors

With an increasing number of archive/compression file formats, is there a single free/open-source command line tool to rule them all? Perhaps something with a consistent / unified set of flags? (note ...
17
votes
6answers
72k views

How to read a .war file?

I am trying to view the content of a .war file. I first did set its permissions with chmod 777 then when I try to accces it using: cd /usr/local/standalone/deployments/Sample.war/WEB-INF/classes/ It ...
16
votes
5answers
51k views

Is it possible to compress a very large file (~30 GB) using gzip?

Is it possible to compress a very large file (~30 GB) using gzip? If so, what commands, switches, and options should I use? Or is there another program (preferably one commonly available on Ubuntu ...
15
votes
9answers
9k views

Memory problems when compressing and transferring a large number of small files (1TB in total)

I have 5 million files which take up about 1TB of storage space. I need to transfer these files to a third party. What's the best way to do this? I have tried reducing the size using .tar.gz, but ...
15
votes
3answers
37k views

speed up gzip compression

Is it possible to speed up the gzip process? I'm using mysqldump "$database_name" | gzip > $BACKUP_DIR/$database_name.sql.gz to backup a database into a directory, $BACKUP_DIR. the manpage says:...
15
votes
3answers
38k views

Determine whether a particular file is compressed

How to know in Linux whether a particular file is compressed or not using a command?
13
votes
8answers
500 views

Is there any program to provide a consistent interface across multiple archive types?

At the moment, if I download a compressed file, it could be any of a .tar.gz archive, a tar.bz2 arhive, a .zip archive or a .gz archive. And each time I do so, I have to remember what the command line ...
11
votes
3answers
9k views

Is there a command to list the compressed file sizes for files within a .zip file?

I'd like to see the absolute size in bytes of each file that has been compressed into single zip file. Having read the zip man page, I'm not sure that that utility can do it. This is on Mac OS X. ...
11
votes
2answers
10k views

How to store a large folder in a single file without compression

I want to take a 78gb folder and store it in a single file (for upload into a cloud service), as if I am compressing it in an archive, but I don't want any compression (I don't have that much CPU time ...
11
votes
3answers
3k views

Why use cpio for initramfs?

I am making my own initramfs following the Gentoo wiki. Instead of the familiar tar and gzip, the page is telling me to use cpio and gzip. Wikipedia says that cpio is used by the 2.6 kernel's ...
11
votes
2answers
18k views

What's the difference between gunzip and unzip?

What is the difference between gunzip and unzip? Tried to search but couldn't find anything relevant with this.
11
votes
1answer
10k views

On the Fly Compression for a Directory

I have a directory that is always going to be storing text files that are rarely (think weekly) used. Naturally this is a great place to use compression. However, rather than having to use tar ever ...
11
votes
3answers
9k views

Private VPN compression for faster Internet

Background: I'm a student, and currently the Wi-Fi connection at my school is very slow (dead slow). I have an 8Gb 4G data-plan, but even with that, I'm running short at the end of the month. I ...
10
votes
3answers
33k views

Unpack file compressed in .txz and .tar with one command

I have a file compressed in *.txz. After unpacking it I received a *.tar file. Is there any way to unpack it twice with one command? I mean unpack file (*.tar).txz with one command? For know I'm do ...
10
votes
2answers
16k views

How to create a tar.gz file from a folder excluding a folder

How do I exclude a file or folder while creating a tar.gz file with tar command?
10
votes
3answers
11k views

How do I reduce the size of a pdf file that contains images?

I have a pdf file that contains images and I want to reduce its size in order to upload it to a site with a size limit. So, how can I reduce the size of a pdf file from the command-line?
10
votes
1answer
4k views

How should I combine many compressed files into one archive?

I have a few hundred .tar.xz files which are almost identical (they are daily database dumps, and the database changes slowly). I believe that due to the similarities in the uncompressed files, they ...
9
votes
1answer
966 views

Uncompressed file estimation wrong?

I had a large (~60G) compressed file (tar.gz). I used split to break it into 4 parts and then cat to join them back together. However, now, when I am trying to estimate the size of the uncompressed ...
9
votes
3answers
794 views

Can tarring files improve compression?

Can tarring a bunch of files together improve compression with the standard tools, e.g. gzip, bzip2, xz? I've long thought this to be the case but never tested it out. If we have 2 copies of the same ...
9
votes
3answers
8k views

why should I use tar.xz instead of tar.gz? xz is a lossless data compression program and file format [duplicate]

Today first time in my life I saw tar.xz download. I searched the internet and found Wikipedia articles (xz and XZ Utils) Interesting quote about the users of xz xz has gained notability for ...
9
votes
2answers
7k views

Clear unused space with zeros (btrfs)

How to clear unused space with zeros ? (I am asking for tool for btrfs filesystem) I'm looking for something smarter than cat /dev/zero > /mnt/X/big_zero ; sync; rm /mnt/X/big_zero Like ...
8
votes
2answers
2k views

Create a gzip file of a predetermined size

I want to test an upload of files of content type application/x-gzip of various sizes ranging from 100 MB to 999 MB. How can go I about creating .gz files of these predetermined sizes? If I do dd if=/...
8
votes
2answers
4k views

Print archive file list instantly (without decompressing entire archive)

A problem with .tar.gz archives is that, when I try to just list an archive's content, the computer actually decompresses it, which would take a very long time if the file is large. Other file ...
8
votes
2answers
8k views

What's the difference between tar.gz and tar.Z?

The downloads on this page, e.g. of NetCDF C/C++/Fortran 4.1.3, show tar.Z and tar.gz formats. The tar.Z doesn't even have a configure file in it. What is it used for?
8
votes
4answers
2k views

How to add a huge file to an archive and delete it in parallel

Say I have a file of 80GB /root/bigfile on a 100GB system and want to put this file in a archive /root/bigarchive.tar I obviously need to delete this file at the same time that it is added in the ...
8
votes
2answers
7k views

Compress zip files with higher compression

I have a large number of zip files that were compressed using the zip command. I would like to recompress them with the -9 flag to improve the compression ratio. Does anyone know if that can be done ...
8
votes
3answers
2k views

How to recompress 2 million gzip files without storing them twice?

I have about 2 million (60GiB) of gzipped small files and I would like to create a compressed archive containing all of them in an uncompressed version. Unfortunately, I cannot just uncompress them ...
8
votes
2answers
10k views

How strong is the encryption of a zip file in Linux Mint

Zip encryption has had often a bad reputation of being weak, but some would argue that a zip file encrypted using certain algorithms (such as by using AES), together with strong password, is really ...
8
votes
2answers
630 views

How do I layer compression on top of encryption on top of BtrFS, per-file?

BtrFS should support this really, but it doesn't. So ECryptFS looks like it will fill the gap. The only thing is, how would I layer compression on top of this? Reasons for compression on top of ...