gzip, bzip2 and zip: very low compression ratio

Question

I need to transfer (via ftp) some movie files (.MPG) on to remote machine. I thought to 1st compress it and then send. But all the compression utilities (gzip, bzip2 and zip) are having very less compression ratio. For example:

ravbholua@ravbholua-Aspire-5315:~/Desktop/MAIL/SNEHAMILAN$ gzip -l MOV04042.MPG.gz
         compressed        uncompressed  ratio uncompressed_name
           61949207            66970629   7.5% MOV04042.MPG
ravbholua@ravbholua-Aspire-5315:~/Desktop/MAIL/SNEHAMILAN$

As you see here, so less compression. Same is the case with bzip2 and zip. Please suggest any better compression utility so that I can compress 400 MB movie files to a good extent and then ftp it. What I w'd do is: 1st I w'd tar all the movie files and then compress it to ftp on remote machine.

Answer 1

Most movie formats/encodings (and image formats too) are already compressed. You can't compress them much further by adding a second compression layer - same as if you try to zip a bzip2-compressed archive (or the other way around).

You won't find a lossless compression algorithm that will compress these files much further. In fact, you might even end up with compressed files slightly larger than the original in some cases.

Your best chance at compressing media files further is to recompress the audio and video with more efficient encoders. Your .mpg extension suggests you are using MPEG-2 now, for example, so you might try switching to H.264 or VP8/WebM. You can also try changing the existing encoder's parameters to get smaller files, by trading away some quality for a smaller file. You will probably spend more CPU time re-encoding than you save in a single file transfer, but it may be worth doing if you are going to transfer these files multiple times.

Answer 2

Most lossless compression (like the algorithms used in gzip, bzip2, and zip) works by eliminating long repeated series of bytes in a file. As a bit of a contrived example, let's say your file has several instances of 100 spaces: a compressed version of the file might create a very short code that means 100 spaces, and replace those instances with this.

The catch is that for this to work, the file has to have a considerable number of repeated sequences of bytes. Most ordinary files do, which is why compression generally works well, but compressed files usually don't (that is, after all, the point of compression). That's why double-compression doesn't usually work very well: after you compress a file once, you've already taken out most of the things that made it compressible. Interestingly, it is possible to make files that compression would actually make larger, but from a realistic standpoint, files like that do not occur very often.

Lossy compression isn't so different, really. In its simplest sense, it alters the file in certain ways to make it compress better, but it tries to do this in ways that the user won't notice. In audio compression, these are called psychoacoustic algorithms, because it's all about changing the audio in ways that a human mind won't detect; I assume there's a similar word for video compression, but I don't know what it is. Anyway, once they've done these tricks to make the file more compressible, then they compress it in the usual way. But this means that lossy-compressed files also don't have many of the things that lossless algorithms look for.

Because of this, I'm afraid you're mostly out of luck. @WarrenYoung's answer can work if you have access to the original video, but if you only have these .mpg files, you'll need to be careful about re-encoding them: you won't break the file or anything, but the results might not look as good as if you had started with the original file. Different lossy algorithms throw out different kinds of data, but effects of doing that can stack up and become noticeable. When you're done re-encoding, you should watch the file to make sure it still looks good.

Answer 3

There are some more efficient compression formats available out there (like ZPAQ, PEA or KGB, see the Wikipedia PAQ article), but they come at the expense of processing time.

As Mat pointed out, it's doubtful they will compress your (already compressed) movie further by much.