Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have an application which I can not change, that writes to a big 250 GB plain text file. After compressing with GZ it is only 30 GB.

This application has no option of compressing it's output, and it can only write to a file name (not to stdout).

Is there a way that I can set up for the output to be compressed immediately, without first storing the 250 GB file on my disk?

I also need the other way around, to fool the application so that it reads a plaintext file which in reality is compressed.

share|improve this question
    
chattr A file with the `c' attribute set is automatically compressed on the disk by the kernel. A read from this file returns uncompressed data. A write to this file compresses data before storing them on the disk. Note: please make sure to read the bugs and limitations section at the end of this document. –  xenoterracide Nov 6 '10 at 20:38
    
I'm not sure what filesystems might actually support the c flag.... man page says ext2, ext3 doesn't... but it might be worth investigation. –  xenoterracide Nov 6 '10 at 20:39
    
@xenoterracide: The flag is implemented in chattr and the underlying kernel code. There's just no special behavior (such as changing the way the contents of the file is stored) in the Linux kernel if you use it. –  Gilles Nov 7 '10 at 16:18
    
@Gilles so you're saying it won't compress data on the disk? –  xenoterracide Nov 7 '10 at 18:33
    
@xenoterracide: Yes, as far as I know the mainline Linux kernel has never supported file compression in filesystems suitable for general use (as opposed to filesystems specialized for flash like JFFS2, and perhaps foreign filesystems (Stacker/Drivespace on FAT?)). There have been proposed patches for at least ext2 and reiserfs though. –  Gilles Nov 7 '10 at 19:44

4 Answers 4

up vote 6 down vote accepted

You may have success using /dev/stdout as the filename and piping the output of your application to gzip.

/dev/stdout is a symlink to /proc/self/fd/1.

Similarly, you may be able to use /dev/stdin as a filename and pipe the output of gzip to the application.

I say may, because the application may be expecting a seekable file that it writes to (reads from), but /dev/std{in,out} will not be seekable. If this is the case then you are probably lost. You will need to use a seekable file as the target for the application.

share|improve this answer
1  
If the application has a constraint on the file name (e.g. it must end in .foo), make a symbolic link to /dev/stdout (for output) or /dev/stdin (for input) that matches the name constraint. –  Gilles Nov 7 '10 at 15:56
    
Thanks, this is exactly what I needed! –  Peter Smit Nov 8 '10 at 8:49
  • Use can use mkfifo to setup a named pipe in which the program will write, while you gzip from that pipe into final destination.

  • If you may pass the output filename to the program on command line, then this (probably bash-specific) trick should also work: program >(gzip - >output.gz) as this will be translated by the shell into something like gzip - </dev/fd/63 >output.gz &; program /dev/fd/63.

share|improve this answer

Originally, I thought, sure that's easy: just mount a loopback device with a compressed filesystem where the program expects to write to. Unfortunately, upon searching, I found that there aren't many read/write filesystems, and what's there (jffs2) can't be mounted via a loopback device.

I did find FuseCompress which may be what you're looking for, but if you need high reliability, I'd skip it.

Another alternative would be to store the file on a USB hard drive, and make a symlink at the location the program writes to. This may be too much of a hassle if you frequently work with the program or if you don't already have a 250GB+ USB drive hanging around.

share|improve this answer
    
Why would you skip FuseCompress? –  Gilles Nov 7 '10 at 16:16
    
Only because it's not well-enough known (at least to me), so it's reliability hasn't been vetted yet. –  Shawn J. Goff Nov 7 '10 at 19:18

If the application doesn't require its input and output to be seekable, pass it /dev/stdout or <(gunzip <data.gz) — see camh's answer and alex's answer.

If the application does require a seekable file, your best bet is a filesystem that implements compression. There are a few unix filesystem implementations that support compression:

  • Through FUSE, which is available on most unices, there are a few compression filesystems. FuseCompress and CompFUSEd are two options, as well as the various archive filesystems.
  • Zfs supports everything including the kitchen sink and compression. It's the native filesystem under Solaris these days (that where it came from). It's available through FUSE at least on Linux. FreeBSD and NetBSD have at least partial native implementations of zfs.
  • On Linux, there are patches floating around to implement compression on ext2 and derivatives. I don't know how reliable they are or how compatible they are with ext3 and ext4.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.