Is there a way to redirect a file that is written

Question

I have an application which I can not change, that writes to a big 250 GB plain text file. After compressing with GZ it is only 30 GB.

This application has no option of compressing it's output, and it can only write to a file name (not to stdout).

Is there a way that I can set up for the output to be compressed immediately, without first storing the 250 GB file on my disk?

I also need the other way around, to fool the application so that it reads a plaintext file which in reality is compressed.

Answer 1

You may have success using /dev/stdout as the filename and piping the output of your application to gzip.

/dev/stdout is a symlink to /proc/self/fd/1.

Similarly, you may be able to use /dev/stdin as a filename and pipe the output of gzip to the application.

I say may, because the application may be expecting a seekable file that it writes to (reads from), but /dev/std{in,out} will not be seekable. If this is the case then you are probably lost. You will need to use a seekable file as the target for the application.

Answer 2

• Use can use mkfifo to setup a named pipe in which the program will write, while you gzip from that pipe into final destination.

• If you may pass the output filename to the program on command line, then this (probably bash-specific) trick should also work: program >(gzip - >output.gz) as this will be translated by the shell into something like gzip - </dev/fd/63 >output.gz &; program /dev/fd/63.

Answer 3

Originally, I thought, sure that's easy: just mount a loopback device with a compressed filesystem where the program expects to write to. Unfortunately, upon searching, I found that there aren't many read/write filesystems, and what's there (jffs2) can't be mounted via a loopback device.

I did find FuseCompress which may be what you're looking for, but if you need high reliability, I'd skip it.

Another alternative would be to store the file on a USB hard drive, and make a symlink at the location the program writes to. This may be too much of a hassle if you frequently work with the program or if you don't already have a 250GB+ USB drive hanging around.

Answer 4

If the application doesn't require its input and output to be seekable, pass it /dev/stdout or <(gunzip <data.gz) — see camh's answer and alex's answer.

If the application does require a seekable file, your best bet is a filesystem that implements compression. There are a few unix filesystem implementations that support compression:

• Through FUSE, which is available on most unices, there are a few compression filesystems. FuseCompress and CompFUSEd are two options, as well as the various archive filesystems.
• Zfs supports everything including the kitchen sink and compression. It's the native filesystem under Solaris these days (that where it came from). It's available through FUSE at least on Linux. FreeBSD and NetBSD have at least partial native implementations of zfs.
• On Linux, there are patches floating around to implement compression on ext2 and derivatives. I don't know how reliable they are or how compatible they are with ext3 and ext4.