I have 44 TB worth of gzipped files that are 1.5 GB each when compressed or 2.0 GB when uncompressed. I have a tool that can read only uncompressed files. I would like to avoid the overhead of uncompressing the entire file and writing the result to disk, since I might only need access to a small part of the file. The tool cannot read gzip-compressed data on the fly.
Is there a way to create a (read-only) file-like object that has all the features of a file from the application point of view, but rather than storing any data to disk, calculates data on-the-fly (possibly caching in memory)? I could try a named pipe, but this doesn't allow seeking. Uncompressing to tmpfs is somewhat (10–20%) faster than uncompressing to disk, but still requires uncompressing the entire file. I do not need any write access.
The machine has 2 TB RAM and runs on Red Hat Enterprise Linux Server release 6.7. The data are on a panfs filesystem. Other filesystems are a small (20 GB) tmpfs and and some scratch space (15 TB shared with others). I do not have system administrator privileges.