How to store endless stream to file?

Question

• Let's have any stream of binary data (like /dev/random, /dev/zero, etc.).
• Let's have a file which size can be N at maximum.
• Let's the N is in order of gigabytes.

Is there any elegant method/technology-to-use/hack in linux, to achieve such file continuously written is at maximum N bytes long and always contains only last written data (sequentially)? This means no large moves (file to file, memory to file) just little tweaking with the last/first data blocks.

The trick I'm looking for is the file beginning moving forward and effectively forgetting any too-old content (which would increase the file size over N) - content rotation.

The desired principle could be expressed as:

inp = fopen(stream, "r");
out = fopen(file, "wN"); // "Special open" with maximal size of N

while (is_reading)
{        
    if (rd = fread(buff, block_size, 1, inp))
    {
        fwrite(buff, rd, 1, out); // Old content forgotten
        fflush(out);              // Allow others to instantly see the current content
    }
}

fclose(inp);
fclose(out);

Answer 1

On Linux, you can use fallocate() to deallocate the data at the beginning of the file.

Deallocating file space

Specifying the FALLOC_FL_PUNCH_HOLE flag (available since Linux 2.6.38) in mode deallocates space (i.e., creates a hole) in the byte range starting at offset and continuing for len bytes. Within the specified range, partial filesystem blocks are zeroed, and whole filesystem blocks are removed from the file. After a successful call, subsequent reads from this range will return zeroes.

...

That won't change the file size reported by ls or stat or similar, but it will reduce the actual disk usage as the file will be sparse. Trying to read from the "holes" in the file will still succeed and will return 0-filled bytes to the reading process.

Something like this:

size_t maxSize = 512UL * 1024UL * 1024UL;
char buffer[ 8 * 1024 ];
struct stat sb;
int in = open( inputFile, O_RDONLY );
int out = open( outputFile, O_WRONLY | O_CREAT | O_APPEND, 0644 );
fstat( out, &sb );
size_t fileSize = sb.st_size;

for ( ;; )
{
    ssize_t bytesRead = read( in, buffer, sizeof( buffer ) );
    if ( bytesRead < 0 )
    {
        break;
    }

    ssize_t bytesWritten = write( out, buffer, bytesRead );
    if ( bytesWritten < 0 )
    {
        break;
    }

    fileSize += bytesWritten;

    if ( fileSize > maxSize )
    {
        fsync( out );
        off_t endOfHole = fileSize - maxSize;
        fallocate( out, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
            0UL, endOfHole );
    }
}

That's only supported on XFS, BTRFS, EXT4, and tmpfs.

It also needs a lot more error checking, and may not even compile as-is. It's also very inefficient as once the max size is reached, it will call fallocate() for every read()/write() cycle, and it will punch the "hole" from the beginning of the file every time.

There's also no point in using buffered fread()/fwrite() for this IO pattern. Just read()/write() large enough chunks.