0

Slightly unusual usecase: I have a download from firefox which is actually a stream of data. I want to process it in (near) real-time, without saving the data (it's high volume and would fill my drive quickly). Is there a way I can convert the file being downloaded to a pipe/socket of some sort and just read from it directly programmatically somehow?

Things I tried so far:

  • Having a script move the file to a different location periodically, read the moved file into a buffer, and repeat to read the stream piece by piece. Unfortunately doing so stops the download (firefox keeps downloading, but there is no new .part file that appears after moving the original one).

  • I found that if I just wrote to the file (echo '' > file.part), the download does continue. So I could read the contents, overwrite the file (to avoid filling up space), and repeat indefinitely - however between reading the file and overwriting it, a few more bytes get written by firefox in the meantime so the stream gets corrupted.

Ideas?

Some details as per questions in comments:

The download is a capture (pcap) file whose download is initiated by a Fritzbox web UI. I can't curl it because it's the fritzbox that somehow initiates the download, there's no url I can use in curl. Not sure how the download is implemented, as it appears as a "normal" download in Firefox but has no set size and it keeps adding data until I press "stop" in the web ui.

2
  • 1
    How is the download initiated? Jan 25, 2023 at 8:03
  • 2
    Does it have to be Firefox? Piping from curl should do what you want
    – Panki
    Jan 25, 2023 at 9:06

1 Answer 1

1

If your filesystem supports the fallocate() system call, like ext4, then you can use it to zero out parts of the file you have read, and the space used for them will become zero too, as the file will become sparse.

For example:

file=/tmp/somefile
length=4096
offset=0

tail -n +1 -f "$file" |
while timeout 5 dd status=none count=1 bs=$length
do  fallocate --punch-hole -o $offset -l $length $file
    let  offset=offset+length
done |
myprogram

This shell script uses tail to read the file from block 0, pipes the data to dd to read a single block and pipe it on to your program. It uses the utility fallocate to then zero out that block.

An ls -ls $file will show a growing size for the file written by firefox, but the first number, the number of blocks used, will be close to zero, assuming the loop manages to keep up. If necessary increase the length, keeping it a multiple of the filesystem block size. The timeout is to detect final end-of-file, and may need to be extended.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .