1

A web service writes its output in a text file.

I can fetch that log via:

curl -u user:password -k https://log.my-app.com/log.2016-04-04.txt

Yet this always only fetches a current "full" file even though it is still being written to.

I want to fetch the remote txt file as if it was a stream and have it constantly refetch itself after one minute but only show the updated content.

Is something like this possible using curl?

  • I bet there is a recipe using rsync that would suit this use-case. – glenn jackman Apr 4 '16 at 13:58
1

You're looking for the -C --continue-at and -f --fail arguments.

curl -f -# -u user:password -k -C - -O https://log.my-app.com/log.2016-04-04.txt

Without -f the output file will be appended with html from the 416 (invalid range) error.

1

You could use curl -C - in a loop to get it to fetch only what has been added, and use tail -f on the local file to watch for the new data. The remote server has to allow for this sort of access.

If it does not, use N=$(stat -c %s file) on the local file to find its size, then pipe the curl through tail -c +N | tee -a file to get the new stuff to the file and stdout.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.