I am trying to archive a folder using tar and then use curl to stream upload it without having to take up any space in my hard drive or memory. I have

tar cf - "FOLDERNAME" | curl -v -T - \
    -X PUT \
    -H "content-transfer-encoding: binary" \
    --output here.txt \

which appears to work and upload the file. However, I am wondering if it is truly streaming or perhaps is it first creating the tar file, writing it to cache or memory, then uploading it? How can I tell if it is writing it to a stream, vs creating the tar file first, saving it somewhere (in memory), then uploading?

  • Unix pipes use an in-memory buffer so this will take some space in memory. The exact amount varies depending on the variant and version of Unix used, which you didn't tell us, and sometimes on configuration options as well. However it is rarely more than a few kilobtyes, and tars are typically fairly large, so probably the entire tar is not present in the buffer at any point in time. In addition both of the processes will have some data in their private memory. PS: HTTP, although partly based on concepts taken from MIME, does not use the content-transfer-encoding header, and never has. – dave_thompson_085 Oct 11 '20 at 0:51
  • sudo lsof -c tar;sudo lsof -c curl while it's running. man lsof. – waltinator Oct 11 '20 at 2:09
  • @waltinator Thank you. I have read over man lsof. It appears I want to look at SIZE/OFF. I noticed it changes as a function of time, for each time I am calling it. Is the value at SIZE/OFF (in bytes) the usage at any given time? Thanks! – user321627 Oct 12 '20 at 1:31
  • @dave_thompson_085 Thank you, I am using CentOS and Solaris. The Solaris one is Sun solaris 5.11 while for CentOS it is CentOS Linux 7.6.1810. For both OS's I have tar (GNU tar) version 1.32 and curl 7.45.0. – user321627 Oct 12 '20 at 3:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.