I am using the following code to download streaming financial data

curl -s -H "Content-Type: application/json" -H "Authorization: Bearer XXX..." "https://...=EUR_USD" | jq --raw-output '[.time, .bids[0].price, .asks[0].price] | @csv'

which is successfully streaming in the terminal thus


However, what I would really like is to further process this, using sed, to get


and then append this to a file, let's call it "output," but when I try piping to sed I no longer see the streaming data although I'm sure my sed syntax is correct as I've checked it against a static test file.

So my question is: how do I further pipe to sed and then append to the file "output?"

  • 3
    Its probably just being buffered.
    – muru
    May 5 '20 at 10:45
  • @muru How would I check if this is the case? May 5 '20 at 11:25
  • 1
    Use stdbuf, unbuffer or similar (unix.stackexchange.com/a/25378/70524) and see if you get output
    – muru
    May 5 '20 at 11:27
  • 1
    @babelproofreader: jq itself has an option to unbuffer with --unbuffered flag
    – Inian
    May 5 '20 at 11:35
  • A combination of stdbuf, --unbuffered flag in jq and -u flag in sed accomplishes what I want. Thanks to muru and Inian. May 5 '20 at 12:53

In response to Glenn Jackman's comment requesting that I post an answer, the syntax that I find works for me is:

stdbuf -oL -eL curl -s -H "Content-Type: application/json" -H "Authorization: Bearer XXX..." "https://...=EUR_USD" | jq --raw-output --unbuffered '[.time, .bids[0].price, .asks[0].price] | @csv' | sed -u 's/["Z]//g' | sed -u 's/[-T:]/,/g' >> ~/path/to/append/to/output

which appends to the file in the desired format described in my original question. I can also look at the "output" file from time to time and see that the streaming data is continually being appended.

Not the answer you're looking for? Browse other questions tagged or ask your own question.