consider a simple server API, in which you can consume data in iterations, using a "stateless token", which might look like: rQAAMTQ2MzU4MDA1MjgxM3x8fC9wZXJtaWQub3JnfHx8, so:

curl -v "http://ws.foo.bar/_consume?position=rQAAMTQ2MzU4MDA1MjgxM3x8fC9wZXJtaWQub3JnfHx8"
* Connected to ws.foo.bar ( port 80 (#0)
> GET /_consume?position=rQAAMTQ2MzU4MDA1MjgxM3x8fC9wZXJtaWQub3JnfHx8 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: ws.foo.bar
> Accept: */*
< HTTP/1.1 200 OK
... many irrelevant headers
< Connection: close
< Date: Wed, 01 Jun 2016 15:02:42 GMT
{ datadatadatadatadatadatadata... }

output the data to stdout, and headers (along with other verbose info), goes to stderr. one of these headers is X-POSITION, storing the next position I should supply as query parameter for the next chunk of data. I tried to come up with a one-liner I could just hit up button and execute again to retrieve the next chunk of data

I tried the following (assume initial position: 1foo2BAR3baz4QUUX5blah):

$ POS=1foo2BAR3baz4QUUX5blah
$ POS=$( curl -v "http://ws.foo.bar/_consume?position=$POS" 2>&1 | grep X-POSITION | awk '{print $3}' )

which obviously won't print to screen the data since it's being swallowed by grep, but also, POS only changes once, and is not usable again. curl complains on the second invocation:

* Illegal characters found in URL
* Closing connection -1
curl: (3) Illegal characters found in URL

though it seems that POS got the new value:

$ echo $POS

I suspect maybe EOF?

in any case, even if I solve this, I still want to not ignore stdout. tried this solution:

$ POS=`( curl -v "http://ws.foo.bar/_consume?position=$POS" 3>&1 1>&2- 2>&3- ) | grep X-POSITION | awk '{print $3}'`

but it seems not to work either.


The "faulty" character in the $POS parameter is EOL. HTTP uses CRLF as line ending. That gives for example:

< X-POSITION: xxxxxxxx\r\n

That gives field 3 in awk to be xxxxxxxx\r.

By using print in awk you also re-introduce the last new-line, \n, but as your expression is not quoted, this is lost.

You can see this by doing something like:

curl -v "http://ws.foo.bar/_consume?position=$POS" 2>&1 | cat -v

Those ^Ms at end of lines indicate \r.


printf "%s" "$pos" | xxd
00000000: 7251 4141 4d54 5132 4d7a 5534 4d44 4131  rQAAMTQ2MzU4MDA1
00000010: 4d6a 6778 4d33 7838 6643 3977 5a58 4a74  MjgxM3x8fC9wZXJt
00000020: 6157 5175 6233 4a6e 6648 7838 0d         aWQub3JnfHx8.

That last 0d is a CR. (Optionally do ascii on command prompt.)

There is also no need to mix grep and awk as awk mathces fine on its own.

To get on the right track this could be a start:

pos=$(curl -v "http://foo.x/pos=$pos" 2>&1 | awk -vRS="\r\n" '/^< X-POSITION:/{printf "%s", $3}')

Here one set the RS, or record-separator, in awk to CRLF,

... but this would only give you the token, not the content.

Assuming you do not need to actually print the content to screen, but save it to file one way could be:

pos=$(curl -sD - -o "$pos.out" "http://foo.x/?position=$pos" | awk -vRS="\r\n" '/^X-POSITION:/{printf "%s", $2}')

Herre we redirect header data to stdout by -D - and save content to file by -o "$pos.out".

Another advantage with this is that you only parse the header data.

  • thanks a lot! my awk-foo is weak, didn't know awk could be used like that, thanks for the tip! I actually just wanted to print the data to the screen, so I ended up with: POS=$( ( curl -v "http://ws.foo.bar/_consume?position=$POS" 3>&1 1>&2- 2>&3- ) | awk -vRS="\r\n" '/^< X-POSITION:/{printf "%s", $3}' ), and it works great! – gilad hoch Jun 1 '16 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.