0

I have this command line:

curl https://token-api/42 | awk '{ print $0 }'

which produces this output:

{"id":"PX-12345","expiresAtUtc":"2020-07-05T06:55:52","createdUtc":"2019-07-15T06:55:52"}

The awk '{ print $0 }' is there to ensure there is a newline appended at the end of the output.

Now I want to do the same but for a csv list of user ids (users.csv) instead of a hard-coded 42 by using xargs. So I tried this and a number of variations:

awk -F, '{print "curl https://token-api/$0 | awk \'''\''{ print $0 }\'''\''"}' < users.csv | xargs -P1 -I {} sh -c '{}'

This produces the following error:

awk: cmd. line:1: warning: escape sequence `\'' treated as plain `''
awk: cmd. line:1: {
awk: cmd. line:1:  ^ unexpected newline or end of string

If I remove the xargs part, I get the following output:

curl https://token-api/$0 | awk '{ print $0 }'

which is passed to xargs. I guess my problem is with the $0 that is used twice but I have no idea to escape this.

If there is another (better) way to append a newline to the output of curl, that might help too because without that it's working fine. However, then I get a long list of unseparated JSON documents as output like this:

{"id":"PX-12345","expiresAtUtc":"...","createdUtc":"..."}{"id":"PX-12346","expiresAtUtc":"...","createdUtc":"..."}{"id":"PX-12347","expiresAtUtc":"...","createdUtc":"..."}
  • 1
    what do you want to achieve exactly? just append a line to the output of curl? { curl http://foo.com/bar; echo; }. – mosvy Jul 15 '19 at 7:20
  • What does your users.csv look like? – Kusalananda Jul 15 '19 at 7:31
  • @mosvy the 'trick' with echo works, I was making it too complex I guess. Thanks!! – Ronald Wildenberg Jul 15 '19 at 9:00
1

Rather than executing a command for every line of the csv file, it might be easier to take advantage of curls multiple URL feature.

You can specify multiple URLs or parts of URLs by writing part sets within braces as in: http://site.{one,two,three}.com

I'm not sure what your csv file looks like, seems like it's all in a single line? If so you could just cat the file and combine with the curl command.

curl "https://token-api/{$(cat users.csv)}"

Still needs a bit of cleaning up.

curl "https://token-api/{$(cat users.csv)}" 2>/dev/null | sed '1d;s/--_curl_--.*$//'; echo

The sed command deletes the first line since that will just be the first URL, then removes the rest of the URLs, which leaves a useful newline between entries. Finally just echo to end with a newline.

| improve this answer | |
  • As usual, I made my example code too simple. The request is actually a POST request where the data posted comes from the csv file. So I can't use curls multiple URL feature unfortunately. – Ronald Wildenberg Jul 15 '19 at 13:07
  • That makes more sense, I think if it's getting that complex it might be better making a shell script rather than trying to get it all into a single command. Maybe something like while IFS=, read -r USERID DATA1 _ DATA2 _; do; curl "https://token-api/$USERID" -d "data1=$DATA1&data2=$DATA2"; echo; done < users.csv – Dabombber Jul 16 '19 at 3:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.