1

The command below cats a file full of urls and feeds it into gnu parallel. I would like the output in curlresponse.txt to have also the original request url. How can it be done?

cat urls.txt  | parallel -j 5  curl -IL -k {} > curlresponse.txt

so sample of final output (after all redirects if any) is like this :

OriginalUrl
HTTP/1.1 200
Server: nginx
Date: Mon, 18 Mar 2019 07:35:11 GMT
Content-Type: text/html
Content-Length: 188
Connection: keep-alive

Sometimes the url might be redirected so the final http response should be the one having the original request url.

0

Strictly:

cat urls.txt  |
  parallel -j 5 echo {}';'curl -IL -k {} '|' perl -ne '/^HTTP.* 200/ .. 0 and print' > curlresponse.txt

But for situations like this I usually prefer --tag which will prepend every line with the input:

cat urls.txt  |
  parallel -j 5 --tag curl -IL -k {} '|' perl -ne '/^HTTP.* 200/ .. 0 and print' > curlresponse.txt

It often makes further processing easier.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.