0

I want to get particular string from the files downloaded using curl and redirect the downloaded file content to another shell program. Here is the code :

curl "http://file-list-url" -s | awk 'BEGIN {c=0} $0 ~ "FILE:" {urlpos=index($0,"URI=")+5;  fileurl=substr($0, urlpos, 150-urlpos-2);  print fileurl ; curl -s -H  "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36"   fileurl | hexdump -C | head -1  ; print filecontent }' 

After getting each fileurl, I want to download it using curl , get its content, pass to hexdump then to head and print the final output of head.

0

Instead of trying to run curl etc. inside of awk, use awk as a processing step in the pipeline and run the other parts of the pipeline afterwards:

useragent='User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36

curl -s "http://file-list-url" |
awk '/FILE:/ { urlpos=index($0,"URI=")+5; print substr($0, urlpos, 150-urlpos-2) }' |
xargs -I {} curl -s -H  "$useragent" {} |
hexdump -C | head -n 1

Or, if you want the hexdump and head to act on each of the secondary calls to curl:

curl -s "http://file-list-url" |
awk '/FILE:/ { urlpos=index($0,"URI=")+5; print substr($0, urlpos, 150-urlpos-2) }' |
xargs -I {} sh -c '
    curl -s -H  "$useragent" "$1" |
    hexdump -C | head -n 1' sh {}

Here we use xargs to execute a small shell script that runs curl using its only command line argument and that passes the resulting output through hexdump and head.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.