0

Problem statement:

  1. Read a file line by line
  2. Execute a command on each file
  3. Print the line and output of command in another file

Input file (urls.txt):

www.google.com
www.yahoo.com
www.bing.com

Expected output file (output.csv):

www.google.com,200
www.yahoo.com,500
www.bing.com,404

Script I have so far

while IFS= read -r line
do
    curl -LI "$line" -o /dev/null -w '%{http_code}\n' -s
 >> output.csv
done < urls.txt

not an expert in bash scripting. just a beginner. any help is much appreciated. Thank you. :)

  • So what is your question, exactly? – steeldriver Mar 21 at 18:04
  • What doesn't work? – roaima Mar 21 at 18:05
  • I want the output mentioned in Expected output file. Currently output.csv would only contain the http codes i.e. only the output of the command. Output I want is for 'www.google.com','200' is the response – Jignesh144 Mar 21 at 18:08
  • So please edit your question to tell us that. You've shown us what you've tried - that's great. But what does it do that you don't want it to do, and what doesn't it do that you do want it to do? – roaima Mar 21 at 20:30
1

By using echo and command substitution formatting of each line can be done like that:

while IFS= read -r line
do
    echo "$line,$(curl -LI "$line" -o /dev/null -w '%{http_code}\n' -s)" >> output.csv
done < urls.txt
| improve this answer | |
1

Consider using %{url_effective} in the -w output fields:

awk -v OFS== -v out=/dev/null \
    '!/^(#|[[:blank:]]*$)/{print "url", $0; print "out", out}' urls.txt |

curl -K- -sSLI -w '%{num_redirects},%{url_effective},%{http_code}\n' >> output.csv
  • Because -L is used to follow redirects, there may be some discrepencies between the urls from urls.txt and %{url_effective}. If the first url gives a 3xx response, the first column from the output above will be greater than zero, the second column will contain last url followed (not the initial url from urls.txt)
  • Above, instead of looping over the file and calling curl on each line, awk is used to turn urls.txt into a "config" file for curl which is then piped to curl. The formatting for this is specific to curl – for more info, search for -K in man 1 curl

e.g.

$ cat urls.txt
https://unix.stackexchange.com/
#www.example.com
https://unix.stackexchange.com/question
http://unix.stackexchange.com/

$ awk -v OFS== '!/^(#|[[:blank:]]*$)/{print "url",$0; print "out","/dev/null"}' \
urls.txt | curl -K- -sSLI -w '%{num_redirects},%{url_effective},%{http_code}\n'
0,https://unix.stackexchange.com/,200
0,https://unix.stackexchange.com/question,404
1,https://unix.stackexchange.com/,200
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.