0

Problem statement:

  1. Read a file line by line
  2. Execute a command on each file
  3. Print the line and output of command in another file

Input file (urls.txt):

www.google.com
www.yahoo.com
www.bing.com

Expected output file (output.csv):

www.google.com,200
www.yahoo.com,500
www.bing.com,404

Script I have so far

while IFS= read -r line
do
    curl -LI "$line" -o /dev/null -w '%{http_code}\n' -s
 >> output.csv
done < urls.txt

not an expert in bash scripting. just a beginner. any help is much appreciated. Thank you. :)

4
  • So what is your question, exactly? Mar 21, 2020 at 18:04
  • What doesn't work? Mar 21, 2020 at 18:05
  • I want the output mentioned in Expected output file. Currently output.csv would only contain the http codes i.e. only the output of the command. Output I want is for 'www.google.com','200' is the response
    – Jignesh144
    Mar 21, 2020 at 18:08
  • So please edit your question to tell us that. You've shown us what you've tried - that's great. But what does it do that you don't want it to do, and what doesn't it do that you do want it to do? Mar 21, 2020 at 20:30

2 Answers 2

1

By using echo and command substitution formatting of each line can be done like that:

while IFS= read -r line
do
    echo "$line,$(curl -LI "$line" -o /dev/null -w '%{http_code}\n' -s)" >> output.csv
done < urls.txt
1

Consider using %{url_effective} in the -w output fields:

awk -v OFS== -v out=/dev/null \
    '!/^(#|[[:blank:]]*$)/{print "url", $0; print "out", out}' urls.txt |

curl -K- -sSLI -w '%{num_redirects},%{url_effective},%{http_code}\n' >> output.csv
  • Because -L is used to follow redirects, there may be some discrepencies between the urls from urls.txt and %{url_effective}. If the first url gives a 3xx response, the first column from the output above will be greater than zero, the second column will contain last url followed (not the initial url from urls.txt)
  • Above, instead of looping over the file and calling curl on each line, awk is used to turn urls.txt into a "config" file for curl which is then piped to curl. The formatting for this is specific to curl – for more info, search for -K in man 1 curl

e.g.

$ cat urls.txt
https://unix.stackexchange.com/
#www.example.com
https://unix.stackexchange.com/question
http://unix.stackexchange.com/

$ awk -v OFS== '!/^(#|[[:blank:]]*$)/{print "url",$0; print "out","/dev/null"}' \
urls.txt | curl -K- -sSLI -w '%{num_redirects},%{url_effective},%{http_code}\n'
0,https://unix.stackexchange.com/,200
0,https://unix.stackexchange.com/question,404
1,https://unix.stackexchange.com/,200

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .