0

Say that

https://example.nosuchtld
https://example.net
https://example.org
https://example.willfail

is the content of urls.txt. I want to run <command> <url> for every URL/line of urls.txt —where <command> is, let's say, curl; so,

cat urls.txt | xargs -n1 curl

or

<urls.txt xargs -n1 curl

for instance. I want every URL/line which was unsuccessfully curled (so, the first and last ones) to

  1. be removed from urls.txt; and
  2. be appended to another file —let's say nope.txt— to be created if it doesn't already exist

leaving urls.txt as

https://example.net
https://example.org

and nope.txt as

https://example.nosuchtld
https://example.willfail

I know that the exit status of every command run by the shell is made available via the variable $?, that 0 represents successful execution of a command, and that all other integers represent failure. I'm unsure, though, of how to construct a composite command that incorporates this, deletes lines from the file being read from, and appends 'em to a different file.

2 Answers 2

1

Using bash, you can loop for the urls and test the curl command, --fail option of curl seems to be good for use inside scripts, see: How to check whether a command such as curl completed without error

So it could be like this:

while read -r url; do
    curl -f "$url" && outputfile='success.txt' || outputfile='nope.txt'
    printf "%s\n" "$url" >> "$outputfile"
done < urls.txt

and overwrite your file with the successful urls.

mv success.txt urls.txt

Or use mapfile to put the lines into an array:

mapfile -t urls < urls.txt

for url in "${urls[@]}"; do
    curl -f "$url" && outputfile='success.txt' || outputfile='nope.txt'
    printf "%s\n" "$url" >> "$outputfile"
done

Note that urls have no spaces. If you need to execute a command where the argument would be a whole line, with any characters, then this post is useful on how to read every line: Understanding "IFS= read -r line"

3
  • Thanks! I had in mind leveraging the exit status because, despite having happened to select curl as an example to concertize my question, I'm really looking to understand a solution or solutions that'd be likely to work with any command including those that lack a --fail option. Consider, for instance, the command line app nb: it has no such flag. What could be done, then, when processing a URL file with it? Thoughts? May 15 at 1:27
  • Ahh, gotcha. Okay 🤙 May 15 at 1:35
  • 1
    🤔 I see, I see… the success exit status is being implicitly checked for at the && logical operator, verdad? May 15 at 2:10
0

With zsh, you could do:

while IFS= read -ru3 url; do
  curl -- $url
  print -ru $(( $? ? 4 : 5 )) -- $url
done 3< urls 4> bad 5> good

That way, the bad and good files are opened only once and only if urls itself can be opened for reading.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.