What's a clean way to run a specific command C for each line L of a given file F and then move every L where C(L) ran unsuccesfully?

Question

Say that

https://example.nosuchtld
https://example.net
https://example.org
https://example.willfail

is the content of urls.txt. I want to run <command> <url> for every URL/line of urls.txt —where <command> is, let's say, curl; so,

cat urls.txt | xargs -n1 curl

or

<urls.txt xargs -n1 curl

for instance. I want every URL/line which was unsuccessfully curled (so, the first and last ones) to

1. be removed from urls.txt; and
2. be appended to another file —let's say nope.txt— to be created if it doesn't already exist

leaving urls.txt as

https://example.net
https://example.org

and nope.txt as

https://example.nosuchtld
https://example.willfail

I know that the exit status of every command run by the shell is made available via the variable $?, that 0 represents successful execution of a command, and that all other integers represent failure. I'm unsure, though, of how to construct a composite command that incorporates this, deletes lines from the file being read from, and appends 'em to a different file.

Answer 1

Using bash, you can loop for the urls and test the curl command, --fail option of curl seems to be good for use inside scripts, see: How to check whether a command such as curl completed without error

So it could be like this:

while read -r url; do
    curl -f "$url" && outputfile='success.txt' || outputfile='nope.txt'
    printf "%s\n" "$url" >> "$outputfile"
done < urls.txt

and overwrite your file with the successful urls.

mv success.txt urls.txt

---

Or use mapfile to put the lines into an array:

mapfile -t urls < urls.txt

for url in "${urls[@]}"; do
    curl -f "$url" && outputfile='success.txt' || outputfile='nope.txt'
    printf "%s\n" "$url" >> "$outputfile"
done

---

Note that urls have no spaces. If you need to execute a command where the argument would be a whole line, with any characters, then this post is useful on how to read every line: Understanding "IFS= read -r line"

Answer 2

With zsh, you could do:

while IFS= read -ru3 url; do
  curl -- $url
  print -ru $(( $? ? 4 : 5 )) -- $url
done 3< urls 4> bad 5> good

That way, the bad and good files are opened only once and only if urls itself can be opened for reading.