1

Here is a simple script which is curling https://unix.stackexchange.com/ and storing the result into an array, which is working fine.

#!/usr/local/bin/bash
[ -f pgtoscrap ] && { rm pgtoscrap; };
curl -o pgtoscrap https://unix.stackexchange.com/;
declare -a arr;
fileName="pgtoscrap";

exec 10<&0
exec < $fileName
let count=0
while read LINE; do
    arr[$count]=$LINE
    ((count++))
done
exec 0<10 10<&-

But, each time I run this script; I get some error for the wrong file descriptor.

./shcrap
./shcrap: line 14: 10: No such file or directory

I think I don't understand well how to use exec command in a loop correctly. Can someone explain?

-- Update after implementing mapfile for Bash 4 it became much simpler --

#!/usr/local/bin/bash
## Pass a parameter as e.g. ./linkscrapping.bash https://unix.stackexchange.com/
mapfile -t arr < <(curl -s $1); ## Doing exec stuff with process substitution
regex="<a[[:print:]]*<\/a>"; ELEMENTS=${#arr[@]}; firstline=0;
for((i=0;i<$ELEMENTS;i++)); do
    if [[ ${arr[${i}]} =~ $regex ]]; then
    [[ $firstline<1 ]] &&
        { echo ${BASH_REMATCH[0]} > scrapped; let firstline=$firstline+1; } ||
        { echo ${BASH_REMATCH[0]} >> scrapped; }
    fi
done
pg2scrap="scrapped"; mapfile -t arr2 < <(cat $pg2scrap);
regex="href=[\"\'][0-9a-zA-Z\:\/\.]+"; ELEMENTS2=${#arr2[@]}; line2=0
for ((i=0;i<$ELEMENTS2;i++)); do
    if [[ ${arr2[${i}]} =~ $regex ]]; then
    [[ $line2<1 ]] &&
        { echo ${BASH_REMATCH[0]#href=\"} > links; (( line2++ )); } ||
        { echo ${BASH_REMATCH[0]#href=\"} >> links; }
    fi
done; cat links;
  • 1
    Just for the record, you don't need that last exec at the end at all, unless you are sourcing the script or reusing this code in a function, because the file descriptor redirections don't effect the calling process from which the file descriptors are inherited, and because the file descriptors will be automatically closed when the shell process exits. – mtraceur May 4 at 0:26
6

It surely has to do with how you close the file descriptor that you had opened earlier for stdin. Using the below should just be fine

exec 10<&- 

When you do 0<10, you instruct the shell to look for and to slurp in the contents of a file named 10 in your current directory which makes no sense in this context.

In bash you can also use an alternate form exec 10>&- which achieves the same purpose of closing the descriptor.

But that said, you don't need to use exec on random file descriptor and read your input, you can just read in your input with the process substitution technique in bash of form < <() as

while IFS= read -r line; do
    arr["$count"]="$line"
    ((count++))
done< <(pgtoscrap)
  • 2
    In Bash 4 you could also use readarray arr < pgtoscrap instead of writing a loop. Or even curl -s https://unix.stackexchange.com/ | readarray arr to directly read the curl output into an array. – BlackJack May 2 at 16:15
  • 1
    OP presumably intended to restore FD 0 by writing exec 0<&10 10<&-, and accidentally omitted the & ending up with 0<10. – mtraceur May 4 at 0:24
4

exec 10<&0 clones file descriptor number 0 to number 10, effectively saving the original so you can replace the file on fd 0 on the next line. To undo that, you'd need to reverse the numbers, to clone number 10 to number 0, exec 0<&10 (and then close fd 10 with exec 10<&-).

On the other hand, exec 0<10 without the ampersand is just a redirection with a filename 10. Since you don't have such a file, you get an error.


That said, you don't need to use exec to temporarily set up a redirection for the while loop. Compound commands can take redirections too, like so:

while read LINE; do
    ...
done < "$filename"

If you want to read full lines as they are, without whitespace or backslashes affecting the data, you need to unset IFS for read, and use read -r. Also, if you're appending to an array, you don't need to manually keep up with the indexes, you can just use += to append to the array directly:

arr=()   # declares it an array and clears it, not strictly necessary though
while IFS= read -r line; do
    arr+=("$line")
done < "$filename"

Or use mapfile (readarray) instead of a manual loop like @BlackJack mentions in comments:

mapfile -t arr < "$filename"

Or even without a temporary file at all:

#/bin/bash
mapfile -t arr < <(curl -s https://unix.stackexchange.com/)

(Without -t, mapfile leaves the line terminators in place.)

  • Now, I understand the FD, so it's almost the same as a closing case, if statement esac, fi so on and so forth. I was foolishly thinking for 10<&0 it should be 0&<01. I think it might be better to use single digit FD to avoid confusion? so 9<&0 will end with 0&<9 9<&- Overall, using mapfile -t arr is far better approach I guess for Bash 4. – Rakib Fiha May 4 at 1:55
  • @RakibFiha, no, it'd still be 0<&9 9<&-. It's just the numbers that get switched, not the other characters. The action that happens is the same regardless of which number is cloned to which. Also it doesn't matter if there's more than one digit, they're just numbers. – ilkkachu May 4 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.