18

I'm trying to search for files using find, and put those files into a Bash array so that I can do other operations on them (e.g. ls or grep them). But I can't figure out why readarray isn't reading the find output as it's piped into it.

Say I have two files in the current directory, file1.txt and file2.txt. So the find output is as follows:

$ find . -name "file*"
./file1.txt
./file2.txt

So I want to pipe that into an array whose two elements are the strings "./file1.txt" and "./file2.txt" (without quotes, obviously).

I've tried this, among a few other things:

$ declare -a FILES
$ find . -name "file*" | readarray FILES
$ echo "${FILES[@]}"; echo "${#FILES[@]}"

0

As you can see from the echo output, my array is empty.

So what exactly am I doing wrong here? Why is readarray not reading find's output as its standard input and putting those strings into the array?

26

When using a pipeline, bash runs the commands in subshells¹. Therefore, the array is populated, but in a subshell, so the parent shell has no access to it. You also likely want the -t option so as not to store that line delimiters in the array members as they are not part of the file names.

Use process substitution:

readarray -t FILES < <(find .)

Note that it doesn't work for files with newlines in their paths. Unless you can guarantee if won't be the case, you'd want to use NUL delimited records instead of newline delimited ones:

readarray -td '' < <(find . -print0)

(the -d option was added in bash 4.4)


¹ except for the last pipe component when using the lastpipe option, but that's only for non-interactive invocations of bash.

2
  • 3
    In order to support newlines, this is sufficient: readarray -d '' < <(find your_args -print0) – VasyaNovikov Jan 18 '18 at 13:40
  • @VasyaNovikov Support for -d in readarray seems to be relatively recent. (Like the Redhat on my workplace doesn't know it yet.) – ddekany May 26 at 10:49
7

The correct solution is:

unset a; declare -a a
while IFS= read -r -u3 -d $'\0' file; do
    a+=( "$file" )        # or however you want to process each file
done 3< <(find /tmp -type f -print0)

That's similar to what Greg's BashFAQ 020 explains in detail and this answer covers.

Has no problem with odd named files (that contain no NUL in the name), with spaces or new lines. And the result is set in an array, which makes it useful for further processing.

4
  • Great, this is a better solution to the problem I was trying to solve in the first place. +1 as soon as my rep reaches 15 :) – villapx Feb 17 '16 at 19:12
  • Why does this use 3 instead of stdin? Also I believe -d '' is more usual (the 0 will terminate the string anyway, so it's in effect a 0-length string.) Also I guess the first line could replaced with just a=(). – ddekany May 26 at 10:48
  • 1
    @ddekany, using fd 3 means you leave standard I/O 0, 1, 2 fds untouched so they can be used by commands within the loop. That's good practice. – Stéphane Chazelas May 26 at 14:01
  • @ddekany, using a=() instead of unset a; declare -a a wouldn't work properly if a was previously declared as an associative array which may be why the OP chose that approach. – Stéphane Chazelas May 26 at 14:02
3

readarray can also read from stdin

readarray FILES <<< "$(find . -name "file*")"; echo "${#FILES[@]}"
2
  • 2
    This doesn't work with find -print0 for protecting against "unexpected" file names. – roaima Mar 11 '19 at 14:14
  • readarray always reads from stdin unless you use -u to specify a different fd. The difference here is that you're using <<< to set its stdin instead of | or <. – Stéphane Chazelas May 26 at 14:04
1

With shopt -s lastpipe the solution shown in the original question works, as then the part after the last | runs in the same process as most of the script:

shopt -s lastpipe

# This is just the code form the original question.
# Of course it could be done with -print0, etc.
declare -a FILES
find . -name "name*" | readarray -t FILES
echo find and readarray exit status: "${PIPESTATUS[@]}"
echo "${FILES[@]}"
echo "${#FILES[@]}"

This however is only practical if lastpipe can be made a project-wide standard (maybe you already have such common setup for set -u, etc.). Also know that lastpipe only works for non-interactive shells.

The other typical pipe replacement, < <(...) (process substitution), has the problem that it's quite hard to check if the child process has succeeded.

6
  • You can always do readarray -td '' a < <(find . -print 0); readarray_status=$?; wait "$!"; find_status=$? or if readarray -td '' a < <(find . -print0) && wait "$1"; then echo OK; else ...; fi – Stéphane Chazelas May 26 at 14:08
  • @StéphaneChazelas But $? will reflect the exit code of readarray, so even if find failed, that will be still 0. I also tried this to be sure (although without -td '' and -print0, as -td is not yet supported by readarray where I'm). – ddekany May 27 at 7:10
  • in bash process substitutions are started as background jobs which sets $!, so you can use wait "$!" to retreive their status. Try bash -c 'readarray < <(exit 3); wait "$!"; echo "$?"' – Stéphane Chazelas May 27 at 10:15
  • @StéphaneChazelas Bash is more determined to ruin process substitution than that. (: It says "pid X is not a child of this shell", where X is the PID of the subshell. I actually don't know why. – ddekany May 27 at 11:01
  • Indeed I can reproduce with very old versions. Looks like it was fixed in 4.4 – Stéphane Chazelas May 27 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.