Looping over directory entries in bash and saving to an array

Question

(See update at the foot of the question).

This is a followup question to "Make directory copies using find".

This question involved manipulating a bunch of directories, This got too complicated to handle in a single command, so I decided to go with an approach which saved the list of directories to a bash array, despite reservations about portability. It seems that the POSIX shell standard (the Unix shell standard, as I understand it) does not have arrays.

The makefile I'm using appears below. This works except for the last step. A summary of what I'm trying to do follows.

As discussed in the earlier question, I want to loop over the directory x86-headers, and collect into a bash array, a list of its top level subdirectories that contain the file C/populate.sh (but may also contain other files). In my test setup, for example there is only one directory in x86-headers which contains the file libc/C/populate.sh, namely libc.

I then perform some operations on these subdirectories. The most important of these is that I make a temporary copy of each directory which looks like libc.tmp_g4zlb. Namely dirname followed by 'tmp_' followed by a 5 digit random string.

So, some questions:

1) As discussed in the earlier question, I'm looping over the directory 'x86-headers'. Here I am still using find. @Gilles indicated in his answer that this was not ideal. He might be right. Problems with find here:

a) The values returned look like ./libc. I don't want a leading ./.

b) The find command I'm using lists all directories. I only want to consider those that contain a file with the relative path of C/populate.sh.

The approach Gilles was using might be better, but I don't understand it. See the gilles target below. I'd like to get the list of the directories and save them to an array.

2) The bit that I'm having problems with is the last step, namely

echo "(progn (require 'parse-ffi) (ccl::parse-standard-ffi-files :$$i'.tmp_'$(RND)))" | \
/usr/lib/ccl-bootstrap/lx86cl -I /usr/lib/ccl-bootstrap/lx86cl.image; done \

The relevant bit is that I'm trying to pass the temporary value libc.tmp_g4zlb to ccl, which is a Common Lisp compiler. Without the substitution, it would look like

echo "(progn (require 'parse-ffi) (ccl::parse-standard-ffi-files :libc.tmp_g4zlb))" | \
/usr/lib/ccl-bootstrap/lx86cl -I /usr/lib/ccl-bootstrap/lx86cl.image; done \

This works. The version using $$i above doesn't. The problem seems to be the leading ./. Without that, it should work. For the record, the error I get is

? > Error: Too many arguments in call to #<Compiled-function CCL::PARSE-STANDARD-FFI-FILES #x488B8406>:
>        3 arguments provided, at most 2 accepted. 
> While executing: CCL::PARSE-STANDARD-FFI-FILES, in process listener(1).

This is, however, not the error that I get when passing ./libc.tmp_g4zlb to the compiler. That error looks like

> Error: Filename "/home/faheem/test/foo/x86-headers/.\\/libc.tmp_g4zlb/C/**/" contains illegal character #\/
> While executing: CCL::%SPLIT-DIR, in process listener(1).

So it is possible there is something else going on.

3) Does the overall approach look reasonable? Please feel free to suggest possible improvements, even if it involves a completely different strategy.

#!/usr/bin/make -f
# -*- makefile -*-

export SHELL=/bin/bash
export RND:=$(shell tr -cd a-z0-9 < /dev/urandom | head -c5)
export CCL_DEFAULT_DIRECTORY=$(CURDIR)

clean:
    rm -rf x86-headers/libc.tmp*

foo: clean
    PATH=$$PATH:$(CURDIR)/ffigen4/bin; \
    echo $$PATH; \
    export CURDIR=$(CURDIR); \
    echo $$CURDIR; \
    array=( $$(cd x86-headers && find . -mindepth 1 -maxdepth 1 -type d) ); \
    cd x86-headers && \
    for i in "$${array[@]}"; do \
    echo $$i; done; \
    for i in "$${array[@]}"; do \
    mkdir -p $$i."tmp_"$(RND)/C; done; \
    for i in "$${array[@]}"; do \
    cp -p $$i/C/populate.sh $$i".tmp_"$(RND)/C; done; \
    for i in "$${array[@]}"; do \
    cd $$i".tmp_"$(RND)/C; ./populate.sh; done; \
    for i in "$${array[@]}"; do \
    echo $$i'.tmp_'$(RND); done; \
    for i in "$${array[@]}"; do \
    echo "(progn (require 'parse-ffi) (ccl::parse-standard-ffi-files :$$i'.tmp_'$(RND)))" | \
    /usr/lib/ccl-bootstrap/lx86cl -I /usr/lib/ccl-bootstrap/lx86cl.image; done; \

gilles:
    cd x86-headers;
    for x in */C/populate.sh; do \
    echo -- "$${x%%/*}$$suffix"; done; \

UPDATE: It is possible that the question (or questions) got lost in all the details. So, let me try to simplify things. In his answer, Gilles wrote

for x in */C/populate.sh; do
  mkdir -- "${x%%/*}$suffix"
  mkdir -- "${x%%/*}$suffix/C"
  cp -p -- "$x" "./${x%%/*}$suffix/C"
done

As I commented on his question, x here matches patterns of the form */C/populate.sh. Also, ${x%%/*} matches the first part of the string, namely the top level directory name. Now, something like

for x in */C/populate.sh; do                                                                                                                       
    myarr[] = "${x%%/*}"                                                                                                                    
done

would create an array containing a list of top level directories, which is what I want. However, I don't know what syntax to use. I need to use a counter which runs over the loop, like i=0, 1,... to index myarr on the LHS. If I had a working piece of code like this, it would go some way towards solving my issue.

Answer 1

If you want to append things to an array in a loop, you can use something like:

for x in */C/populate.sh; do
  myarr=(${myarr[@]} "${x%%/*}")
done

See the bash arrays documentation for lots of stuff you can do with them.

Alternatively, you can use += to append to an array (see here) like this:

  myarr+=("${x%%/*}")

---

A few comments:

As I commented on his question, x here matches patterns of the form */C/populate.sh.

That's not how I'd have explained it. */foo/bar is a glob pattern. It is expanded by the shell to a list of all files/directories that match this pattern. (Try echo */C/populate.sh, you'll see all the matches printed.)
The for loop iterates over this set of matches, using $x as the loop variable.

Also, ${x%%/*} matches the first part of the string, namely the top level directory name.

${x%%/*} doesn't "match" anything. It's a string manipulation function that operates on $x. From the docs, ${var%%string} removes the longest match of string from the end of $var. In this case, it removes everything from the first / onward and "returns" (expands to) that.

So to break the above three lines of code down, what happens is:

• the shell generates a list of items (files or directories) that match the glob */C/populate.sh.
• for each of those, the loop body is executed with $x set to that item
• ${myarr[@]} expands to the list of each item in the array
• ${x%%/*} expands to $x minus everything from the first / onward
• myarr is then reconstructed with its old contents plus the new, stripped-down item.