1

(See update at the foot of the question).

This is a followup question to "Make directory copies using find".

This question involved manipulating a bunch of directories, This got too complicated to handle in a single command, so I decided to go with an approach which saved the list of directories to a bash array, despite reservations about portability. It seems that the POSIX shell standard (the Unix shell standard, as I understand it) does not have arrays.

The makefile I'm using appears below. This works except for the last step. A summary of what I'm trying to do follows.

As discussed in the earlier question, I want to loop over the directory x86-headers, and collect into a bash array, a list of its top level subdirectories that contain the file C/populate.sh (but may also contain other files). In my test setup, for example there is only one directory in x86-headers which contains the file libc/C/populate.sh, namely libc.

I then perform some operations on these subdirectories. The most important of these is that I make a temporary copy of each directory which looks like libc.tmp_g4zlb. Namely dirname followed by 'tmp_' followed by a 5 digit random string.

So, some questions:

1) As discussed in the earlier question, I'm looping over the directory 'x86-headers'. Here I am still using find. @Gilles indicated in his answer that this was not ideal. He might be right. Problems with find here:

a) The values returned look like ./libc. I don't want a leading ./.

b) The find command I'm using lists all directories. I only want to consider those that contain a file with the relative path of C/populate.sh.

The approach Gilles was using might be better, but I don't understand it. See the gilles target below. I'd like to get the list of the directories and save them to an array.

2) The bit that I'm having problems with is the last step, namely

echo "(progn (require 'parse-ffi) (ccl::parse-standard-ffi-files :$$i'.tmp_'$(RND)))" | \
/usr/lib/ccl-bootstrap/lx86cl -I /usr/lib/ccl-bootstrap/lx86cl.image; done \

The relevant bit is that I'm trying to pass the temporary value libc.tmp_g4zlb to ccl, which is a Common Lisp compiler. Without the substitution, it would look like

echo "(progn (require 'parse-ffi) (ccl::parse-standard-ffi-files :libc.tmp_g4zlb))" | \
/usr/lib/ccl-bootstrap/lx86cl -I /usr/lib/ccl-bootstrap/lx86cl.image; done \

This works. The version using $$i above doesn't. The problem seems to be the leading ./. Without that, it should work. For the record, the error I get is

? > Error: Too many arguments in call to #<Compiled-function CCL::PARSE-STANDARD-FFI-FILES #x488B8406>:
>        3 arguments provided, at most 2 accepted. 
> While executing: CCL::PARSE-STANDARD-FFI-FILES, in process listener(1).

This is, however, not the error that I get when passing ./libc.tmp_g4zlb to the compiler. That error looks like

> Error: Filename "/home/faheem/test/foo/x86-headers/.\\/libc.tmp_g4zlb/C/**/" contains illegal character #\/
> While executing: CCL::%SPLIT-DIR, in process listener(1).

So it is possible there is something else going on.

3) Does the overall approach look reasonable? Please feel free to suggest possible improvements, even if it involves a completely different strategy.

#!/usr/bin/make -f
# -*- makefile -*-

export SHELL=/bin/bash
export RND:=$(shell tr -cd a-z0-9 < /dev/urandom | head -c5)
export CCL_DEFAULT_DIRECTORY=$(CURDIR)

clean:
    rm -rf x86-headers/libc.tmp*

foo: clean
    PATH=$$PATH:$(CURDIR)/ffigen4/bin; \
    echo $$PATH; \
    export CURDIR=$(CURDIR); \
    echo $$CURDIR; \
    array=( $$(cd x86-headers && find . -mindepth 1 -maxdepth 1 -type d) ); \
    cd x86-headers && \
    for i in "$${array[@]}"; do \
    echo $$i; done; \
    for i in "$${array[@]}"; do \
    mkdir -p $$i."tmp_"$(RND)/C; done; \
    for i in "$${array[@]}"; do \
    cp -p $$i/C/populate.sh $$i".tmp_"$(RND)/C; done; \
    for i in "$${array[@]}"; do \
    cd $$i".tmp_"$(RND)/C; ./populate.sh; done; \
    for i in "$${array[@]}"; do \
    echo $$i'.tmp_'$(RND); done; \
    for i in "$${array[@]}"; do \
    echo "(progn (require 'parse-ffi) (ccl::parse-standard-ffi-files :$$i'.tmp_'$(RND)))" | \
    /usr/lib/ccl-bootstrap/lx86cl -I /usr/lib/ccl-bootstrap/lx86cl.image; done; \

gilles:
    cd x86-headers;
    for x in */C/populate.sh; do \
    echo -- "$${x%%/*}$$suffix"; done; \

UPDATE: It is possible that the question (or questions) got lost in all the details. So, let me try to simplify things. In his answer, Gilles wrote

for x in */C/populate.sh; do
  mkdir -- "${x%%/*}$suffix"
  mkdir -- "${x%%/*}$suffix/C"
  cp -p -- "$x" "./${x%%/*}$suffix/C"
done

As I commented on his question, x here matches patterns of the form */C/populate.sh. Also, ${x%%/*} matches the first part of the string, namely the top level directory name. Now, something like

for x in */C/populate.sh; do                                                                                                                       
    myarr[] = "${x%%/*}"                                                                                                                    
done

would create an array containing a list of top level directories, which is what I want. However, I don't know what syntax to use. I need to use a counter which runs over the loop, like i=0, 1,... to index myarr on the LHS. If I had a working piece of code like this, it would go some way towards solving my issue.

10
  • What does "doesn't work" mean? Please provide the output you get Aug 2, 2012 at 11:26
  • 1
    Why don't you separate the shell parts out into proper scripts? You'll have less trouble with quoting and more readability than inline in that makefile.
    – Mat
    Aug 2, 2012 at 12:34
  • @danielkullmann: updated with the error messages. Aug 2, 2012 at 16:06
  • @Mat: sure, that might make sense, and I have vaguely thought of that. However, does putting it in a makefile make a difference beyond having to double the $'s? I haven't seen a summary of the things you need to watch out for. It is unfortunate that shell syntax and GNU Make syntax are similar, yet different, so that they tend to trip over each other. In any case, since the script is almost working, I;d like to go the extra mile and get it working. Aug 2, 2012 at 16:14
  • @Mat (contd): The first thing I'd like to do is fix the issue with the ./ in front of the directory names, possibly by using something other than find, and then see what happens. Aug 2, 2012 at 16:16

1 Answer 1

4

If you want to append things to an array in a loop, you can use something like:

for x in */C/populate.sh; do
  myarr=(${myarr[@]} "${x%%/*}")
done

See the bash arrays documentation for lots of stuff you can do with them.

Alternatively, you can use += to append to an array (see here) like this:

  myarr+=("${x%%/*}")

A few comments:

As I commented on his question, x here matches patterns of the form */C/populate.sh.

That's not how I'd have explained it. */foo/bar is a glob pattern. It is expanded by the shell to a list of all files/directories that match this pattern. (Try echo */C/populate.sh, you'll see all the matches printed.)
The for loop iterates over this set of matches, using $x as the loop variable.

Also, ${x%%/*} matches the first part of the string, namely the top level directory name.

${x%%/*} doesn't "match" anything. It's a string manipulation function that operates on $x. From the docs, ${var%%string} removes the longest match of string from the end of $var. In this case, it removes everything from the first / onward and "returns" (expands to) that.

So to break the above three lines of code down, what happens is:

  • the shell generates a list of items (files or directories) that match the glob */C/populate.sh.
  • for each of those, the loop body is executed with $x set to that item
  • ${myarr[@]} expands to the list of each item in the array
  • ${x%%/*} expands to $x minus everything from the first / onward
  • myarr is then reconstructed with its old contents plus the new, stripped-down item.
5
  • Thanks, that works. However, can you offer a more specific reference or break it down a bit? I've looked at your reference, but I don't see it. The myarr[@] is some kind of magic for-each operator, it seems. Aug 2, 2012 at 21:41
  • There's nothing much to break down, the append is of the form array=(${array[@]} $appended_element). Search for append in that page. See also tldp.org/LDP/Bash-Beginners-Guide/html/sect_10_02.html (${array[@]} expands to all elements of the array.)
    – Mat
    Aug 2, 2012 at 21:43
  • To append an element to an array, you can use myarr+=("${x%%/*}") (note that the parentheses are necessary in bash). Aug 2, 2012 at 23:14
  • Thanks, I didn't know that was possible. Much nicer syntax.
    – Mat
    Aug 4, 2012 at 8:53
  • @Mat: Thanks for the answer, and sorry for the slow response. I eventually decided not to use arrays in this case after all, but it is good to know how to do so. Just one followup question - does */foo/bar get expanded every time the loop passes over the top, or just the first time it enters the loop? Aug 23, 2012 at 4:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .