Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a directory with a bunch of subdirectories in it. Thus

/usr/local/src/ccl/ccl-1.8/x86-headers$ ls
elf  gl  gmp  gnome2  gtk2  jni  libc

Each of these directories has a further subdirectory C inside it, which contains a file populate.sh. I want to create a parallel group of subdirectories with the same structure, but with a random value appended to the directory name (the random value should be the same in all cases), and only containing the C subdirectory with the populate.sh file. These directories contain other files besides the populate.sh file.

This is for a makefile, so for simplicity should probably use standard unix utilties. I'm thinking find with the -exec flag, or possibly xargs would work, but I'm having trouble making sense of the documentation, and I have little experience with shell scripting. Perl might work, but I have not used it, and would prefer not to use it here.

I've been using something like mktemp -u --tmpdir=. to generate a random string in the past, but it is hardly ideal, so I'm open to other suggestions. Ideally I'd like a name that looks like libc.tmp_xw3st. Ie. tmp_ followed by a 5 digit alphanumeric string.

So far, I've got a way of getting a listing of the top level directories. :-)

find . -maxdepth 1 -type d -print0

../gmp./jni./gl./elf./libc./gtk2./gnome2

A fuller directory listing is at the end of this posting. To summarize, I want to create additional directories like x86-headers/libc.tmpvalue, which only contains the further file x86-headers/libc.tmpvalue/C/populate.sh.

A sketch of a possible approach is to handle this in two steps as follows:

Step 1: Run over the list of top level directories using find, and create a corresponding directory structure eg dirname.tmpvalue/C/ using exec or piping to xargs and using mkdir -p.

Step 2: Run over the list of top level directories again and cp populate.sh into the C subdirectories. This is a bit sloppy, because the list of directories in theory could have altered between the two invocations of find, but this is not an issue in this case.

/usr/local/src/ccl/ccl-1.8/x86-headers$ ls -laR

[...]

./jni:
total 96
drwxr-sr-x 3 faheem staff  4096 Jul 31 00:53 .
drwxr-sr-x 9 faheem staff  4096 Jul 31 00:53 ..
drwxr-sr-x 2 faheem staff  4096 Jul 31 00:53 C
-rw-r--r-- 1 faheem staff 19535 Jul 31 00:53 constants.cdb
[more .cdb files]

./jni/C:
total 12
drwxr-sr-x 2 faheem staff 4096 Jul 31 00:53 .
drwxr-sr-x 3 faheem staff 4096 Jul 31 00:53 ..
-rw-r--r-- 1 faheem staff  148 Jul 31 00:53 populate.sh

./libc:
total 1276
drwxr-sr-x 3 faheem staff   4096 Jul 31 00:53 .
drwxr-sr-x 9 faheem staff   4096 Jul 31 00:53 ..
drwxr-sr-x 2 faheem staff   4096 Jul 31 00:53 C
-rw-r--r-- 1 faheem staff 593125 Jul 31 00:53 constants.cdb
[more .cdb files]

./libc/C:
total 20
drwxr-sr-x 2 faheem staff  4096 Jul 31 00:53 .
drwxr-sr-x 3 faheem staff  4096 Jul 31 00:53 ..
-rwxr-xr-x 1 faheem staff 10544 Jul 31 00:53 populate.sh
share|improve this question
    
@WarrenYoung: Thanks for your comment. To be clear, I didn't say Gilles' answer had problems. I said using find had problems (from my perspective), and I mentioned two. I think Gilles approach is a good way to go, and I just asked him for a couple of clarifications. I think it is the best of the answers here - the others all use find. It is true I haven't accepted an answer yet, but I usually wait for a bit. As you can see from my record, I generally accept an answer eventually. I'll update my other question to hopefully make things a little clearer. –  Faheem Mitha Aug 2 '12 at 19:15
add comment

4 Answers

up vote 4 down vote accepted

This is not a job for find, since there is no recursion involved.

for x in */C/populate.sh; do
  mkdir -- "${x%%/*}$suffix"
  mkdir -- "${x%%/*}$suffix/C"
  cp -p -- "$x" "./${x%%/*}$suffix/C"
done

Remove the -p option to cp if you don't want to preserve the files' modification time.

To generate a random suffix, BSD/Linux mktemp is as portable as it gets.

suffix=$(mktemp -u tmp_XXXXX)

If you want something vaguely random-looking and POSIX-compliant, this gives a string that changes every second and varies from location to location; you can't really do better with only POSIX tools:

suffix=$({ hostname; pwd; date; } |
         cksum | uuencode -m /dev/stdin | awk 'NR==2 {print substr($0,3,5)}')

If you put this code in a makefile, remember to:

  • double all $ signs;
  • put all the code on one line, using ; instead to separate shell instructions (you can use backslash+newline+tab to put a line break in the makefile, but that sequence is removed to build the shell command);
  • start the shell snippet with set -e, so that it aborts if there is any error.
share|improve this answer
    
Hi Gilles, thanks for the pointers. A couple of questions. 1) Use of find seems harmless to me, even if not strictly necessary. Can you elaborate on how your method works? Does your method match all directories, even those like .foo? 2) If you want some line of the makefile (which is a shell snippet) to abort on error, is the best way to prefix it with set -e? –  Faheem Mitha Aug 1 '12 at 7:51
    
@FaheemMitha 1) * doesn't match .foo. In a compilation tree, this is usually the desired behavior, skipping .svn and the like (though here the requirement to have a path C/populate.sh will exclude them anyway). Add .[!.]*/C/populate.sh if you have directories whose name begins with a dot. 2) Start a command with set -e. E.g. set -e; for x in … –  Gilles Aug 1 '12 at 8:17
    
Hi, Gilles. Having studied your example some more, it mostly does what I want, but I have some more questions. for x in */C/populate.sh matches x with anything of the form */C/populate.sh. The construct ${x%%/*} matches the first part of the string, namely the top level directory name. Can you elaborate on these two pieces of syntax, and/or provide references, please? Syntax like %% is a bit hard to google for. Thanks. I've decided to try to save these values to an array. See unix.stackexchange.com/q/44554/4671 - I'll be updating that question shortly. –  Faheem Mitha Aug 2 '12 at 19:06
    
@FaheemMitha Search for % in your shell's manual in the section about parameter expansion. ${x%%/*} is $x with the longest match for the pattern /* removed at the end, i.e. it's $x minus everything after the first /. –  Gilles Aug 2 '12 at 20:21
    
Thanks, that's very helpful. For the record, in the Bash Reference Manual this corresponds to the bit starting with ${parameter%%word}. One more thing: what is the function of the --? I tried it without e.g. mkdir -p "${x%%/*}$suffix/C", and it still works. –  Faheem Mitha Aug 2 '12 at 21:14
show 2 more comments

You can create a copy of every top level directory using

$ find . -maxdepth 1 -type d -exec mkdir {}".tmpvalue" \;

You can then iterate and copy the .sh file to each of these directories.

share|improve this answer
    
Unfortunately this doesn't quite work, since find . -maxdepth 1 -type d actually includes the current directory too, which we don't want here. So that needs to be excluded somehow. –  Faheem Mitha Jul 31 '12 at 4:23
    
Ok, rnd=$(tr -cd a-z0-9 < /dev/urandom | head -c5); find . -mindepth 1 -maxdepth 1 -type d -exec mkdir {}".tmp_"$rnd \; works. –  Faheem Mitha Jul 31 '12 at 4:32
add comment

Try something like this:

#!/bin/sh
# Create a temporary partial clone of the current directory
tcopy=`mktemp -d /tmp/tmp_XXXXX`
tbase=`basename $tcopy`
find . -name populate.sh -print | cpio -pud "$tcopy"

# Randomize the clone
for d in "$tcopy"/*
do
    mv "$d" "$d.$tbase"
done

# Merge the clone back into the current tree
mv "$tcopy"/* .

This depends on a mv implementation that knows how to move whole directories across filesystems. GNU mv can, but I remember working on some old Unix boxes where that final line wouldn't work. You could use another find | cpio pass instead in that case. mv is more efficient if it works.

share|improve this answer
    
Any alphanumeric string would work for the temp name. So, it might look like libc.tmp_txw2e or something. I'd like the random string to look like tmp_ followed by 5 chars if possible. –  Faheem Mitha Jul 31 '12 at 2:16
    
This doesn't do what I want. I want subdirectories to be created at the same level, as in my example. This creates a whole separate directory structure. If I am misunderstanding, can you provide an example session? Thanks. –  Faheem Mitha Jul 31 '12 at 2:28
    
Rewrote my answer to address these comments. If that still doesn't get it, edit your question to give a pair of find . -print outputs, with the before and after. –  Warren Young Jul 31 '12 at 2:49
add comment

I wouldn't worry too much about -print0 as long as the file names are well formed. Below is an example that should do what you want:

rnd=$(tr -cd a-z0-9 < /dev/urandom | head -c5)
find . -maxdepth 1 -type d | while read dir; do
   mkdir -p ${dir}_${rnd}/C
   cp populate.sh ${dir}_${rnd}/C
done

Explanation

  • tr -cd deletes the complement of the specified pattern, alphanumerics in this case.
  • head -c5 takes the first five characters and exits.
  • while runs the read command for every line that find provides.
  • read assigns the directory from find to $dir.
share|improve this answer
    
This doesn't work quite right. If I remove the cp line, then only one directory is created, and that doesn't have the dir name in front of it, it's just a 5 char random string. So, I conclude that $dir is not getting assigned correctly, and also the looping is not happening correctly. Also the first argument in cp would not work - populate.sh is down two levels. If you could explain the while read dir bit that would be helpful. –  Faheem Mitha Jul 31 '12 at 4:14
    
$dir needs to be protected from other types of expansion, so to fix this put braces around it: ${dir}. I've updated the example. –  Thor Jul 31 '12 at 8:32
    
Use the populate.sh that makes sense in your case. –  Thor Jul 31 '12 at 8:34
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.