I wrote some script foo that accepts a file path, reads the file, applies some changes to it, and outputs the changed file to stdout:

foo src/file.foo > dest/file.changed.foo # works fine
cat src/file.foo | foo > dest/file.changed.foo # also works fine

Now, I want to apply this command to multiple files in a directory and for every file do:

src/file.foo --> foo --> dest/file.changed.foo

Based on the answer to a similar question I've come up with the following:

find src -name '*.foo' \
  -exec bash -c 'for x; do dest=${x/src/dest}; foo ${x} > ${dest/\.foo$/.changed.foo}; done' _ {} +

The above works, but the problem is that it's too complicated for everyday use. Isn't there a simpler way to do it? I was wondering if it's the way foo is done that complicates things.


It's a lot less typing and fiddling if you use recursive globbing. In bash, put shopt -s globstar in your .bashrc. In ksh93, recursive globbing requires set -o globstar; in zsh it works out of the box. Beware that in bash, recursive globbing also recurses under symbolic links to directories.

To save on string manipulation, first change to the top of the source tree (or to the top of the destination tree).

cd foo
for x in **/*.foo; do foo "$x" >"../dest/${x%.*}.changed.foo"; done

You can omit the double quotes if you know that your file names don't contain whitespace or globbing characters.

In zsh, the double quotes are never necessary, and you can save some more typing even without changing the current directory.

for x in src/**/*.foo; do foo $x >../dest/${${x#*/}%.*}.changed.foo; done
for x in src/**/*.foo; do foo $x >../dest/${x#*/}:r.changed.foo; done
for x (src/**/*.foo) foo $x >../dest/${x#*/}:r.changed.foo

If you do this often, you should define a build rule, e.g. in a GNUmakefile (use a tab where I used 8 spaces):

source_files = $(shell find src -name '*.foo')
destination_files = $(patsubst src/,dest/,$($(source_files)%.foo=.changed.foo))

default: all-foo
all-foo: $(destination_files)

dest/%.changed.foo: src/%.foo
        foo $< >$@
| improve this answer | |

Assuming your filenames don't have embedded newlines:

find src -name '*.foo' -type f | \
    while IFS= read -r sf; do
        df=dest/${sf##./src/} && \
        mkdir -p "$(dirname "$df")" && \
        foo "$sf" >"${df%%.foo}.changed.foo"
| improve this answer | |
  • FWIW, you only need a single # and % in those expansions because there's no wildcards and so you're removing the "smallest" part; eg ${sf#./src/} – Stephen Harris Aug 14 '16 at 15:49
  • @StephenHarris FWIW, if ./src/ and .foo are fixed, ## and %% have the exact same effect as # and %. :) – Satō Katsura Aug 14 '16 at 15:50
  • Yes, they do the same thing but are unnecessary extra typing ;-) – Stephen Harris Aug 14 '16 at 15:51
  • 1
    Is it really simpler than the original solution? I mean, if I ask my colleagues (front-end devs) to enter that in the terminal, they will throw chairs at me and spill coffee onto my keyboard on purpose. – Oleg Aug 14 '16 at 16:02
  • 1
    @Oleg Then I suppose the easiest for casual users would be to have a wrapper script that takes src and dest as arguments, and applies foo to all .foo files. Otherwise it isn't a problem that has a simple and safe solution, I think (not substantially simpler than the above, anyway). – Satō Katsura Aug 14 '16 at 16:18

Assuming you don't have a tree of source files (eg no src/dir1/a.foo; they're all directly in the src directory) then you don't need the find wrapper and just do it directly in shell:

for x in src/*.foo
  foo "$x" > "${dest/%.foo/.changed.foo}"

The complication, here, is that you want to change filenames, and isn't a consequence of foo.

We can make it slightly easier if foo doesn't require to be run in the current directory

cd src
for x in *.foo
  foo "$x" > "../dest/${x/%.foo/.changed.foo}"

If you didn't want the extension changed then it's about the simplest loop possible

cd src
for x in *.foo
  foo "$x" > "../dest/$x"

All of these loops can be changed into a single line by strategic use of the ; character. eg the first loop becomes

for x in src/*.foo; do dest="${x/src/dest}" ; foo "$x" > "${dest/%.foo/.changed.foo}" ; done
| improve this answer | |
  • This doesn't work if the source directory has .foo files in subdirectories. – Satō Katsura Aug 14 '16 at 15:48
  • Right; I was assuming no subdirectories because then we also need to ensure that the dest subdirectories have also been created; can't just do just >dest/$name when $name may have directory components. – Stephen Harris Aug 14 '16 at 15:50
  • I do have nested src directory structure, forgot to mention it in the question, but it's probably not a concern. I was just wondering: is for loop available on all *nix flavors or there're some gotchas? (sorry for being a noob... I should really read more about all this stuff) – Oleg Aug 14 '16 at 16:22
  • The for loop is standard sh. The ${../../..} syntax is a bash and ksh; dunno if it's POSIX or not. – Stephen Harris Aug 14 '16 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.