Replicate directory structure applying a command to each file instead of simply copying it?

Question

Over time, I've encountered the same pattern again and again: I have some kind of directory structure:

example/
├── a
│   └── c
│       ├── d.txt (120k)
│       └── e.txt (60k)
└── b
    └── f.txt (280k)

And I want to "copy" the files over to another directory, say, example_grepped, applying a command to each as if in place of cp - say, grep ERROR so that say, I end up with a folder with the same structure but with files filtered through the grep.

example_grepped/
├── a
│   └── c
│       ├── d.txt (1k)
│       └── e.txt (0b)
└── b
    └── f.txt (12k)

Same pattern for converting media files (FLACs to MP3s, PNGs to JPGs), and this time when converting different schema formats as part of a build process.

Is there a generic command that I could use? Something like foobar example example_grepped --command 'grep ERROR' or foobar flacs mp3s --command 'ffmpeg -i {} {}.mp3'?

An obscure xargs flag perhaps? (a find piped through xargs would almost suffice, but most if not all commands expect the directory structure to already exist.)

Answer 1

Closest answer I can find without separately recreating the directory structure is to use install:

cd example
find . -type f -exec sh -c 'grep ERROR {} | install -D /dev/stdin /tmp/example_grepped/{}' \;

Unfortunately the above can only work if your command can throw its result to STDOUT.

Answer 2

Another way to approach this is to use a program that does recursive copies anyway. I checked rsync, but could not find a callback option on a quick glance. But gnu tar has an option --to-command for which you can provide a command to be run that gets the file's input into stdin. But how to get the file created then? Well, the command called finds the current file name in $TAR_FILENAME.

Putting it all together, the basic call is

tar cf - example | tar xf - --to-command="./script example_grepped 'grep-pattern'"

where script could be something like

#!/bin/bash
mkdir -p $(dirname "$1/$TAR_FILENAME")
grep '$2' >"$1/$TAR_FILENAME"
exit 0

Another way to approach this, would be to wrap the tar pipe in a script that gets the command to run on the command line. Yet the escaping for the mkdir ...dirname construct will be a bit challenging.

Answer 3

#!/bin/bash

filter() {

    local target_root="${@: -1}"

    target_path=$(sed -E "s/[^/]*/$target_root/" <<< "$1")
    target_dir=$(dirname "$target_path")

    mkdir -p "$target_dir"

    if [[ -f $1 ]]; then
        # do your grep thing here
        grep burger "$1" > "$target_path"
    fi
}

export -f filter
source_root="example"
target_root="example_grepped"

find "$source_root/" -print0 | xargs -0 -I content bash -c "filter 'content' '$target_root'"

This script also works with directory and file names that contain spaces.

Run this script where the source directory ("example") is located.

Answer 4

Using GNU Parallel you can do something like this:

cd src
find . -type f | parallel 'mkdir -p ../dst/{//}; dostuff --input {} --output ../dst/{}'