DataPipeline - Repeated replace that has issues with sed (as explained) and needs to write files to new location

Question

I need to perform a global (repeated) replace operation on the text of files recursively in a directory, but there are a few constraints that have given me issues:

1. I need to replace in a way that includes a newline character, which breaks sed, as mentioned here: Why is sed giving me an error about an unterminated `s'?
2. I need to output the files in a different directory because I'm using AWS DataPipeline and need to work around an issue that was occurring due to making changes to the files from the InputDirectory during the command execution. (So, I need to write each modified file to a file with the same filename but located in the OutputDirectory.)
3. I also ran into issues with trying to glob the output filename

So, I'm left with something that perhaps would:

1. use find to enumerate the files
2. loop through the files and for each file, replace the input path with the output path (perhaps even with sed)
3. cat the file and pipe the contents to awk and write the contents to the updated path.
4. construct any directories needed in the process (which I believe should happen automatically from output redirection, but I want to say it explicitly just in case)

Is there a straightforward way to do this?

Answer 1

1. If you use GNU sed, you can use \n for newline in the replacement string. With other sed, use a literal newline, but escape it with a backslash.

2. use find ... -print | sed -e 's:oldpath:newpath:' rather than a shell loop to change the paths.

   If either oldpath or newpath contain : characters, use a different delimiter.

   And if any of the filenames might contain a newline character (which IS a valid filename character) and GNU sed is available to you (default/standard on linux), use find's -print0 option instead of -print and GNU sed's -z option: sed -z -e 's:oldpath:newpath:'

3. you can create the directory path with something like mkdir -p "$(dirname "$filename")" for each input filename.

4. why not just use rsync -a sourcedir destdir, optionally with some relevant --exclude and/or --include options?