0

I need to perform a global (repeated) replace operation on the text of files recursively in a directory, but there are a few constraints that have given me issues:

  1. I need to replace in a way that includes a newline character, which breaks sed, as mentioned here: Why is sed giving me an error about an unterminated `s'?
  2. I need to output the files in a different directory because I'm using AWS DataPipeline and need to work around an issue that was occurring due to making changes to the files from the InputDirectory during the command execution. (So, I need to write each modified file to a file with the same filename but located in the OutputDirectory.)
  3. I also ran into issues with trying to glob the output filename

So, I'm left with something that perhaps would:

  1. use find to enumerate the files
  2. loop through the files and for each file, replace the input path with the output path (perhaps even with sed)
  3. cat the file and pipe the contents to awk and write the contents to the updated path.
  4. construct any directories needed in the process (which I believe should happen automatically from output redirection, but I want to say it explicitly just in case)

Is there a straightforward way to do this?

Improve this question
4
  • use \n for newline in the replacement string.
    – cas
    Jan 17, 2018 at 5:24
  • Can you elaborate on issue 3? Have you tried using globstar?
    – m0dular
    Jan 17, 2018 at 5:37
  • 1
    @cas \n in the replacement string is not portable! We don't know whether devinbost or future readers use GNU sed. Better use a newline escaped by a backslash.
    – Philippos
    Jan 17, 2018 at 8:03
  • Using \n breaks when I run it in AWS DataPipeline, as per the link that I included in the post.
    – devinbost
    Jan 17, 2018 at 16:53

1 Answer 1

Reset to default
0

  1. If you use GNU sed, you can use \n for newline in the replacement string. With other sed, use a literal newline, but escape it with a backslash.

  2. use find ... -print | sed -e 's:oldpath:newpath:' rather than a shell loop to change the paths.

    If either oldpath or newpath contain : characters, use a different delimiter.

    And if any of the filenames might contain a newline character (which IS a valid filename character) and GNU sed is available to you (default/standard on linux), use find's -print0 option instead of -print and GNU sed's -z option: sed -z -e 's:oldpath:newpath:'

  3. you can create the directory path with something like mkdir -p "$(dirname "$filename")" for each input filename.

  4. why not just use rsync -a sourcedir destdir, optionally with some relevant --exclude and/or --include options?

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .