I have a file which contains some text and - on a single line - a marker which indicates where new content should be added

foo
bar
%SUBSTITUTE%
foo

The line line substitute should be replaced by a new multiline string text="some text" (note that I do not know the string it may for instance be the result of reading a file text=cat "file"``). The result of the replacement should read

foo
bar
some text
%SUBSTITUTE%
foo

I had a working version based on perl which stopped working (apparently due to perl version change). Now I am trying standard utilities like tr and sed to replace the line. I am having some trouble with it, because the string to be pasted may contain arbitrary characters including backslashes etc. I do not want to escape these before pasting.

Is there a safe way to do it which works with standard tools? The other questions I find for the problem are particular solutions where the text to be pasted is known.

  • You might be able to use the venerable (and somewhat scary) m4. The use case you describe is very similar to its intended functionality. – Faheem Mitha Nov 15 '14 at 12:48
  • Do you have an example? – highsciguy Nov 15 '14 at 12:52
  • I'm unclear on your example. Shouldn't your marker %SUBSTITUTE% be gone in the second piece of text? It's still there. You also have The line line substitute. iIs that intentional? – Faheem Mitha Nov 15 '14 at 13:02
  • It should stay (which complicates the problem). The percent signs are to make it a tex comment. It indicates where the next chunk of text shall be inserted. – highsciguy Nov 15 '14 at 13:06
  • use this command: sed '/%SUBSTITUTE%/i\some text\nnew line' file – αғsнιη Nov 15 '14 at 13:19

If you have the GNU version of sed, you should be able to use the r command to read and insert new content from a file, then delete the marker line e.g.

sed '/%SUBSTITUTE%/{
r path/to/newcontent
d
}' file

If you want to retain the %SUBSTITUTE% marker after the insertion, that's tricky because whatever you do, the GNU r extension queues the file contents until the end of the current pattern cycle (retaining it before would just be a matter of removing the d command). Probably the simplest way is to append it to the newcontent file: you could do that on the fly like

sed '/%SUBSTITUTE%/{
r /dev/stdin
d
}' file < <(sed '$a %SUBSTITUTE%' path/to/newcontent)

Taking a completely different approach, you could split the first file on %SUBSTITUTE% and then cat in the new content

csplit -s file '/%SUBSTITUTE%/'
cat xx00 newcontent xx01

You could also do a bash read loop over the lines of the first file, and cat the new content file when you match the marker string - however I've had my knuckles rapped for suggesting read loops for text processing on this forum. Unfortunately neither provides an in-place solution.

  • It would better be GNU independent but I will try it. I would however need to re-insert the %SUBSTITUTE% because it should stay in the file to indicate where the next substitution is to be made. – highsciguy Nov 15 '14 at 12:50
  • sed -e "/%SUBSTITUTE%/i$(sed 's/$/\\/' path/to/newcontent)" -e 's/\\$//' file works too. The main trick is to blackslash newlines in newcontent than get it off. Note: If you have \n, \t, etc. its will substituted by its real meaning: «newline», «tab», etc. – Costas Nov 15 '14 at 15:58
  • 1
    Additionally in my sed (GNU sed) 4.2.2 I found undocumented command e, so if supported you can do sed -e '/%SUBSTITUTE%/e cat /path/to/newcontent' file – Costas Nov 15 '14 at 16:13
  • You don't need GNU sed for r - but neither thr GNU nor any other version will insert arbitrary text as a replacement to marker - ronly works on the end of the line cycle. @Costas suggestion, however, would work. But Costas - it is documented. Do info sed. – mikeserv Feb 19 '15 at 2:38

Here's another way using ed.
Insert the whole content of FILE before the marker (i.e. before the line containing %SUBSTITUTE%):

ed -s originalfile <<< $'/%SUBSTITUTE%/- r FILE\nw\nq'

where:
/%SUBSTITUTE%/ : sets address at first line matching %SUBSTITUTE%
- or -1 : offsets address one line before
r FILE : Reads FILE to after the addressed line.
w : writes to originalfile (replace with ,p to just print the content instead of writing)
q : quits editor

Replacing FILE with !echo "$TEXT" will insert the content of $TEXT before the marker:

export TEXT
ed -s originalfile <<'IN'
/%SUBSTITUTE%/-1 r !echo "$TEXT"
w
q
IN

I found the following solution based on awk:

text=`cat "filepaste"`
export text;
<"$file" awk '
    BEGIN {REPLACE=ENVIRON["text"] "\n%SUBSTITUTE%" }
    {gsub(/^%SUBSTITUTE%$/, REPLACE); print}
'

Here "filepaste" contains the content to substitute for %SUBSTITUTE%. An advantage is that this string may be acted upon using different shell tools without the need to save it back to a file. Reading the awk variable REPLACE from the environment variable avoids the expansion of escaped characters in text.

Since you had a Perl solution, here's another. I am using rep.txt to store the replacement:

$ perl -pe '$re=`cat rep.txt`; chomp($re); s/%SUBSTITUTE%\n/$re/' file.txt
foo
bar
multi
  line
string
foo

That reads each line of the target file (file.txt), then applies the script given by -e to it and prints (-p).

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.