I've noticed that, if I add \n to a pattern for substituting using sed, it does not match. Example:

$ cat > alpha.txt
This is
a test
Please do not
be alarmed

$ sed -i'.original' 's/a test\nPlease do not/not a test\nBe/' alpha.txt

$ diff alpha.txt{,.original}

$ # No differences printed out

How can I get this to work?

  • Smart workaround here: unix.stackexchange.com/a/445666/61742 . Of course it is not performatic! Other good options for performing a replace according to your needs may be awk, perl and python. There are many others, but I believe that awk is the most universal in the various Linux distributions (for example). Thanks! – Eduardo Lucio May 25 at 0:17

12 Answers 12

up vote 196 down vote accepted

In the simplest calling of sed, it has one line of text in the pattern space, ie. 1 line of \n delimited text from the input. The single line in the pattern space has no \n... That's why your regex is not finding anything.

You can read multiple lines into the pattern-space and manipulate things surprisingly well, but with a more than normal effort.. Sed has a set of commands which allow this type of thing... Here is a link to a Command Summary for sed. It is the best one I've found, and got me rolling.

However forget the "one-liner" idea once you start using sed's micro-commands. It is useful to lay it out like a structured program until you get the feel of it... It is surprisingly simple, and equally unusual. You could think of it as the "assembler language" of text editing.

Summary: Use sed for simple things, and maybe a bit more, but in general, when it gets beyond working with a single line, most people prefer something else...
I'll let someone else suggest something else.. I'm really not sure what the best choice would be (I'd use sed, but that's because I don't know perl well enough.)


sed '/^a test$/{
       $!{ N        # append the next line when not on the last line
         s/^a test\nPlease do not$/not a test\nBe/
                    # now test for a successful substitution, otherwise
                    #+  unpaired "a test" lines would be mis-handled
         t sub-yes  # branch_on_substitute (goto label :sub-yes)
         :sub-not   # a label (not essential; here to self document)
                    # if no substituion, print only the first line
         P          # pattern_first_line_print
         D          # pattern_ltrunc(line+nl)_top/cycle
         :sub-yes   # a label (the goto target of the 't' branch)
                    # fall through to final auto-pattern_print (2 lines)
       }    
     }' alpha.txt  

Here it is the same script, condensed into what is obviously harder to read and work with, but some would dubiously call a one-liner

sed '/^a test$/{$!{N;s/^a test\nPlease do not$/not a test\nBe/;ty;P;D;:y}}' alpha.txt

Here is my command "cheat-sheet"

:  # label
=  # line_number
a  # append_text_to_stdout_after_flush
b  # branch_unconditional             
c  # range_change                     
d  # pattern_delete_top/cycle          
D  # pattern_ltrunc(line+nl)_top/cycle 
g  # pattern=hold                      
G  # pattern+=nl+hold                  
h  # hold=pattern                      
H  # hold+=nl+pattern                  
i  # insert_text_to_stdout_now         
l  # pattern_list                       
n  # pattern_flush=nextline_continue   
N  # pattern+=nl+nextline              
p  # pattern_print                     
P  # pattern_first_line_print          
q  # flush_quit                        
r  # append_file_to_stdout_after_flush 
s  # substitute                                          
t  # branch_on_substitute              
w  # append_pattern_to_file_now         
x  # swap_pattern_and_hold             
y  # transform_chars                   
  • 124
    Shoot me now. Worst syntax ever! – Gili May 20 '14 at 19:51
  • 38
    This is a fantastic explanation, but I'm inclined to agree with @Gili. – gatoatigrado May 21 '14 at 20:44
  • 11
    Your cheat-sheet has it all. – konsolebox Jul 19 '14 at 12:30
  • 3
    You don't need a label to use the t command here—when not given a label it defaults to branching to the end of the script. So sed '/^a test$/{$!{N;s/^a test\nPlease do not$/not a test\nBe/;t;P;D}}' alpha.txt does exactly the same as your command in all circumstances. Of course for this particular file, sed '/test/{N;s/.*/not a test\nBe/}' alpha.txt does the same thing also, but my first example is logically equivalent for all possible files. Also note that \n in a replacement string doesn't produce a newline; you need a backslash ` \ ` followed by an actual newline to do that. – Wildcard Oct 24 '15 at 13:13
  • 9
    Note that that syntax is GNU specific (# command not separated from the previous one, \n in RHS of s). With GNU sed you can also use -z to use NUL delimited records (and then slurp in the whole input if it's text (which by definition doesn't contain NULs)). – Stéphane Chazelas Aug 11 '16 at 6:33

Use perl instead of sed:

$ perl -0777 -i.original -pe 's/a test\nPlease do not/not a test\nBe/igs' alpha.txt
$ diff alpha.txt{,.original}
2,3c2,3
< not a test
< Be
---
> a test
> Please do not

-pi -e is your standard "replace in place" command-line sequence, and -0777 causes perl to slurp files whole. See perldoc perlrun to find out more about it.

  • 3
    Thanks! For multiline work, perl wins hands down! I ended up using ` $ perl -pi -e 's/bar/baz/' fileA` to change the file in-place. – Nicholas Tolley Cottrell Feb 4 '13 at 14:36
  • 3
    It is very common that the original poster ask for sed and replies using awk or perl appear. I think it is not on topic, hence, sorry, but I fired a minus one. – Rho Phi Aug 26 '15 at 22:07
  • 61
    +1 & disagree with Roberto. Often questions phrased specifically for ignorance of better methods. When there isn't a substantive contextual difference (like here), optimal solutions should get at least as much profile as the question-specific ones. – geotheory Sep 4 '15 at 15:47
  • 45
    I think the sed answer above proves that a Perl answer is on topic. – reinierpost Nov 24 '15 at 10:46
  • 5
    A little bit easier: With "-p0e" the "-0777" is not necessary. unix.stackexchange.com/a/181215/197502 – Weidenrinde Mar 3 '17 at 11:59

I think, it's better to replace \n symbol with some other symbol, and then work as usual:

e.g. not-worked source code:

cat alpha.txt | sed -e 's/a test\nPlease do not/not a test\nBe/'

can be changed to:

cat alpha.txt | tr '\n' '\r' | sed -e 's/a test\rPlease do not/not a test\rBe/'  | tr '\r' '\n'

If anybody doesn't know, \n is UNIX line ending, \r\n - windows, \r - classic Mac OS. Normal UNIX text doesn't use \r symbol, so it's safe to use it for this case.

You can also use some exotic symbol to temporarily replace \n. As an example - \f (form feed symbol). You can find more symbols here.

cat alpha.txt | tr '\n' '\f' | sed -e 's/a test\fPlease do not/not a test\fBe/'  | tr '\f' '\n'
  • 8
    +1 for this clever hack! Especially useful is the advice about using an exotic symbol to temporarily replace newline unless you're absolutely certain about the content of the file you're editing. – L0j1k Feb 6 '15 at 21:26
  • This doesn't work as written on OS X. Instead, one needs to replace all instances of \r in the argument to sed with $(printf '\r'). – abeboparebop Feb 11 '16 at 15:33
  • @abeboparebop: great find! 👍 alternatively, install GNU sed using homebrew: stackoverflow.com/a/30005262 – ssc Jan 24 '17 at 20:05

sed has three commands to manage multi-line operations: N, D and P (compare them to normal n, d and p).

In this case, you can match the first line of your pattern, use N to append the second line to pattern space and then use s to do your substitution.

Something like:

/a test$/{
  N
  s/a test\nPlease do not/not a test\nBe/
}
  • 1
    This is awesome! Simpler than the accepted answer and still effective. – jeyk Nov 21 '15 at 20:35
  • And all the ones involving the hold space (G, H, x...). More lines can be added into the pattern space with the s command as well. – Stéphane Chazelas Aug 11 '16 at 6:38
  • this solution does not work with following case "This is\n a test\n a test\n Please do not\n be alarmed" – mug896 Jan 14 '17 at 15:13
  • @mug896 you most likely need multiple N commands – loa_in_ May 24 at 8:30

All things considered, gobbling the entire file may be the fastest way to go.

Basic syntax is as follows:

sed -e '1h;2,$H;$!d;g' -e 's/__YOUR_REGEX_GOES_HERE__...'

Mind you, gobbling the entire file may not be an option if the file is tremendously large. For such cases, other answers provided here offer customized solutions that are guaranteed to work on a small memory footprint.

For all other hack and slash situations, merely prepending -e '1h;2,$H;$!d;g' followed by your original sed regex argument pretty much gets the job done.

e.g.

$ echo -e "Dog\nFox\nCat\nSnake\n" | sed -e '1h;2,$H;$!d;g' -re 's/([^\n]*)\n([^\n]*)\n/Quick \2\nLazy \1\n/g'
Quick Fox
Lazy Dog
Quick Snake
Lazy Cat

What does -e '1h;2,$H;$!d;g' do?

The 1, 2,$, $! parts are line specifiers that limit which lines the directly following command runs on.

  • 1: First line only
  • 2,$: All lines starting from the second
  • $!: Every line other than last

So expanded, this is what happens on each line of an N line input.

  1: h, d
  2: H, d
  3: H, d
  .
  .
N-2: H, d
N-1: H, d
  N: H, g

The g command is not given a line specifier, but the preceding d command has a special clause "Start next cycle.", and this prevents g from running on all lines except the last.

As for the meaning of each command:

  • The first h followed by Hs on each line copies said lines of input into sed's hold space. (Think arbitrary text buffer.)
  • Afterwards, d discards each line to prevents these lines from being written to the output. The hold space however is preserved.
  • Finally, on the very last line, g restores the accumulation of every line from the hold space so that sed is able to run its regex on the whole input (rather than in a line-at-a-time fashion), and hence is able to match on \ns.
  • 2
    -1, care to comment? – antak Oct 19 '15 at 3:54
  • It wasn't me, but you might consider explaining what -e '1h;2,$H;$!d;g' does, exactly. – jeyk Nov 21 '15 at 20:33
  • 1
    Right, explanation added – antak Nov 24 '15 at 8:35
  • Great explanation. I learned so much about sed from it, that I wish I could upvote more than once. – jeyk Nov 25 '15 at 9:35

You can but it's difficult. I recommend switching to a different tool. If there's a regular expression that never matches any part of the text you want to replace, you can use it as an awk record separator in GNU awk.

awk -v RS='a' '{gsub(/hello/, "world"); print}'

If there are never two consecutive newlines in your search string, you can use awk's "paragraph mode" (one or more blank lines separate records).

awk -v RS='' '{gsub(/hello/, "world"); print}'

An easy solution is to use Perl and load the file fully into memory.

perl -0777 -pe 's/hello/world/g'
  • 1
    How to apply the perl command to a file? – sebix Feb 2 '16 at 13:34
  • 2
    @sebix perl -0777 -pe '…' <input-file >output-file. To modify a file in place, perl -0777 -i -pe '…' filename – Gilles Feb 2 '16 at 13:56
  • 2
    See also GNU sed's -z option (added in 2012 after that answer was posted): seq 10 | sed -z 's/4\n5/a\nb/'. – Stéphane Chazelas Aug 11 '16 at 6:42
sed -i'.original' '/a test/,/Please do not/c not a test \nBe' alpha.txt

Here /a test/,/Please do not/ is considered as a block of (multi line) text, c is the change command followed by new text not a test \nBe

In the case of the text to be replaced is very long, I would suggest ex syntax.

  • oops the problem is that sed will replace all eventual text between /a test/ and /Please do not/ as well... :( – noonex Nov 24 '16 at 8:47

Apart from Perl, a general and handy approach for multiline editing for streams (and files too) is:

First create some new UNIQUE line separator as you like, for instance

$ S=__ABC__                     # simple
$ S=__$RANDOM$RANDOM$RANDOM__   # better
$ S=$(openssl rand -hex 16)     # ultimate

Then in your sed command (or any other tool) you replace \n by ${S}, like

$ cat file.txt | awk 1 ORS=$S |  sed -e "s/a test${S}Please do not/not a test\nBe/" | awk 1 RS=$S > file_new.txt

( awk replaces ASCII line separator with yours and vice versa. )

sed -e'$!N;s/^\(a test\n\)Please do not be$/not \1Be/;P;D' <in >out

Just widen your window on input a little bit.

It's pretty easy. Besides the standard substitution; you need only $!N, P, and D here.

I think this is the sed solution for 2 lines matching.

sed -n '$!N;s@a test\nPlease do not@not a test\nBe@;P;D' alpha.txt

If you want 3 lines matching then ...

sed -n '1{$!N};$!N;s@aaa\nbbb\nccc@xxx\nyyy\nzzz@;P;D'

If you want 4 lines matching then ...

sed -n '1{$!N;$!N};$!N;s@ ... @ ... @;P;D'

If the replacement part in the "s" command shrink lines then a bit more complicated like this

# aaa\nbbb\nccc shrink to one line "xxx"

sed -n '1{$!N};$!N;/aaa\nbbb\nccc/{s@@xxx@;$!N;$!N};P;D'

If the repacement part grow lines then a bit more complicated like this

# aaa\nbbb\nccc grow to five lines vvv\nwww\nxxx\nyyy\nzzz

sed -n '1{$!N};$!N;/aaa\nbbb\nccc/{s@@vvv\nwww\nxxx\nyyy\nzzz@;P;s/.*\n//M;P;s/.*\n//M};P;D'
  • This should make its way to the top! I just used the "-i" instead of "-n" for the two line substitution, because that's what I need, and incidentally, it's also in the asker's example. – Nagev May 15 at 13:22

This is a small modification of xara's clever answer to make it work on OS X (I'm using 10.10):

cat alpha.txt | tr '\n' '\r' | sed -e 's/a test$(printf '\r')Please do not/not a test$(printf '\r')Be/'  | tr '\r' '\n'

Instead of explicitly using \r, you have to use $(printf '\r').

  • 1
    While printf '\r' (or echo -e '\r') do work properly, please note that you can just use the shell syntax $'\r' to refer to escaped literals. For example, echo hi$'\n'there will echo a newline between hi and there. Similarly, you can wrap the entire string so that each backslash \ will escape its subsequent character: echo $'hi\nthere' – Dejay Clayton Feb 28 at 1:40

I wanted to add a few lines of HTML to a file using sed, (and ended up here). Normally I'd just use perl, but I was on box that had sed,bash and not much else. I found that if I changed the string to a single line and let bash/sed interpolate the \t\n everything worked out:

HTML_FILE='a.html' #contains an anchor in the form <a name="nchor" />
BASH_STRING_A='apples'
BASH_STRING_B='bananas'
INSERT="\t<li>$BASH_STRING_A<\/li>\n\t<li>$BASH_STRING_B<\/li>\n<a name=\"nchor\"\/>"
sed -i "s/<a name=\"nchor"\/>/$INSERT/" $HTML_FILE

It would be cleaner to have a function to escape the double quotes and forward-slashes, but sometimes abstraction is the thief of time.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.