1

I need to understand why sed is able to work for 1) and not for 2). Please do not post me any alternative solutions. I have already found them on this forum. I just need to understand the behavior of sed regarding point 1) and 2).

1) sed -i s/\\r//g file.txt

On checking od -c file.txt, sed has successfully removed \r

2) sed -i s/\\n//g file.txt

On checking od -c file.txt, sed has not removed \n

My question here is to just understand why its not working for point-2. Please do not post any alternative solutions. Wish to understand the internals thats it!

7

From GNU sed manual - How sed Works

sed operates by performing the following cycle on each line of input: first, sed reads one line from the input stream, removes any trailing newline, and places it in the pattern space. Then commands are executed; each command can have an address associated to it: addresses are a kind of condition code, and a command is only executed if the condition is verified before the command is to be executed.

When the end of the script is reached, unless the -n option is in use, the contents of pattern space are printed out to the output stream, adding back the trailing newline if it was removed. Then the next cycle starts for the next input line.

From POSIX spec (thanks steeldriver for the link)

In default operation, sed cyclically shall append a line of input, less its terminating newline, into the pattern space. Normally the pattern space will be empty, unless a D command terminated the last cycle. The sed utility shall then apply in sequence all commands whose addresses select that pattern space, and at the end of the script copy the pattern space to standard output (except when -n is specified) and delete the pattern space. Whenever the pattern space is written to standard output or a named file, sed shall immediately follow it with a newline.


tl;dr the input record separator (which is newline by default) is removed before executing the commands and then added back while printing the record


There are, however, cases where the newline character can be manipulated. Some examples given below:

$ # this would still not allow newline of second line to be manipulated
$ seq 5 | sed 'N; s/\n/ : /'
1 : 2
3 : 4
5

$ # here ASCII NUL is input record separator, so newline can be freely changed
$ seq 5 | sed -z 's/\n/ : /g'
1 : 2 : 3 : 4 : 5 :  

$ # default newline separator, so NUL character can be changed
$ printf 'foo\0baz\0xyz\0' | sed 's/\x0/-/g'
foo-baz-xyz-
$ # NUL character is separator, so it cannot be changed now
$ printf 'foo\0baz\0xyz\0' | sed -z 's/\x0/-/g' | cat -A
foo^@baz^@xyz^@
| improve this answer | |
  • 1
    It appears to be mandated by POSIX as well: "In default operation, sed cyclically shall append a line of input, less its terminating <newline>, into the pattern space." – steeldriver Apr 7 '18 at 13:57
  • 1
    The -z trick for newline manipulation rocks man.... – George Vasiliou Apr 7 '18 at 15:53
  • Need some clarification from the quote you made. 1) sed reads one line from the input stream, removes any trailing newline So if there were 5 lines in the input stream, it will iterate one line at a time and for each line it will remove the newline. 2) Then commands are executed What commands; sed just has search and replace ? 3) each command can have an address associated to it Since i dont really understand what are commands address doesnt make sense to me. Please clarify above points. Thanks. – LoveWithMaths Apr 8 '18 at 4:03
  • @linuxuser 1) yes, by default sed would work line by line 2) sed has lots of commands - see this section and the one that follows 3) by default, the commands you specify will act upon all lines.. but sometimes you want to apply only for specific line number(s) or lines that match a specific regex or certain groups of lines and so on.. see sed addresses section .. in short, sed has grep like filtering capability plus some more ability.. it is versatile – Sundeep Apr 8 '18 at 5:01
  • 1
    Yes i was going through sed documentation. And also realised that explaining it here would be out of scope. Thanks. – LoveWithMaths Apr 8 '18 at 5:09
2

A file for sed is a stream of lines delimited by \n. If \n is the delimiter, of course it can't handle it in substitutions.

| improve this answer | |
  • 1
    @Kusalananda thanks for rephrase. Now, text is seeing match better! I'm not native English speaker and can do some mistakes or wrong phrase order. – Yurij Goncharuk Apr 7 '18 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.