5

I am testing a 2-command sed script. The first command deletes a line from the input, and the second command appends a line to the end of the input, thus:

$ cat input.txt 
Line 1
Line 2
Line 3

$ sed '/Line 1/d;$aAppended line' input.txt
Line 2
Line 3
Appended line

$ sed '/Line 2/d;$aAppended line' input.txt
Line 1
Line 3
Appended line

So far, so good. If, however, the line that I choose to delete happens to be the last line of the input, then the second command in my script never runs:

$ sed '/Line 3/d;$aAppended line' input.txt
Line 1
Line 2

I have learnt that sed reads the input line at a time, "executing" the script against each line, until there are no more lines left to read, so the fact that my script never "completes" (i.e. Appended line is never appended) kind-of makes sense.

Nonetheless, this is a problem for me (I still need to append Appended line). Is anyone familiar with this problem, and if yes, is there a "standard workaround"? I suppose that one alternative (to deleting the last line then appending a line) is to substitute:

$ sed 's/Line 3/Appended line/' file.txt
Line 1
Line 2
Appended line

Is this my only option?

0

6 Answers 6

9

The d command in sed man pages:

Delete pattern space. Start next cycle.

Since multiple sed commands are performed in the order they are provided, "starting next cycle" practically means that after it deletes the line, it stops performing any following commands that might apply to the (current cycle's) deleted line. So if Line 3 is the last line, and it gets deleted, a new cycle would start and the append command that should apply to the last line won't be performed.

In order to solve this problem, just switch the orders of the commands.

sed -e '$aAppended line' -e '/Line 3/d'

Or the standard equivalent to that GNU-specific code:

sed '$a\
Appended line
/Line 3/d'

That way, the text would be appended after the last line before Line 3 is deleted (and the cycle would end).

Note that a doesn't append the line to the pattern space; quoting POSIX, it schedules text for later output, the text [...] shall be written to standard output just before the next attempt to fetch a line of input [...], so it's not affected by d, it's not even affected by the -n option.

2
  • Why this works? I have tested it and it does. However I am reading the commands as "append the text [to the pattern space]" and "delete pattern space ..." which would together mean that the appended text would become deleted, too.
    – minorChaos
    Commented Aug 28, 2023 at 1:16
  • 3
    @minorChaos Command a actually appends text to the standard output. If you wanted to append to pattern space you would do s/$/text/. Furthermore, d deletes pattern space but also stops the execution of the rest of the script commands since it immediately starts a new cycle, by reading in the next input line. Hope this helps.
    – seshoumara
    Commented Aug 28, 2023 at 3:49
4

With the -n option to sed you can turn the problem from what to delete into what to print.

#!/usr/bin/sed -nf
/Line 3/!p
$aAppended line

Note that /text/! is a negation, i.e. match only the lines not containing text. And p has no side effect of stopping the execution of the rest of the script, no new cycle happens.

2

Others have already explained why this happens, so I will just provide a different solution, using perl, which doesn't have this issue, instead of sed:

$ cat input.txt 
Line 1
Line 2
Line 3
$ perl -lne 'print unless /Line 3/; END{print "Appended line"}' input.txt 
Line 1
Line 2
Appended line
4
  • Nice. I should have learn perl better to discover this gem. If various tools are to compete, there is a solution that uses just grep: cat input.txt | { grep -v 'Line 3'; echo Appended line; }
    – minorChaos
    Commented Aug 28, 2023 at 11:23
  • @minorChaos nice! Note that that would be better expressed as {grep -v 'Line 3' input.txt ; echo Appended line;} to avoid the UUoC.
    – terdon
    Commented Aug 28, 2023 at 11:51
  • Yes. (We can then also remove { and }.)
    – minorChaos
    Commented Aug 28, 2023 at 12:19
  • @minorChaos no, you still need those to be able to redirect the output to a new file. If you're not redirecting, you never needed them in the first place and could have done cat input.txt | grep -v 'Line 3'; echo Appended line directly, without the grouping anyway.
    – terdon
    Commented Aug 28, 2023 at 12:26
2

According to man sed, this is the meaning of d:

  d      Delete pattern space.  Start next cycle.

Your task can be accomplished by any of the following

$ cat input.txt | { sed '/Line 1/d;';  echo "Appended line"; } 

$ { sed '/Line 1/d;' input.txt;  echo "Appended line"; }           # by @terdon

$ SedAppend() { sed '/Line 1/d;' "$1";  echo "Appended line"; } 
$ SedAppend input.txt

provided you indeed wish to delete all lines that contain string "Line 1".

1

Any time you find yourself using sed constructs other than s, g, and p (with -n), you're better off just using awk (the other mandatory POSIX text processing tool), for some combination of clarity, robustness, portability, etc.

Using any awk in any shell on every Unix machine:

$ awk 'NR!=1; END{print "Appended line"}' input.txt
Line 2
Line 3
Appended line

$ awk 'NR!=2; END{print "Appended line"}' input.txt
Line 1
Line 3
Appended line

$ awk 'NR!=3; END{print "Appended line"}' input.txt
Line 1
Line 2
Appended line
0

Like terdon, I will take your question "Is this my only option?" to mean that something besides sed may be an option.

The answers showing you how to use sed properly are correct for your purpose. I will show how to do this with shell scripting, so there is greater flexibility and there are more possibilities.

Your input.txt is given.

#!/bin/bash  # or wherever your bash is

IFS=$'\012'; input=( $(<input.txt) ); unset IFS # read the file into an array, line by line

rm input.txt # your file is stored; this is not dangerous :-) But be careful.

y=${#input[@]} # number of elements in the array: y=3

z=2

input[$z]="Appended line" # works for your example. Zero-based array.

# z=$((y-1)) and you can calculate that if the file has an unknown number of lines

x=0; while [[ $x -lt $y ]]; do echo ${input[$x]}>>input.txt;x=$((x+1));done

cat input.txt
line 1
line 2
Appended line

There are advantages to this approach. You can modify any line in the file by computing z on an ad hoc basis. If you want to put your text in place of "line 2" then z=1. This is useful if the lines in your file are ordered data and you need to modify a line of data. You can substitute the last line of the file, no matter how many lines there are. You can "append" a line by setting z equal to y, adding an element to the array, and recomputing y. "echo >>" is easier!

sed is useful for some things. I rarely use it. As you might guess, I use this kind of text processing all the time.

5
  • 1
    You should copy/paste that into shellcheck.net and fix the issues it'll tell you about. Also consider using (( x < y )) instead of [[ $x -lt $y ]]. This would still end up orders of magnitude slower as well as much longer, more complicated, and less portable than an awk script though.
    – Ed Morton
    Commented Aug 29, 2023 at 9:50
  • Thank you. I knew this would be criticized. It is a system I use. I use "<" and ">" as redirection operators and stick to the standards of the test command: "arg1 OP arg2 Arithmetic tests. OP is one of -eq, -ne, -lt, -le, -gt, or -ge." As for shellcheck, I have used it on occasion and it throws errors that do not need correction. I use this system to avoid using sed and awk. The problem with sed and awk is not that they have too few options -- as was originally said about sendmail.
    – Wastrel
    Commented Aug 29, 2023 at 13:48
  • Your script will fail in various ways in all but some sunny-day scenarios, e.g. if the input contains formfeeds, or globbing chars, or tabs, or chains of blanks, and if anything at all goes wrong during processing will wipe out the input file. It's hard to get text manipulation right with a shell script and even if/when you can it's immensely slow as that's just not the task shell was invented to do - the people who invented shell to create/destroy files and processes and sequence calls to tools also invented sed and awk for shell to call to manipulate text. You should listen to shellcheck.
    – Ed Morton
    Commented Aug 29, 2023 at 14:46
  • I only use this method in sunny-day scenarios. The OP's text file qualifies.
    – Wastrel
    Commented Aug 29, 2023 at 14:55
  • We have no idea what the OPs files contain (I'm pretty sure it's not always the exact text Line 1, etc.) so YMMV. Anyway, all the best with your coding.
    – Ed Morton
    Commented Aug 29, 2023 at 15:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .