5

TARGET

How to delete all lines in a text file before a matching one, including this one?

Input file example:

apple
pear
banana
HI_THERE
lemon
coconut
orange

Desired output:

lemon
coconut
orange

The aim is to do it in sed to use the "-i" option (direct editing).

CLEAN SOLUTION ?

Most answers for similar problems propose something like:

sed -n '/HI_THERE/,$p' input_file

But the matched line is not deleted:

HI_THERE
lemon
coconut
orange

Then, knowing this will delete all from matched line (including it) to end of file:

sed '/HI_THERE/,$d' input_file

I tried something like this:

sed '^,/HI_THERE/d' input_file

But then sed complains:

sed: -e expression #1, char 1: unknown command: `^'

DIRTY SOLUTION

The last (dirty) solution is using pipeline:

sed -n '/HI_THERE/,$p' input_file | tail -n +2

but then, direct edit of the file doesn't work:

sed -n '/HI_THERE/,$p' input_file | tail -n +2 > input_file
cat input_file # returns nothing

and one must use a temporary file like that...

sed -n '/HI_THERE/,$p' input_file | tail -n +2 > tmp_file
mv tmp_file input_file
2
  • 2
    I think you wanted 1, not ^. Does that work for you? Aug 2, 2019 at 8:00
  • Ach.... it works :) (please, publish this as an answer so that I can validate it :-) )
    – taalf
    Aug 2, 2019 at 8:02

6 Answers 6

6

Similar to your "clean solution":

sed -e '1,/HI_THERE/d' input_file

The first line in the file is line 1 - there's no special ^ address because you always know that, while $ is needed for the end because you don't (necessarily) know which line that is.

This does fall over if the matching line is the first line of the file. With GNU sed you can use 0 instead of 1 to deal with that. For POSIX sed and for portability (which seem to be different in this case) it's more complex (see comments below and this follow-up question).

13
  • 2
    IIRC this will fail to match (hence deleting to the end of the file) if HI_THERE occurs in line 1 (in GNU sed, one can use 0,/HI_THERE/ to cover this special case) Aug 2, 2019 at 8:08
  • POSIX-compatibly you can use sed -e '1{/HI_THERE/d;};1,//d' if that is an issue. Aug 2, 2019 at 8:19
  • @MichaelHomer This POSIX version will not help, I'm afraid. All lines will get killed by the 1,//d anyhow.
    – Philippos
    Aug 2, 2019 at 8:43
  • 1
    I'm still not sure which is wrong, but as a minimal case printf 'a\nb\n'|sed -e '1{d;};1,/a/p' behaves how I expect on BSD/macOS sed and Solaris' POSIX sed (one output line), and not on GNU sed (two output lines), including in its POSIX mode. Busybox matches GNU sed. I will try to have a prod at more exotic systems. There is an argument for either interpretation of the specification text and I'm no longer certain which one I expect from it. Presumably Solaris and macOS have passed the conformance suites, though, so that ought to be correct. Thanks for the comments! I've just edited yours in. Aug 2, 2019 at 9:13
  • 1
    A sed -e '1,/HI/d' file fails if the first line match the regex. A sed -e '1{/HI/d;};1,//d' file will remove up to the second regex match or the whole file if no second match exists. A sed '/HI/,$!d;//d' file will remove all lines where the regex match.
    – user232326
    Aug 8, 2019 at 21:15
1

If you want to stick to Posix sed you could use this:

sed -ne '
  /HI_THERE/!d
  :loop
    n
    p
  bloop
' inp.file

Or, written in a condensed manner:

sed -n '/HI_THERE/!d;:a;n;p;ba' inp.file
0
$ perl -ne 'print if 1 <(/HI_THERE/...eof)' input_file

Where we make use of the range operator ... to form the proper range and further constrain it to reject the first element within the selected range.

0

sed

There is no simple way in sed to do what you ask for.
The simplest portable POSIXly solution for sed is something like:

sed -ne '/HI/{:1' -e 'n;p;b1' -e '}'

Other simple solutions are:

sed '0,/HI/d'      ./file             # GNU sed
awk 'f; /HI/{f=1}' ./file

ed

The closest POSIXly non-sed solution is with ed. Just remove the range from the first line of the file to the regex /HI/, even if HI is on the same first line.

printf '%s\n' 1,/HI/d ,p Q | ed -s file

Or

ed -Gs imfile2 <<-\edscript
1,/HI/d
,p
Q
edscript

Which means:

  • printing no additional information (number of lines read) with option -s
  • remove (delete) all lines from the first line of the file to the regex /HI/ (1,/HI/d).
  • then print the whole file (,p).
  • and exit even if the file has been modified (Q).

If you want to modify the file replace ,p Q with just w (write to file).

Why sed fails on 1,/HI/d while ed works ?

Because sed expects the regex to match on the next line.

In ed, either 3,3 3,/3/ /3/,3 /3/,/3/ will print only one 3:

$ printf '%s\n' 3,3p   3,/3/p   /3/3p   /3/,/3/p   Q | ed -s <(seq 5)
3
3
3
3

While sed will do this:

$ sed -ne 3,3p  <(seq 5)
3

$ sed -ne 3,/3/p  <(seq 5)
3
4
5

$ sed -ne /3/,3p  <(seq 5)
3

$ sed -ne /3/,/3/p <(seq 5)
3
4
5

A regex at the end of a range is expected to match a line following the line matched on the start of the range (the left of the ,). There is no 3 following line numbered 3, so sed prints all following lines (4 and 5).

That is why GNU sed solves the issue with 0,/HI/.

Why :1;n;p;b1 ?

A way to print all lines (except the first) of a range is to use a loop that first asks for the next line and then prints it:

$     sed -n '5{:1;n;p;b1}' <(seq 8)    # GNU syntax
6
7
8

So, simply match the regex you need /HI/ and get inside such loop.

sed -n '/HI/{:1;n;p;b1}' file        # GNU syntax

Which needs to be expanded to a more complex script because some older seds don't allow labels to be finished by the ;.

sed -n -e '/HI/{:1' -e 'n;p;b1' -e '}' file        # portable syntax
0

grep -nw HI_THERE file.txt |awk -F":" '{print $1}' | xargs -I % sed '1,%d' file.txt

Explanation: I grep using exact word w, get the line number n
Next, I pull the line number using awk, the separator is :
Further I pipe this using xargs and delete until then.

-1

sed is for doing s/old/new on individual strings that is all. For anything else you should use awk, e.g. with any awk in any shell on every UNIX box and handling "HI_THERE" appearing on any line:

$ awk 'f; /HI_THERE/{f=1}' file
lemon
coconut
orange

If you want GNU sed for -i then use GNU awk for -i inplace instead. See https://stackoverflow.com/a/17914105/1745001 for other scripts to select sections of files.

6
  • Hi, Ed, why not better <file instead of file for really odd named files?
    – user232326
    Aug 8, 2019 at 18:28
  • @Isaac Because <file strips you of the ability to access FILENAME and isn't extensible to use with multiple input files which is a very common situation when using awk. If your file names contain odd characters just quote them appropriately in either case and it makes no difference which approach you use in the call to awk. The only mild benefit to using <file is in the error case - if file doesn't exist or can't be read then if you write awk '1' foo > bar then bar is created while with awk '1' <foo > bar it isn't and you get an error message from the shell instead of from awk.
    – Ed Morton
    Aug 8, 2019 at 18:31
  • 1
    That makes sense. However, if a file is named var=value file, awk will set a variable inside awk in either awk '...' "var=value file" or in awk '...' 'var=value file' even in awk '...' "$file" or the should be avoided in any case awk '...' $file. How to quote that?. Thanks in advance.
    – user232326
    Aug 8, 2019 at 18:45
  • That's just the well-documented awk 'script' './var=value file'
    – Ed Morton
    Aug 8, 2019 at 18:53
  • 1
    Simple solution !. Got it, thanks Ed.
    – user232326
    Aug 8, 2019 at 18:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .