9

Take the following file :

$ cat f1
stu vwx yza
uvw xyz abc
abc def ghi
def ghi jkl
ghi jkl mno
jkl mno pqr
mno pqr stu
pqr stu vwx
stu vwx yza

To print all lines from the first one containing abc to the first one containing mno with GNU sed :

$ sed -n '/abc/,/mno/p' f1
uvw xyz abc
abc def ghi
def ghi jkl
ghi jkl mno

How could I print all lines until the last one containing mno, e.g. how could I get the following result :

uvw xyz abc
abc def ghi
def ghi jkl
ghi jkl mno
jkl mno pqr
mno pqr stu

In other words, is there a way to make GNU sed's range selection greedy ?

Update

In my setting :

  • If mno is missing, it should print out everything until the end of the file.
  • mno cannot occur before the first abc.
  • There's always at least one abc, and abc and mno are never on the same line

EDIT I just added a dummy stu vwx yza line at the start, so that the file doesn't start with a line including abc (to avoid solutions that start from the first line - they should start from the first line having abc in it)

4
  • One problem here is that this might require holding an arbitrary number of lines in memory, however it might be implemented.
    – muru
    Sep 13, 2021 at 9:59
  • is the input always going to be in a file or could it come from a pipe? If it's not always stored in a file then that would rule out several of the solutions posted so far.
    – Ed Morton
    Sep 13, 2021 at 13:01
  • 1
    @EdMorton : in my case the input is always a file
    – ChennyStar
    Sep 13, 2021 at 14:00
  • @αғsнιη : In my specific case there's always at least one abc, and abc and mno are never on the same line. Only mno can be missing. I'm not looking for a solution that can handle all theoretic cases. So far I have 3 awk, 2 sed and 1 perl solutions that are all perfectly working.
    – ChennyStar
    Sep 13, 2021 at 15:34

10 Answers 10

6

You could use awk if it is an option. You could mark the lines for pattern start and pattern stop and print those lines in one-pass of the file (involves storing lines starting from the first line containing abc till the last line in a buffer)

awk '/abc/ && !start {
  start = NR
}
/mno/ {
  stop = NR
}
start { line[NR] = $0 }
END {
  if ( !stop ) {
    stop = NR
  }
  for ( s = start; s <= stop; s++ )
    print line[s]
}' file

Note that this will not work, when the start pattern is not present, printing only a series of blank lines.

0
5
sed '/abc/,$!d;0,/mno/b;:1;/mno/b;$d;N;b1' file

Work algorithm:
Two address ranges are used.
The first /abc/,$!d; removes everything up to the first pattern match.
The second 0,/mno/b; up to a match with the pattern /mno/, sends each line buffer(pattern space) to the output bypassing the remaining script, thereby preventing the deletion if the pattern is not found in the file.
The rest of the script :1;/mno/b;$d;N;b1 works in a loop. In the editor buffer, lines are appended until a pattern match occurs. If a /mno/ pattern is encountered, the entire buffer is sent to the output, bypassing the rest of the script. If no match occurs, the buffer is deleted at the last line.

2
  • 1
    A lot of nice solutions have been given (awk, Perl, sed and ex). I mark this one as accepted because (a) my question was about sed, and (b) it's nice and short (there's also Philippos' sed solution below, but it's a little longer). Quasimodo's ex solution is nice and short too (printf '%s\n' '1?mno?' ke $/abc/ ks "'s,'ep" | ex file). awk and perl solutions seem to be significantly longer than sed or ed
    – ChennyStar
    Sep 14, 2021 at 15:02
  • 1
    Of course, the shorter they are the more arcane (less readable) they are. (And I say this as someone who regularly uses sed and, occasionally, ex.) It's all about trade-offs. :)
    – B Layer
    Sep 18, 2021 at 14:03
4

Another awk solution with less buffering:

awk '!f&&/abc/{f=1} f==1; f==2{buf=buf $0 ORS} f&&/mno/{f=2; printf "%s",buf; buf=""}' input.txt
  • This will print everything starting with the first occurence of abc (where it sets a flag f to 1) up to and including the first occurence of mno. The f==1 statement outside of the rule blocks instructs awk to print the current line as long as f is set to 1.
  • Then, the content of all lines after each occurence of mno (where f now has the value 2) is stored in a buffer buf, which is printed and cleared at the next occurence of mno. To ensure we correctly cope with situations where the first mno occurs before the first abc, we demand that f be set at least to 1 before applying that logic.

It will therefore store at most the text between two occurences occurence of mno, or the last occurence of mno and end-of-file (only that the latter part will never be printed).

If you want to exchange speed with memory efficiency, the following two-pass method won't rely on buffering at all:

awk 'FNR==NR{if (/abc/&&!start) {start=FNR} else if (/mno/) {end=FNR}; next} FNR>=start&&(!end||FNR<=end)' input.txt input.txt

This will process the file twice (hence it is specified twice as argument).

  • The first time, when FNR, the per-file line-counter is equal to NR, the global line counter, we look for the first occurence of abc and the last occurence of mno, and store their line numbers in start and end, respectively.
  • In the second pass, we print lines as long as the FNR counter is between (and including) start end end (or simply if it is greater/equal than start if end is unset).
0
3

I don't think sed can be made greedy, no. A possible workaround, simple but inefficient, would be to process the file twice. Once to get the line range and another to print. For instance:

$ perl -lne '$s||=$. if /abc/; $e=$. if /mno/; }{ print "$s $e"' file | 
    while read start end; do sed -n "$start,${end}p" file; done
abc def ghi
def ghi jkl
ghi jkl mno
jkl mno pqr
mno pqr stu

Alternatively, if you want to handle cases where either one or both patterns are missing:

perl -lne '$s||=$. if /abc/; $e=$. if /mno/; }{ $s||=1; $e||=$.; print "$s $e"' file | 
    while read start end; do sed -n "$start,${end}p" file; done

If abc isn't found, it will print from the beginning of the file. If mnc is not found, it will print from abc (or the beginning, if abc isn't there) until the end. If neither pattern is found it will, of course, print nothing.

0
2

You can collect all lines, starting from the abc line in hold space and then use the greedy nature of .* to delete everything after the last mno:

sed '/abc/,$!d;H;$!d;x;s/\n//;s/\(.*mno[^\n]*\).*/\1/'
  • /abc/,$!d is to delete everything before the first abc line (or the whole file, if there is no abc line at all)
  • H;$!d is the classical pattern to collect the whole file in the hold space (note that this can be a problem for very big files)
  • we xchange buffers instead of using g to avoid copying a large buffer
  • s/\n// removes the wrong newline at the beginning, produced by appending to the empty hold space
  • s/\(.*mno[^\n]*\n\).*/\1/ removes everything after the last mno line (or prints the whole remaining file, if there is no mno line, as requested). Note that [^\n] is not POSIX and will work only on some versions like GNU sed.
0
2

Collect lines until a mno sequence makes them ready to print:

sed -e '/abc/,$!d;:loop' -e'/mno/{p;d;}' -e '$d;N;bloop'
  • /abc/,$!d deletes everything except for the range from first abc line to the end. This also handles the case when there is no abc at all.
  • Then we need to :loop.
  • /mno/{p;d;} if there is mno in the pattern space, print and start over.
  • $d if we reach the last line without mno, delete everything in the buffer. Unfortunately, this means no output, if there is no mno at all.
  • Otherwise append the Next line and continue the loop.
2
  • @ikkachu : exact, I didn't see that one. Yes, you are right, this answer doesn't exactly give me what I'm looking for. I tested all solutions, so far the only one that handles that case is terdon's Perl one. But it doesn't handle the case where mno is missing (see AdminBee's comment above)
    – ChennyStar
    Sep 13, 2021 at 11:23
  • 1
    Doesn"t handle the case where mno is missing
    – ChennyStar
    Sep 13, 2021 at 14:06
1

Using GNU sed Note: first abc line does not have mno as per OP so we can exploit that fact in the Below sed code.

sed -e '
  /abc/,$!d
  /mno/{h;b;}
  $!{N;s/^/\n/;D;}
  x;/./d;x
' file

In this method we use the slurp mode -z to read in the full file in the pattern space. Then we delete till before the first line containing abc. After that reach the last mno line using the greediness of regex.

sed -Ez '
  s/abc/\x0&/
  s/.*\n(.*)\x0/\1/
  s/(.*mno[^\n]*\n).*/\1/
' file

Yet another way is a two-pass approach where we record the line numbers of the first abc line and the last mno line. In case no mno present we fill in $ in it's place. Then using these two numbers we construct a sed command begin,endp;endq

sed -n '/abc/{=;:a;n;/mno/=;ba}' file |
sed -En '
  1{h;$s/.*/$/;}
  ${x;G;}
  s/\n(.*)/,\1p&q/p
' | sed -nf - file

We can use perl to slurp the file and then the whole file is one long string which we burn from both ends and stop when our conditions are met.

perl -0777 -pe '
  s/^.*\n// until /^.*abc/;    /mno/||next;
  s/.*\n$// until /mno.*$/;
' file
1

Here are more ways using the sed editor to get the desired output.

sed -n '
  /\n/{/mno/!d;P;D;}
  /abc/,$H;$!d
  z;x;G;/mno/D
  s/.//;s/.$//p
' file
  • Save the file from the first /abc/ till eof in the hold space.
  • print the top of pattern space while we still can see /mno/ anywhere in it.
  • Then clip the top of pattern space and repeat previous step.
  • Stop when /mno/ is no longer visible.
  • Alternative ly, before beginning this P;D cycle if there is no /mno/ then just print the whole of hold space.

Another method where we store lines in hold only until /mno/ is seen. At which point we flip and print what was in hold.

sed -n '
  /abc/,$!d
  /mno/!{H;ba;}
  x;p;:a
  ${x;//P;//!p}
' file | sed 1d

Here is the Python way using the versatile itertools module groupby method to get the job done.

python3 -c 'import sys, itertools as it
ifile,start,stop = sys.argv[1:]
G,K,F = [],[],lambda x: x.find(stop)
with open(ifile) as f:
  for _ in f:
    if not _.find(start): continue
    for t in it.groupby(f,F):
      G.append(list(t[1]))
      K += [t[0] > -1]
if len(K) > 1 and not K[-1]: G.pop()
print(*[e for L in G for e in L], sep="",end="")
' file "abc" "mno"
1

Using Raku (formerly known as Perl_6)

raku -e '(S:g/ <( ^ .*? $$ \n )> ^^ .*? abc .*? $$ // andthen S:g/ ^^ .* mno .*? $$  <( .*? $)> //).put for lines.join("\n");'

Sample Input:

1. stu vwx yza
2. uvw xyz abc
3. abc def ghi
4. def ghi jkl
5. ghi jkl mno
6. jkl mno pqr
7. mno pqr stu
8. pqr stu vwx
9. stu vwx yza
10. mno pqr stu
11. xyz xyz xyz

Sample Output:

2. uvw xyz abc
3. abc def ghi
4. def ghi jkl
5. ghi jkl mno
6. jkl mno pqr
7. mno pqr stu
8. pqr stu vwx
9. stu vwx yza
10. mno pqr stu

Note above that Sample Output is a return of lines 2-through-10 when fed the 11 line Sample Input. Also, when the Sample Input is truncated to only lines 1-through-10 (i.e. with mno on the last line), the Raku code above still (correctly) returns lines 2-through-10.

Thanks to @ImHere and @ChennyStar for prodding me in the comments to come up with a more robust Raku solution.

https://raku.org

6
  • Nope, add the line aaa abc bbb at the end of your example and see what it does.
    – done
    Sep 15, 2021 at 17:41
  • 1
    It seems quite surprising that this solution doesn't print lines 8 and 9 and then prints line 10. That is not one (or a) range. The OP question did ask for a range from the first abc found to the last mno found. So, technically, an abc after the last mno should not be printed. YMMV
    – done
    Sep 15, 2021 at 19:25
  • Answers that work with only one input are not that much useful. Given that you are set in showing that raku is very capable. And given that raku is a high level language, a solution that works for simple conditions should be possible.
    – done
    Sep 16, 2021 at 18:56
  • I just noticed that this solution doesn't work if the last line contains mno, which clearly hasn't be excluded by the OP's (me!) use case. So this solution doesn't cover my use case.
    – ChennyStar
    Oct 15, 2021 at 11:53
  • @ImHere edited above. Oct 17, 2021 at 2:17
0

Short

If abc always exists, this will work (with GNU ed):

printf '%s\n' '?mno?;#' ke '$;#' "/abc/,'ep" | ed -s file

Long answer

No, there is no way AFAICT to make a range greedy in sed.

In fact, sed ends being kind of awkward to perform this task.

A simpler tool ed could do the task quite easily as it has a way to search backward by re: ?re?.

printf '%s\n' '1;#kb' '$;#ke' '?mno?;#' ke '$;#' '1;/abc/#' 'kb' "'b,'e p" |
    ed -s file

Will print:

uvw xyz abc
abc def ghi
def ghi jkl
ghi jkl mno
jkl mno pqr
mno pqr stu

As requested, and will print from the beginning (if abc is not found) until the end (if mno is not found) plus an additional ? for each limit that is not found.

Description

  • Select the first line as the begin 1;#kb
  • Select the last line as the default end $;#ke
  • Change end to the line where mno is found (if found): '?mno?;#' ke.
  • Reset present line selected to the last line $;#
  • Change begin to the line where abc is found: '1;/abc/#' 'kb'
  • Print the begin - end range: "'b,'e p".
3
  • Just as mine, your solutions fail if the last line contains mno.
    – Quasímodo
    Sep 13, 2021 at 13:14
  • I'll solve it tomorrow .... no time now. @Quasímodo –
    – done
    Sep 13, 2021 at 13:41
  • This answer is incomplete, work in progress ....
    – done
    Sep 15, 2021 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.