extract lines from bottom until regex match

Question

I have this output.

[root@linux ~]# cat /tmp/file.txt
virt-top time  11:25:14 Host foo.example.com x86_64 32/32CPU 1200MHz 65501MB
   ID S RDRQ WRRQ RXBY TXBY %CPU %MEM   TIME    NAME
    1 R    0    0    0    0  0.0  0.0  96:02:53 instance-0000036f
    2 R    0    0    0    0  0.0  0.0  95:44:07 instance-00000372
virt-top time  11:25:17 Host foo.example.com x86_64 32/32CPU 1200MHz 65501MB
   ID S RDRQ WRRQ RXBY TXBY %CPU %MEM   TIME    NAME
    1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
    2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

You can see it has two blocks and i want to extract last block (if you see first block it has all CPU zero which i don't care) inshort i want to extract following last lines (Notes: sometime i have more than two instance-*) otherwise i could use "tail -n 2"

1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

I have tried sed/awk/grep and all possible way but not get close to desire result.

It can help us help you if you provide some of your "all possible ways" you tried — Philippos, Sep 10 '19 at 15:39
I think i am closed sed '1,/instance/p' /tmp/foo.txt still its printing header line which is ID S RDRQ WRRQ RXBY TXBY %CPU %MEM TIME NAME i can trim it down with | grep -v ID but i believe there must be better way :) — Satish, Sep 10 '19 at 15:42
How about extracting all lines with non-zero CPU-Uage like (untested) grep -vE -e virt-top -e RDRQ -e "([^ ]* *){6}0.0"? — Philippos, Sep 10 '19 at 15:47

Stéphane Chazelas · Accepted Answer · 2019-09-10 16:07:54Z

This feels a bit silly, but:

$ tac file.txt |sed -e '/^virt-top/q' |tac
virt-top time  11:25:17 Host foo.example.com x86_64 32/32CPU 1200MHz 65501MB
   ID S RDRQ WRRQ RXBY TXBY %CPU %MEM   TIME    NAME
    1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
    2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

GNU tac reverses the file (many non-GNU systems have tail -r instead), the sed picks lines until the first that starts with virt-top. You can add sed 1,2d or tail -n +3 to remove the headers.

Or in awk:

$ awk '/^virt-top/ { a = "" } { a = a $0 ORS } END {printf "%s", a}' file.txt 
virt-top time  11:25:17 Host foo.example.com x86_64 32/32CPU 1200MHz 65501MB
   ID S RDRQ WRRQ RXBY TXBY %CPU %MEM   TIME    NAME
    1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
    2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

It just collects all the lines to a variable, and clears that variable on a line starting with virt-top.

If the file is very large, the tac+sed solution is bound to be faster since it only needs to read the tail end of the file while the awk solution reads the full file from the top.

Only the tac solution is practical if file.txt is very large and the match is (as expected) towards its end. Also, it could be better written as tac file.txt | sed '/^virt-top/q' | tac — mosvy, Sep 10 '19 at 15:55

steeldriver · Accepted Answer · 2019-09-10 15:45:00Z

4

With ed you can regex-search upward using ?pattern? in place of the usual /pattern/ (which searches from above the current position). So for example:

$ printf '%s\n' '?ID?+1,$p' q | ed -s file.txt
    1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
    2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

answered Sep 10 '19 at 15:45

steeldriver

62.3k11 gold badges82 silver badges124 bronze badges

very simple and sweet, didn't thought about printf can run the show :) – Satish Sep 10 '19 at 15:53
2

ed sucks with very large files. – mosvy Sep 10 '19 at 15:58
... because it reads the whole file into a buffer in memory before operating on it. – Ed Morton Sep 10 '19 at 16:20
2

@Ed, I suppose it depends on the implementation, but at least GNU ed, while it reads the file fully, doesn't store it whole in memory (probably only records the offsets of each line within the file which explains why it's more efficient at dealing with longer lines). It does create a temporary copy of the file though. – Stéphane Chazelas Sep 10 '19 at 16:58

Add a comment |

Stéphane Chazelas · Accepted Answer · 2019-09-10 17:06:47Z

3

If your input has a fixed number of blocks you could also do something like:

awk '/^virt-top/ && ++n == 2, 0' <your-file

To output the lines from the 2^nd occurrence of virt-top to the end of the file (0 meaning false, means the end of that first,last range is never found).

answered Sep 10 '19 at 17:06

Stéphane Chazelas

419k75 gold badges824 silver badges1245 bronze badges

Add a comment |

Kusalananda · Accepted Answer · 2019-09-10 16:06:29Z

Getting the last record from the line that starts with the string virt-top to the end using ed (suitable for dealing with files of the size comparable with what you show, not several megabytes big):

$ printf '%s\n' '?^virt-top?,$p' | ed -s file
virt-top time  11:25:17 Host foo.example.com x86_64 32/32CPU 1200MHz 65501MB
   ID S RDRQ WRRQ RXBY TXBY %CPU %MEM   TIME    NAME
    1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
    2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

Or, just the lines after the last virt-top line that contain the substring instance:

$ printf '%s\n' '?^virt-top?,$g/instance/p' | ed -s file
    1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
    2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

That last ed command, ?^virt-top?,$g/instance/p, first designates a range of lines from the last line that starts with virt-top (last, because ed starts you off on the last line of the buffer and ?re? searches backwards) to the end of the buffer ($) and applies the command g/instance/p to these lines. The g/re/p command prints all lines within the range that matches the given regular expression (that's where grep got its name from, by the way).

Using awk:

$ awk '/^virt-top/ { lines = "" } /instance/ { lines = (lines == "" ? $0 : lines ORS $0) } END { print lines }' file
    1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
    2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

This saves lines of input in lines whenever there's a line that contains the substring instance. These lines are printed at the end. Whenever a line that starts with virt-top is found, the saved lines are discarded.

Virtually the same thing with sed, using the hold space to serve as the equivalent of the lines variable in the awk code:

$ sed -n '/^virt-top.*/{ s///; x; d; }; /instance/H; ${ x; s/\n//; p; }' file
    1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
    2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

unix.stackexchange.com/questions/540029/… – mosvy Sep 10 '19 at 15:58 — mosvy, Sep 10 '19 at 15:58

Rakesh Sharma · Accepted Answer · 2019-09-11 00:43:25Z

1

Here's yet another method to deal with it:

$ sed -e '
   /\n/q
   /virt-top/{h;d;}
   H;$!d;g
   s/\n//;D
' file.txt

Results

1 R    0    0    0    0  0.6 12.0  96:02:53 instance-0000036f
2 R    0    0    0    0  0.2 12.0  95:44:08 instance-00000372

answered Sep 11 '19 at 0:43

Rakesh Sharma

8411 gold badge2 silver badges3 bronze badges

Add a comment |

Stack Exchange Network

current community

your communities

more stack exchange communities

extract lines from bottom until regex match

5 Answers 5

Your Answer

Not the answer you're looking for? Browse other questions tagged linux awk sed command-line regular-expression or ask your own question.

Hot Network Questions

extract lines from bottom until regex match

5 Answers 5

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged linux awk sed command-line regular-expression or ask your own question.

Related

Hot Network Questions