5

I have a large file to parse and reformat, preferably with sed (under bash). The file contains repetitive sequences starting with PATTERN_START and ending with PATTERN_END. These sequences are intermixed with other text that I have to keep unchanged. In the sequences, there are several records (numbered from 1 to n, where n can be from 1 to 12). A record is a group of lines beginning with a line of the form Record i, where i is an integer between 1 and n, and ends with another such line (Record (i+1)) or a PATTERN_END line. The length of a record can be from 1 line to 30 lines.

Here's a generic representation of an input file:

unrelated data          (possibly many lines)                      ⎤
PATTERN_START                                                      |
Record 1                                  ⎤                        |
data for Record 1(up to 30 lines)    |                        |  (many repetitions)
      ︙           ⎦                      |  (up to 12 records)    |
Record 2                                 |                        |
data for Record 2                        ⎦                         |
PATTERN_END                                                       ⎦
unrelated data          (possibly many lines)

So, I would like, ONLY for the records located between PATTERN_START and PATTERN_END, to have all the data lines of each record gathered on the Record line.

Can anybody help?

Hereunder is a sample of the file that I have to parse, and the kind of result I would like to have:

Input

Blabla
Blabla
PATTERN_OTHER
Record 1         <- record not between PATTERN_START and PATTERN_END tags => do not touch it
Data
Data
PATTERN_END
Blabla
PATTERN_START
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Record 3         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Data
Data
PATTERN_END
Blabla
Blabla
Blabla
Blabla
PATTERN_START
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Data
PATTERN_END
Blabla
Blabla
PATTERN_OTHER
Record 1         <- record not between PATTERN_START and PATTERN_END tags => do not touch it
Data
Data
Record 2         <- record not between PATTERN_START and PATTERN_END tags => do not touch it
Data
PATTERN_END
Blabla
Blabla
PATTERN_START
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Data
PATTERN_END
Blabla
Blabla

Output

Blabla
Blabla
PATTERN_OTHER
Record 1         <- was not between PATTERN_START and PATTERN_END tags => not modified
Data
Data
PATTERN_END
Blabla
PATTERN_START
Record 1 Data Data Data        <- record data grouped in one line
Record 2 Data Data             <- record data grouped in one line
Record 3 Data Data Data Data   <- record data grouped in one line
PATTERN_END
Blabla
Blabla
Blabla
Blabla
PATTERN_START
Record 1 Data Data Data        <- record data grouped in one line
PATTERN_END
Blabla
Blabla
PATTERN_OTHER
Record 1         <- was not between PATTERN_START and PATTERN_END tags => not modified
Data
Data
Record 2         <- was not between PATTERN_START and PATTERN_END tags => not modified
Data
PATTERN_END
Blabla
Blabla
PATTERN_START
Record 1 Data                  <- record data grouped in one line
Record 2 Data Data Data        <- record data grouped in one line
PATTERN_END
Blabla
Blabla
  • consider using awk instead of sed – Skaperen Jun 22 '15 at 10:27
  • special characters in either pattern could have an impact on ways to do this – Skaperen Jun 22 '15 at 10:29
8

Think this is what you want using GNU sed

 sed -n '/^PATTERN_START/,/^PATTERN_END/{
         //!{H;/^Record/!{x;s/\n\([^\n]*\)$/ \1/;x}};
         /^PATTERN_START/{h};/^PATTERN_END/{x;p;x;p};d
         };p' file

Explanation

sed -n #Non printing


'/^PATTERN_START/,/^PATTERN_END/{
#If the line falls between these two patterns execute the next block

  //!{
  #If the previous pattern matched from the line above is not on matched(so skip 
         the start and end lines), then execute next block

        H;
        #append the line to the hold buffer, so this appends all lines between 
       #`/^PATTERN_START/` and `/^PATTERN_END/` not including those.

        /^Record/!{
        #If the line does not begin with record then execute next block

            x;s/\n\([^\n]*\)$/ \1/;x
            #Swap current line with pattern buffer holding all our other lines 
            #up to now.Then remove the last newline. As this only executed when 
            #record is not matched it just removes the newline from the start 
            #of `data`.
            #The line is then put switched back into the hold buffer.

        }
        #End of not record block

    }; 
    #End of previous pattern match block

    /^PATTERN_START/{h};

    #If line begins with `PATTERN_START` then the hold buffer is overwritten 
    #with this line removing all the previous matched lines.

    /^PATTERN_END/{x;p;x;p}
    #If line begins with `PATTERN_END` the swap in our saved lines, print them,
    #then swap back in the PATTERN END line and print that as well.

    ;d
    #Delete all the lines within the range, as we print them explicitly in the 
    #Pattern end block above


         };p' file
         # Print everything that's not in the range print, and the name of the file
  • 1
    Thanks a lot User112638726, terdon, rpax, nitishch, and don_crissti ! – Syl87 Jun 23 '15 at 14:58
6

This was the best I could come up with:

sed -n '/^PATTERN_START/, /^PATTERN_END/{
            /^PATTERN_START/{x;s/^.*$//;x};
            /^Record/{x;/^\n/{s/^\n//p;d};s/\n/ /gp};
            /^PATTERN_END/{x;/^\n/{s/^\n//p;d};s/\n/ /gp;g;p};
            /^Record/!H
        };   
        /^PATTERN_START/, /^PATTERN_END/!p'

Explanation

I assume you are familiar with the idea of hold space and pattern space in sed. In this solution, we will be doing lot of manipulations in pattern space. So, first point is to disable automatic printing with -n option and print wherever required.

First task is to join all the lines that are between Record lines.

Consider the following file:

a
b
Record 1
c
d
Record 2
e
f
Record 3

After joining lines, we want it to be

a
b
Record 1 c d
Record 2 e f
Record 3

So, here is the plan:

  1. We read a line, append it to the hold space.
  2. If the line starts with Record, it means that the previous record has finished and a new record has started. So we print out the hold space, flush it and start with point 1 again.

Point 1 is implemented by the code /^Record/!H (5th line in the command). What it means is "if the line doesn't start with Record, add a new line to the hold space and append this line to the hold space".

Point 2 can be implemented by the code /^Record/{x;s/\n/ /gp;} where x swaps hold and pattern spaces, s command replaces all \ns with s and p flag prints the pattern space. Usage of x also has the advantage that now the hold space contains the current Record line so that we can begin another cycle of points 1 and 2.

But, this has a problem. In the given example, there are two lines a b before the first Record line. We don't want to substitute \n for in these lines. Since they don't begin with Record, according to point 1, \n is added to hold space and then these lines are appended. So, if the first character of the hold space is \n, it means that no Record has been encountered before and we should not substitute \n with . This is done with the command

/^\n/{s/^\n//p;d}

So the entire command becomes

/^Record/{x;/^\n/{s/^\n//p;d};s/\n/ /gp};

Now, the second complication is, we want to join lines, even if a Record line is not terminated by a Record line but by a PATTERN_END line. We want to do the exact same things as in point 2, even when the line starts with PATTERN_END. So the command becomes

/^PATTERN_END/{x;/^\n/?s/^\n//p;d};s/\n/ /gp}

But, there is a problem with this. As in the case of Record lines, the PATTERN_END line now ends up in the hold space. But we know that there will be no more joining of lines after PATTERN_END line. So, we can print this out. So, we bring the PATTERN_END line to pattern space with g and print it with p. So the final command becomes

/^PATTERN_END/{x;/^\n/?s/^\n//p;d};s/\n/ /gp;g;p}

Next issue is with the PATTERN_START lines. In the above explanation we assumed that at the start, hold space is empty. But after a PATTERN_END, there is something in the hold space. (That something is just PATTERN_END line). When we start a new cycle with PATTERN_START, we want to clear the hold space.

So, what we do is when we encounter PATTERN_START, swap the contents of hold and pattern spaces, clear the pattern space and swap again. This makes hold space clean. This is exactly what the following command does:

/^PATTERN_START/{x;s/^.*$//;x}

The final stroke is that we want to do all this fiddling only between PATTERN_START and PATTERN_END lines. Others, we just print them. This is done by the commands

/^PATTERN_START/, /^PATTERN_END/{
    ----above commands go here----
};
/^PATTERN_START/, /^PATTERN_END/!p

Put all these together and this gives the final command :)

3

Other ways with sed:

sed '/PATTERN_START/,/PATTERN_END/{   # in this range
//!{                                  # if not start or end of range
/^Record/{                            # if line matches Record
x                                     # exchange pattern and hold space
/^$/d                                 # if pattern space is empty, delete it
s/\n/ /g                              # replace newlines with spaces
}
/^Record/!{                           # if line doesn't match Record
H                                     # append it to hold space
d                                     # then delete it
}
}
/PATTERN_END/{                        # at end of range
x                                     # exchange pattern and hold space
s/\n/ /g                              # replace newlines with space
G                                     # append hold space to pattern space
x                                     # exchange again
s/.*//                                # empty pattern space
x                                     # exchange again > empty line in hold space
}
}' infile

or

sed '/PATTERN_START/,/PATTERN_END/{     # same as above
//!{                                    # same as above
: again
N                                       # pull the next line into pattern space
/\nRecord/!{                            # if pattern space doesn't match this
/\nPATTERN_END/!{                       # and doesn't match this either
s/\n/ /                                 # replace newline with space
b again                                 # go to : again
}
}
P                                       # print up to first newline
D                                       # then delete up to first newline
}
}' infile

In one line:

sed '/PATTERN_START/,/PATTERN_END/{//!{/^Record/{x;/^$/d;s/\n/ /g};/^Record/!{H;d}};/PATTERN_END/{x;s/\n/ /g;G;x;s/.*//;x}}' infile

and

sed '/PATTERN_START/,/PATTERN_END/{//!{: again;N;/\nRecord/!{/\nPATTERN_END/!{s/\n/ /;b again}};P;D}}' infile
2

A Perl way:

perl -lne 'if(/^PATTERN_START/){$i=1; %data=(); print; next};
           if(/^PATTERN_END/){
             $i=0;
             if(defined($r)){
               print "$_ @{$data{$_}}" for sort keys(%data)
             }         
           }
           if($i==1){
            if(/^(Record\s*\d+)/){$r=$1;}
            else{push @{$data{$r}},$_}
           }   
           else{print}' file

And the same thing as a commented script:

#!/usr/bin/env perl

## Read the input file line by line (this is the same as the `-n` flag)
while (<>) {
    ## If this line matches the start pattern, set $i to 1, clear
    ## the %data hash, print the line and skip to the next one.
    if(/^PATTERN_START/){
        $i=1;
        %data=();
        print;
        next
    }
    ## If this line matches the end pattern, set $i to 0
    if(/^PATTERN_END/){
        $i=0;
        ## If we're at the end pattern, we may have saved some
        ## data. This is the case if $r is defined.
        if(defined($r)){
            ## Print the current line, then the saved data associated
            ## with each of the records of this section.
            print "$_ @{$data{$_}}\n" for sort keys(%data)
        }          
    }
    ## If $i is 1, if we're inside a START and END pattern.
    if($i==1){
        ## Remove trailing newlines. This is equivallent to `-l` with
        ## the difference that `-l` also adds a newline to each print()
        ## call. We will add them manually here. 
        chomp;
        ## If this line starts with Record, 0 or more spaces and one
        ## or more numbers, save that as $r. 
        if(/^(Record\s*\d+)/){$r=$1;}
        ## If this line does not start with the above pattern, add it
        ## to the data saved for the current record. 
        else{push @{$data{$r}},$_}
    }
    ## For all other lines, print them as is. 
    else{print}
}
0

I did three versions of this.


v1


sed     -e'/^PATTERN_START/!b'  -e:n -eN  \
        -e'/\nPATTERN_END$/!bn' -eh\;s/// \
        -e'x;s/\n[[:print:]]*$//;x'       \
        -e's/\(\nRecord [[:print:]]*\)\{0,1\}\n/\1 /g'  \
        -e'G;P;D'       data

That one prints out the whole file after only applying edits to the Record lines which occur between PATTERN_{START,END}.


v2


sed   -ne'/\n/P;:n'    \
       -e'/^PATTERN_[OS]/!D'   -eN     \
       -e'/\nPATTERN_END$/!bn' -es///  \
       -e'/^PATTERN_S/s/\(\nRecord [[:print:]]*\)\{0,1\}\n/\1 /g'      \
       -eG\;D  ./data                 ###<gd data> haha

That one prints Record lines within either of PATTERN_{(START|OTHER),END} but only applies edits to those which occur between PATTERN_{START,END}.


v3


sed   -ne'/\n/P;:n'    \
       -e'/^PATTERN_START/!D'  -eN     \
       -e'/\nPATTERN_END$/!bn' -es///  \
       -e's/\(\nRecord [[:print:]]*\)\{0,1\}\n/\1 /g'      \
       -eG\;D  ./data

And that one only edits and only prints Record lines which occur between PATTERN_{START,END}.

What follows is the output of each after having run your input sample through it. The output samples are presented in reverse order, tbough shortest to longest.


v3


Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data
Record 3         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data Data
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data

v2


Record 1         <- record not between PATTERN_START and PATTERN_END tags => do not touch it
Data
Data
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data
Record 3         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data Data
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data
Record 1         <- record not between PATTERN_START and PATTERN_END tags => do not touch it
Data
Data
Record 2         <- record not between PATTERN_START and PATTERN_END tags => do not touch it
Data
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data

v1


Blabla
Blabla
PATTERN_OTHER
Record 1         <- record not between PATTERN_START and PATTERN_END tags => do not touch it
Data
Data
PATTERN_END
Blabla
PATTERN_START
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data
Record 3         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data Data
PATTERN_START
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Record 3         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Data
Data
Blabla
Blabla
Blabla
Blabla
PATTERN_START
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data
PATTERN_START
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Data
Blabla
Blabla
PATTERN_OTHER
Record 1         <- record not between PATTERN_START and PATTERN_END tags => do not touch it
Data
Data
Record 2         <- record not between PATTERN_START and PATTERN_END tags => do not touch it
Data
PATTERN_END
Blabla
Blabla
PATTERN_START
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line Data Data Data
PATTERN_START
Record 1         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Record 2         <- record between PATTERN_START and PATTERN_END tags => to put in one line
Data
Data
Data
Blabla
Blabla

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.