Extract and rearrange from file

Question

I have file which I want to extract and rearrange certain data , Old file contains a raw data this file is Input

reference:cve,2017-8962
sid:45885
reference:cve,2016-10033
reference:cve,2016-10034
reference:cve,2016-10045
reference:cve,2016-10074
sid:45917
reference:cve,2017-8046
sid:45976
reference:cve,2018-6577
reference:cve,2018-6578
sid:46062

and the below file is the New file contains the required output

reference:cve,2017-8962
sid:45885
reference:cve,2016-10033
sid:45917
reference:cve,2016-10034
sid:45917
reference:cve,2016-10045
sid:45917
reference:cve,2016-10074
sid:45917
reference:cve,2017-8046
sid:45976
reference:cve,2018-6577
sid:46062
reference:cve,2018-6578
sid:46062

.

Explanation:for eample sid:45917 there are four references they are (reference:cve,2016-10033 reference:cve,2016-10034 reference:cve,2016-10045 reference:cve,2016-10074), we need to split each reference and append sid one below the other (note: sid is always followed by reference), like this there are repetitive blocks, so if there are multiple references we need to append them in New file order.

Answer 1

As you seem to use post-ponned sid:s (multipe references: followed by their single sids: => pairs of references: and sid:), two solution.

---

Solution 1 : reversing

Simple use the tac command (it's cat in the reverse order) to reverse the input and the output : tac input | awk | tac > output

For awk part, just duplicate the sid:s:

gawk '/^sid:/{sid=$0};/^reference:/{print sid "\n" $0}'

---

Solution 2 : array

Store the reference:s in an array as they come and then spit them back out when encountering corresponding sid:

gawk 'BEGIN{r=0};/^reference:/{ref[r++]=$0};/^sid:/{for(n=0;n<r;n++){print ref[n] "\n" $0};r=0}' /tmp/test.txt

/^reference:/{ref[r++]=$0} : for each line which begins by ref... store the line in an array and move the 'r' pointer to the next element.

/^sid:/{for(n=0;n<r;n++){print ref[n] "\n" $0};r=0} : whenever a line begins with sid, walk the whole array until the r pointer (for...) and for each element, print the stored ref and the current line (=sid), then reset the r back to beginning so we begin again with the next references.

Answer 2

awk 'BEGIN { i=0; }
/^reference:/ { ref[i++] = $0; }
/^sid:/ { for(j=0; j<i; j++) { print ref[j]; print; } i=0; }' inputfile > outputfile

Explanation:

• BEGIN { i=0; } Initialize variable to make sure it is interpreted as a numeric value 0, not as an empty string "".
• /^reference:/ { ref[i++] = $0; } For every line that starts with reference: (^ is an anchor to the beginning of the line) copy the whole line $0 to an array element ref[i] and increment the index i++
• /^sid:/ { ... } for every line that startts with sid: ...
• for(j=0; j<i; j++) { ... } As i points to the array element after the last used one, loop over all array elements that have been written to using index j,
• print ref[j]; print the contents of the array element, i.e. a saved reference: line
• print; print the current line, i.e. the sid: line
• i=0; reset the array index to the beginning for the next group of reference: lines

The script is based on the following assumptions:

• The input consists of a series of block where every block contains
  • a sequence of one or more reference: lines followed by
  • a single sid: line
• The last line must be a sid: line.
• Non-matching lines will be ignored.

With the original question I assumed the wrong direction of the conversion. The second script converts in the opposite direction:

awk 'BEGIN { oldsid=""; ref=""; }
/^reference:/ { ref=$0; }
/^sid:/ { if(oldsid != $0) { if(oldsid != "") print oldsid; } if(ref!="")print ref; oldsid=$0; }
END { if (oldsid != "") print oldsid; }' inputfile > outputfile

Explanation:

• BEGIN { oldsid=""; ref=""; } Initialize variables for clarity, not really necessary.
• /^reference:/ { ref=$0; } For every line that startts with reference: save the line $0 to variable ref, don't print it yet.
• /^sid:/ { ... } For every line that starts with sid: ...
• if(oldsid != $0) { if(oldsid != "") print oldsid; } If the sid: line has changed now, the last reference: line saved in ref belongs to the new sid:, so we don't print it yet. If oldsid is not empty we can print it now as the previous block of reference: lines with the same sid: is finished. oldsid will be empty when we find the first sid:.
• if(ref!="")print ref; If we have a saved reference:, print it now. (Either we have just closed the previous block with the corresponding sid: line or we know now that the current reference: has the same sid: as the previous one.) The check for empty string is not really necessary as I assume every sid: line is preceded by a reference: line.
• oldsid=$0; save the current sid: line for the comparison when we get the next one. The current line is not printed yet.
• END { if (oldsid != "") print oldsid; } At the end print the last saved sid: line if there is any. (If the input file is empty it will not print an empty line here.)

This script is based on these assumptions:

• every reference: is followed by a sid:
• all pairs of reference: and sid: with the same sid: line follow each other