0

I have file which I want to extract and rearrange certain data , Old file contains a raw data this file is Input

reference:cve,2017-8962
sid:45885
reference:cve,2016-10033
reference:cve,2016-10034
reference:cve,2016-10045
reference:cve,2016-10074
sid:45917
reference:cve,2017-8046
sid:45976
reference:cve,2018-6577
reference:cve,2018-6578
sid:46062

and the below file is the New file contains the required output

reference:cve,2017-8962
sid:45885
reference:cve,2016-10033
sid:45917
reference:cve,2016-10034
sid:45917
reference:cve,2016-10045
sid:45917
reference:cve,2016-10074
sid:45917
reference:cve,2017-8046
sid:45976
reference:cve,2018-6577
sid:46062
reference:cve,2018-6578
sid:46062
.

Explanation:for eample sid:45917 there are four references they are (reference:cve,2016-10033 reference:cve,2016-10034 reference:cve,2016-10045 reference:cve,2016-10074), we need to split each reference and append sid one below the other (note: sid is always followed by reference), like this there are repetitive blocks, so if there are multiple references we need to append them in New file order.

  • I have no idea what your goal is. Please try explaining more precisly what your input and expected output is. – Panki Jan 16 '19 at 12:34
  • Please copy your complete input data as a code block, not as a picture. – Bodo Jan 16 '19 at 14:36
  • Kindly post the input data and required output – Praveen Kumar BS Jan 16 '19 at 15:58
  • Guys edited, Please have a look. – chandu Jan 16 '19 at 17:25
  • example and explanation are conflicting. In the source, is sid: before or after reference: ? – DrYak Jan 16 '19 at 17:33
1

As you seem to use post-ponned sid:s (multipe references: followed by their single sids: => pairs of references: and sid:), two solution.


Solution 1 : reversing

Simple use the tac command (it's cat in the reverse order) to reverse the input and the output : tac input | awk | tac > output

For awk part, just duplicate the sid:s:

gawk '/^sid:/{sid=$0};/^reference:/{print sid "\n" $0}'

Solution 2 : array

Store the reference:s in an array as they come and then spit them back out when encountering corresponding sid:

gawk 'BEGIN{r=0};/^reference:/{ref[r++]=$0};/^sid:/{for(n=0;n<r;n++){print ref[n] "\n" $0};r=0}' /tmp/test.txt

/^reference:/{ref[r++]=$0} : for each line which begins by ref... store the line in an array and move the 'r' pointer to the next element.

/^sid:/{for(n=0;n<r;n++){print ref[n] "\n" $0};r=0} : whenever a line begins with sid, walk the whole array until the r pointer (for...) and for each element, print the stored ref and the current line (=sid), then reset the r back to beginning so we begin again with the next references.

  • Solution 2 : I have tried it works , please @DrYark kindly explain me the solution 2. just osm* – chandu Jan 16 '19 at 18:23
1
awk 'BEGIN { i=0; }
/^reference:/ { ref[i++] = $0; }
/^sid:/ { for(j=0; j<i; j++) { print ref[j]; print; } i=0; }' inputfile > outputfile

Explanation:

  • BEGIN { i=0; } Initialize variable to make sure it is interpreted as a numeric value 0, not as an empty string "".
  • /^reference:/ { ref[i++] = $0; } For every line that starts with reference: (^ is an anchor to the beginning of the line) copy the whole line $0 to an array element ref[i] and increment the index i++
  • /^sid:/ { ... } for every line that startts with sid: ...
  • for(j=0; j<i; j++) { ... } As i points to the array element after the last used one, loop over all array elements that have been written to using index j,
  • print ref[j]; print the contents of the array element, i.e. a saved reference: line
  • print; print the current line, i.e. the sid: line
  • i=0; reset the array index to the beginning for the next group of reference: lines

The script is based on the following assumptions:

  • The input consists of a series of block where every block contains
    • a sequence of one or more reference: lines followed by
    • a single sid: line
  • The last line must be a sid: line.
  • Non-matching lines will be ignored.

With the original question I assumed the wrong direction of the conversion. The second script converts in the opposite direction:

awk 'BEGIN { oldsid=""; ref=""; }
/^reference:/ { ref=$0; }
/^sid:/ { if(oldsid != $0) { if(oldsid != "") print oldsid; } if(ref!="")print ref; oldsid=$0; }
END { if (oldsid != "") print oldsid; }' inputfile > outputfile

Explanation:

  • BEGIN { oldsid=""; ref=""; } Initialize variables for clarity, not really necessary.
  • /^reference:/ { ref=$0; } For every line that startts with reference: save the line $0 to variable ref, don't print it yet.
  • /^sid:/ { ... } For every line that starts with sid: ...
  • if(oldsid != $0) { if(oldsid != "") print oldsid; } If the sid: line has changed now, the last reference: line saved in ref belongs to the new sid:, so we don't print it yet. If oldsid is not empty we can print it now as the previous block of reference: lines with the same sid: is finished. oldsid will be empty when we find the first sid:.
  • if(ref!="")print ref; If we have a saved reference:, print it now. (Either we have just closed the previous block with the corresponding sid: line or we know now that the current reference: has the same sid: as the previous one.) The check for empty string is not really necessary as I assume every sid: line is preceded by a reference: line.
  • oldsid=$0; save the current sid: line for the comparison when we get the next one. The current line is not printed yet.
  • END { if (oldsid != "") print oldsid; } At the end print the last saved sid: line if there is any. (If the input file is empty it will not print an empty line here.)

This script is based on these assumptions:

  • every reference: is followed by a sid:
  • all pairs of reference: and sid: with the same sid: line follow each other
  • Sorry edited, Please check – chandu Jan 16 '19 at 17:25
  • Thanks @Bodo request you to kindly explain me the awk second snippet.Magic – chandu Jan 16 '19 at 18:24
  • @chandu: I added explanation for both awk scripts – Bodo Jan 17 '19 at 8:07
  • Thanks @Bodo for your time and so nicely explained.. – chandu Jan 17 '19 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.