4

I have a file with data structured as follows

1, p1, p2, p3, p4, p5, p6, p7
2, p9, p10,,
2, q1, q2, q3, q4, q5, q6, q7
2, q9, q10,,
2, r1, r2, r3, r4, r5, r6, r7
2, r9, r10,,
1, s1, s2, s3, s4, s5, s6, s7
2, s9, s10,,
...

I want to get all the lines starting with 1 and ending with ,, so that I get

1, p1, p2, p3, p4, p5, p6, p7
2, p9, p10,,
1, s1, s2, s3, s4, s5, s6, s7
2, s9, s10,,

and if possible get it like,

1, p1, p2, p3, p4, p5, p6, p7, 2, p9, p10,,
1, s1, s2, s3, s4, s5, s6, s7, 2, s9, s10,,

How can I do this with sed or awk?

  • Edited to show a more accurate picture of data set. In my initial question the data was too ambiguous. – jasmaar Dec 30 '14 at 16:07
2

Here is a sed solution that will string together any sequence of the ,,$ lines following the last occurring ^1 line:

sed -e '/^1/{x;s/\n/ /gp;d' -e '};/,,$/H;$G;D
' <<\IN                                                                          
1, p1, p2, p3, p4, p5, p6, p7
2, p9, p10,,
2, q1, q2, q3, q4, q5, q6, q7
2, q9, q10,,
2, r1, r2, r3, r4, r5, r6, r7
2, r9, r10,,
1, s1, s2, s3, s4, s5, s6, s7
2, s9, s10,,
IN

It exchanges hold and pattern spaces on lines that begin with a ^1 and s///prints whatever the contents of hold space was previous to that only in the event of a successful substitution. Lines ending with ,,$ are appended to Hold space following a \newline character and then all lines are Deleted up to the first occurring \newline character. On the $last line Hold space is appended to pattern space following a \newline - so when it is Deleted it restarts the line cycle at the top of the script containing only what Hold space contained - which gets it printed as necessary.

OUTPUT:

1, p1, p2, p3, p4, p5, p6, p7 2, p9, p10,, 2, q9, q10,, 2, r9, r10,,
1, s1, s2, s3, s4, s5, s6, s7 2, s9, s10,,

If, on the other hand, you do not want the subsequent /,,$/ occurrences then this might do:

sed -e '/^1/{x;y/\n/ /;s/,,.*/,,/p;d' -e '};/,,$/H;$G;D'

Given the same input that prints instead:

1, p1, p2, p3, p4, p5, p6, p7 2, p9, p10,,
1, s1, s2, s3, s4, s5, s6, s7 2, s9, s10,,

But that will get /,,$/ lines printed even if they do not immediately follow a /^1/ match. If you want the pairs only if they are immediately sequential in input - you can do that as well:

sed -n '/^1/!d;$p;N;/\n1/P;/,,$/s/\n/ /p;D'

That works thus:

  • It first deletes from output all lines which do !not begin with /^1/
    • This includes lines brought in with N that do not end with /,,$/.
  • If the this is the $last input line pattern space is here printed, because the next command will end the script.
  • On /^1/ matches it appends the Next input line to pattern space following a \newline character.
  • If the appended line also begins with a /\n1/ it Prints the previous.
    • P prints only up to the first occurring \newline in pattern space.
  • After pulling in the Next input line if pattern space $ends with a /,,$/ match it s///ubstitutes a space character for the inserted \newline character and prints the results.
  • Pattern space is always Deleted up to and including the first occurring \newline character.
    • ...so when a Next input line doesn't match /,,$/ it is sent back to the top of the script as the head of the line. If at that point it does not match ^1 it is deleted completely.
    • ...because /,,$/ have already had their \newline completely removed by this point, they are removed from flow comletely here.

All of this means that if /^1/ lines follow one another they are still printed, and if lines which do not end in ,,$ follow a ^1 they are not printed.

  • sed '/^1/!d;$!N;/\n1/P;//D;/,,$/!s/\n.*//;y/\n/ /' worked perfectly. Thanks. And thanks for all the additional knowledge as well. Will note for future reference – jasmaar Dec 30 '14 at 16:52
  • @jasmaar - please see the edit. I like this better. – mikeserv Dec 30 '14 at 18:37
2

Regarding your first query, you can use -e to combine expression in sed:

~$ sed -n -e '/^1/p' -e '/,,$/p' f
1, p1, p2, p3, p4, p5, p6, p7
2, p9, p10,,
2, p9, p10,,
2, p9, p10,,
1, p1, p2, p3, p4, p5, p6, p7
2, p9, p10,,

and if you don't want the duplicate:

~$ sed -n -e '/^1/p' -e '/,,$/p' f | uniq
1, p1, p2, p3, p4, p5, p6, p7
2, p9, p10,,
1, p1, p2, p3, p4, p5, p6, p7
2, p9, p10,,

With awk, combine the regex with ;:

~$ awk '/^1/;/,,$/' f
1, p1, p2, p3, p4, p5, p6, p7
2, p9, p10,,
2, p9, p10,,
2, p9, p10,,
1, p1, p2, p3, p4, p5, p6, p7
2, p9, p10,,
  • You can combine the expressions in sed in the same way you can with awk - and you can do so portably. – mikeserv Dec 30 '14 at 16:13
  • @fredtantini: Thanks for the assistance. My bad for not providing the proper info upfront. – jasmaar Dec 30 '14 at 16:53
1

You may conditionally set ORS to get get output in desired format

awk '/^1/,/,,$/{ORS = /^1/? ", ": "\n"; print}' file
1, p1, p2, p3, p4, p5, p6, p7, 2, p9, p10,,
1, p1, p2, p3, p4, p5, p6, p7, 2, p9, p10,,

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.