2

I have the below input (a thousand lines). I want a sed command to merge duplicated lines with the same first three fields which adds only different fields or deletes "N/A":

D04005;4;279;0;0;SSM-4-1
D04005;5;40;0;0;SSM-5-1
LE040A;1;363;(26.3);N/A;SM-1-1
LE040A;1;363;(27.4);N/A;SM-1-2

Expected output:

D04005;4;279;0;0;SSM-4-1
D04005;5;40;0;0;SSM-5-1
LE040A;1;363;(26.3);(27.4);SM-1-1/SM-1-2
  • Are they always going to be sequential? – mikeserv Jun 30 '15 at 17:14
  • 3
    Why sed? It is way simpler with awk. – jimmij Jun 30 '15 at 17:19
  • @ mikeserv: yes they have to be sequential – Tony Jun 30 '15 at 17:26
  • In your last output line the last 2 fields are separated with a '/'. Is that a typo? Should it be ';'? – Mel Jun 30 '15 at 17:27
  • @ jimmij: its ok with awk. I just thought SED is faster since I have thousands of lines – Tony Jun 30 '15 at 17:27
3
sed ':n
    s|;N/A;|;|g;$!N
    s|^\(\([^;]*;\)\{3\}\)\(.*\)\n\1|\1\3;|;tn
    P;D
'   <<\IN
D04005;4;279;0;0;SSM-4-1
D04005;5;40;0;0;SSM-5-1
LE040A;1;363;(26.3);N/A;SM-1-1
LE040A;1;363;(27.4);N/A;SM-1-2
LE040A;1;363;(28.5);N/A;SM-1-3
LE040A;1;363;(29.6);N/A;SM-1-4
IN

That will continue to branch back to test for every sequential in input, merging only the tails for each.

That's portably written, but it is a little easier to write if you can use -E xtended regular expressions (as you might w/ BSD or GNU versions)...

sed -E ':n
        s|;N/A;|;|g;$!N
        s|^(([^;]*;){3})(.*)\n\1|\1\3;|;tn
P;D'

If you wanted it all on one line:

sed -Ee:n -e's|;N/A;|;|g;$!N;s|^(([^;]*;){3})(.*)\n\1|\1\3;|;tn' -eP\;D

...would work, but I've never been very fond of one-liners like that...

Anyway, the output from the first there, is:

OUTPUT

D04005;4;279;0;0;SSM-4-1
D04005;5;40;0;0;SSM-5-1
LE040A;1;363;(26.3);SM-1-1;(27.4);SM-1-2;(28.5);SM-1-3;(29.6);SM-1-4

To also move any trailing field which starts w/ SM- to the tail of the line, and to separate each of those with /, I believe the following should work:

sed -E ':n
        s|;N/A;|;|g
        s|;(SM-[^;]*)$|/\1|;$!N
        s|^(([^;]*;){3})(.*)\n\1|\1\3;|;tn
P;D'

You know, by the way, this can get a lot easier - and a lot faster - if you can be more clear and more specific about what it is you need. To me it doesn't look like you want to merge only the first three identical fields on any two sequential lines, and to remove a field which matches N/A from any line, and to afterward move SM- fields to the tail of any line. Rather, to me it looks like all of those individual jobs you name are actually one and the same, and that you really want something like:

  • If an input line is found with three alpha-numeric, semicolon delimited fields, then a parenthesized float field followed by another colon delimiter, then an N/A field, we should do the following:
    1. Check if the next line also matches this description, and if, so, compare the first three fields for the current line and the next.
    2. If a match is found, retain only the last field from the next line, and then recurse to try again.
    3. Regardless, always remove the field which matches N/A, and replace the last ; with /.
  • Regardless, print all that remains to stdout.

Do you see how that differs? It is a series of tasks performed which depend on a single initial condition. If you can be that clear, then your matches won't have to make up for your generality in processing time.

If I am correct, then the following could work:

sed '   \|;N/A;|!b
        s||/|;$!N
        \|^\(.*(\)\(.*)\)\(.*\)\(\n\1\)\(.*)\)|!P
        s||\4\2;(\5\3|;D
' <<\IN
D04005;4;279;0;0;SSM-4-1
D04005;5;40;0;0;SSM-5-1
LE040A;1;363;(26.3);N/A;SM-1-1
LE040A;1;363;(27.4);N/A;SM-1-2
LE040A;1;363;(28.5);N/A;SM-1-3
LE040A;1;363;(29.6);N/A;SM-1-4
IN

D04005;4;279;0;0;SSM-4-1
D04005;5;40;0;0;SSM-5-1
LE040A;1;363;(26.3);(27.4);(28.5);(29.6)/SM-1-1/SM-1-2/SM-1-3/SM-1-4

...Here it is -E on one line...

sed -e'\|;N/A;|!b' -e's||/|;$!N;\|^\(.*(\)\(.*)\)\(.*\)\(\n\1\)\(.*)\)|!P;s||\4\2;(\5\3|;D'

...or in -E xtended syntax...

sed -Ee'\|;N/A;|!b' -e's||/|;$!N;\|^(.*\()(.*\))(.*)(\n\1)(.*\))|!P;s||\4\2(\5\3|;D'
  • lets try to move any field start with SM- to the end and separate them with / – Tony Jun 30 '15 at 19:10
  • @Tony - ok. Please see my edit. – mikeserv Jun 30 '15 at 19:48
1

TXR:

@(repeat)
@  (cases)
@id;@f2;@f3;@val1;@nil;@sm1
@id;@f2;@f3;@val2;@nil;@sm2
@    (do (put-line `@id;@f2;@f3;@val1;@val2;@sm1/@sm2`))
@  (or)
@line
@    (do (put-line line))
@  (end)
@(end)

$ txr data.txr data
D04005;4;279;0;0;SSM-4-1
D04005;5;40;0;0;SSM-5-1
LE040A;1;363;(26.3);(27.4);SM-1-1/SM-1-2
1
sed -r ':a;N;s!;N/A!!g;s/^(([^;]*;){3})(.*)\n\1/\1\3;/;T;s!^(([^S][^;]*;){3,})(S*SM-[^;]*);(([^S][^;]*;){1,})(.*)!\1\4\3/\6!;ta' inputfile

Even though the implied math of your comment for only 6 fields in the output suggests that there are only pairs, this is the looping version with changes to output SM-1-1/SM-1-2 when matched.

  • How do six fields imply pairs? And the other one - which didn't loop, could miss even pairs anyway, if there were a pair followed by a single then followed by another pair. In fact, it would continue to miss all pairs until another odd match or no match occurred. This one should work, but only with GNU sed - could make it also work for BSD sed if you split out another -execution statement following your :label definition, and used -E rather than -r. – mikeserv Jun 30 '15 at 23:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.