4

I am trying to insert line breaks on a file based on the same names which happens to be the second field in my file. So, my input file is like below.

17412193|name1|organization
43979400|name1|organization
1405541|name2|organization
53595498|name2|organization
50439202|name2|organization
54678379|name3|Not Found
21757330|name3|organization

So I am trying to get the output like,

17412193|name1|organization
43979400|name1|organization
###linebreak inserted here
1405541|name2|organization
53595498|name2|organization
50439202|name2|organization
###linebreak inserted here
54678379|name3|Not Found
21757330|name3|organization
###linebreak inserted here

I am using this line break to split the input file to group the similar names together. I would later apply a similarity algorithm on these grouped names. So, for the above example, after the output I would apply the similarity algorithm on 3 pairs of names.

8

Assuming your file is sorted/grouped by the 2nd field already

awk -F '|' 'NR>1 && $2 != prev {print ""} {prev=$2; print}' file
1
  • It would be helpful to us, if you explain your command. – Avinash Raj Mar 29 '14 at 6:57
4

Here's a case insensitive Perl solution:

perl -F'\|' -lape 'unless($F[1]=~/^$pre$/i || $.==1){print ""}; $pre=$F[1]' file 

Explanation:

  • The -a splits the input lines into the @F array, making perl act like awk.
  • The -F is the field separator
  • -p means print each input line
  • -l adds a \n to each print call, so print "" prints a newline.
  • unless($F[1]=~/^$pre$/i || $.==1) : Unless the second field is the same as that of the previous line (the i in //i makes the match case insensitive), or unless this is the first line.
  • $pre=$F[1] : save this lines second field as $pre.

@GlennJackman suggested a slightly different version in the comments which will probably be faster for larger files:

perl -F'\|' -lape 'unless(lc($F[1]) eq lc($pre) || $.==1){print ""}; $pre=$F[1]' file 
6
  • @Ramesh you're welcome, I added an explanation of how it works. – terdon Mar 26 '14 at 16:10
  • It is really an elegant solution. Thanks again :) – Ramesh Mar 26 '14 at 16:13
  • This seems like regex overkill: $F[1]=~/^$pre$/i -- I wonder if this would be simpler: lc($F[1]) eq lc($pre) – glenn jackman Mar 26 '14 at 21:05
  • 2
    Also, -l automatically chomps each incoming line, so print $_ outputs the right number of newlines (one). – glenn jackman Mar 26 '14 at 21:06
  • @glennjackman you're probably right. At the very least, using lc() would make it faster. It also makes it not fit on the same line in the post though and it's not that big of a difference so I'll leave it. Not sure what you mean about -l, yes it does imply chomp() but why is that relevant? – terdon Mar 26 '14 at 21:14
4

A little shorter Perl solution:

perl -pe 'print "\n" if ($l =~ /name\d+/ && $_ !~ /$&/);$l=$_;' input
  • if last line ($l) was name\d+ and current line not last match, then print new line
  • assign current line to $l

A more general solution

perl -pe 'print "\n" if ($l =~ /\|([^\|]+)/ && $_ !~ /$1/);$l=$_;'  input
3
  • 1
    Heh, +1 for the clever matching but I don't think the names the OP has will actually be in the form of name\d+. More likely they will be arbitrary strings. – terdon Mar 26 '14 at 23:32
  • thanks, yes that's right, I'm thinking about /\|.*?\|/, but it seems not to work. – user55518 Mar 26 '14 at 23:53
  • You need to quote the match because of the |: perl -pe 'print "\n" if ($l =~ /\|.*?\|/ && $_ !~ /"$&"/);$l=$_;' – terdon Mar 27 '14 at 0:04
0
#!/bin/sh
#shell basics, POSIX compliant
(set -f ;IFS='|
' ; set -- $(cat) ; while [ -n "$3" ] ;do {
[ "${t=$2}" != "$2" ] && echo && t=$2
printf '%s|%s|%s\n' "$1" "$2" "$3" ; shift 3 
} ; done )<<\SAMPLE
17412193|name1|organization 
43979400|name1|organization 
1405541|name2|organization 
53595498|name2|organization 
50439202|name2|organization 
54678379|name3|Not Found 
21757330|name3|organization
SAMPLE

Output

17412193|name1|organization 
43979400|name1|organization 

1405541|name2|organization 
53595498|name2|organization 
50439202|name2|organization 

54678379|name3|Not Found 
21757330|name3|organization

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.