3

Can you please give me suggestion how do I can uniquely sort from a line or row? I have information like this:

Special   c1,c2,c5,c7,c1,c2   
Special2  C6

(that's a TAB character in between Special and c1...).

I want the output like this:

Special   c1,c2,c5,c7  
Special2  C6

How can I accomplish this?

  • 2
    Considering that there is no reliable delimiter in your example, I doubt you'd be able to sort them yourself to your satisfaction. Programming involves defining the rules to apply. – Julie Pelletier Apr 24 '17 at 4:36
  • yes they are tab delimited. – Masum Billah Apr 24 '17 at 4:40
  • 1
    And yet in your example, they are comma-delimited. – DopeGhoti Apr 24 '17 at 4:56
  • I mean Special and the numbers are in two columns actually and they are tab delimited. sorry for my misinformation. – Masum Billah Apr 24 '17 at 5:00
  • 2
    "Yes, I didn't read the question properly" or "no, I need both"? – tripleee Apr 24 '17 at 5:26
9

Using this answer,

perl -MList::MoreUtils=uniq -laF'\t' -ne '
    $F[1] = join(",", uniq(sort(split(",", $F[1])))); print join("\t", @F)'

This depends on an external package List::MoreUtils. If you don't want to install an external dependency, reimplementing the uniq function is just a few more lines of Perl. (Though I seem to have it installed as part of the base system on macOS.)

7
perl -F'\t|,' -lane 'my %h; print shift @F, "\t", join ",", sort grep !$h{$_}++, @F' dataf

Explanation

  • -F'\t|,' => will split each record fields into the array @F on TAB or comma characters.
  • -l will set the RS to newline and ORS to newline as well.
  • -a will autosplit each record into words based on the FS chosen by -F.
  • -n will setup an implicit record read in loop on the input AND will print things only when asked to.
  • -e is the Perl code to be executed upon each record of the input based on RS choen by -l above.
  • The first element will be given by shift and the remaining elements will be uniquified via storing them as keys of a hash, %h, which shall be regenerated every time a record read in. The unique elements are then sorted and joined with a comma and printed.
5

Tested with OpenBSD awk, GNU awk and mawk:

awk -F ',| +' '{ for (i = 2; i <= NF; ++i) { print $1, $i } }' data.in |
sort -u |
awk '{ f[$1] = (f[$1] ? f[$1] "," : "") $2 } END { for (k in f) { print k, f[k] } }'

The first awk expands the given data into

Special c1
Special c2
Special c5
Special c7
Special c1
Special c2
Special2 C6

It uses both commas and multiple spaces as field delimiter and, for each record (line) of input, it prints the first field followed by each of the other fields in turn on separate lines. This assumes that there are no other spaces or commas on the lines other than where they will be properly interpreted as delimiters.

The sort in the middle sorts it into

Special2 C6
Special c1
Special c2
Special c5
Special c7

It does a sort using the full line as the sorting key and discards any duplicate line.

The last awk recombines the data into

Special c1,c2,c5,c7
Special2 C6

It does this by using the first field as a key into an associative array and stores the comma-separated concatenation of the corresponding data in the second field as the value. At the end, all collected data is printed.

5

gawk(GNU awk) approach:

 awk '{if($2~/.*,.*/){l=split($2,a,","); asort(a); $2=a[1]; b[a[1]]++;
      for(i=2;i<=l;i++) $2=(!b[a[i]]++)? $2","a[i] : $2 }}1' file

The output:

Special c1,c2,c5,c7
Special2  C6

if($2~/.*,.*/) - if we have multiple comma-separated items in the second field

l=split($2,a,",") - dividing string into pieces separated by ,.
l is assigned with number of pieces

asort(a) - sorts an array of substrings

$2=(!b[a[i]]++)? $2","a[i] : $2 - considering unique items

  • asort is a GNU Awk extension, isn't it? – tripleee Apr 24 '17 at 7:11
  • 1
    @tripleee, and I added g -> gawk – RomanPerekhrest Apr 24 '17 at 7:12
2

Another way in one line:

while read line; do echo "$line" | awk '{print $1}' | tr '\n' ' ';  echo "$line" | awk '{print $2}' | tr ',' '\n' | sort -u | tr '\n' ',' | sed -e 's/.$//g'; echo; done < file_to_sort

It takes the first column of every line (echo $line | awk '{print $1}' | tr '\n' ' ';) and sort the second column values separated by ',' after converting it in a single column in order to apply sort and then converting it back to a single row with the original formatting (echo $line | awk '{print $2}' | tr ',' '\n' | sort -u | tr '\n' ',').

Doing line splitting as suggested by @tripleee:

while IFS=$'\t' read first second; do printf "%s\t%s\n" "$first" "$(tr ',' '\n' <<<"$second" | sort | tr '\n' ',' | sed -e 's/.$//g';)"; done < file_to_sort
  • 1
    You could do the line splitting in the shell with while IFS=$'\t' read first second; do printf "%s\t%s\n" "$first" "$(tr ',' '\n' <<<"$second" | sort -u | tr '\n' ',' | sed 's/.$//')"; done (this obviously uses some Bash constructs, so #!/bin/bash not #!/bin/sh). – tripleee Apr 24 '17 at 7:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.