Sorting uniquely from a row

Question

Can you please give me suggestion how do I can uniquely sort from a line or row? I have information like this:

Special   c1,c2,c5,c7,c1,c2   
Special2  C6

(that's a TAB character in between Special and c1...).

I want the output like this:

Special   c1,c2,c5,c7  
Special2  C6

How can I accomplish this?

Answer 1

Using this answer,

perl -MList::MoreUtils=uniq -laF'\t' -ne '
    $F[1] = join(",", uniq(sort(split(",", $F[1])))); print join("\t", @F)'

This depends on an external package List::MoreUtils. If you don't want to install an external dependency, reimplementing the uniq function is just a few more lines of Perl. (Though I seem to have it installed as part of the base system on macOS.)

Answer 2

perl -F'\t|,' -lane 'my %h; print shift @F, "\t", join ",", sort grep !$h{$_}++, @F' dataf

---

Explanation

• -F'\t|,' => will split each record fields into the array @F on TAB or comma characters.
• -l will set the RS to newline and ORS to newline as well.
• -a will autosplit each record into words based on the FS chosen by -F.
• -n will setup an implicit record read in loop on the input AND will print things only when asked to.
• -e is the Perl code to be executed upon each record of the input based on RS choen by -l above.
• The first element will be given by shift and the remaining elements will be uniquified via storing them as keys of a hash, %h, which shall be regenerated every time a record read in. The unique elements are then sorted and joined with a comma and printed.

Answer 3

Tested with OpenBSD awk, GNU awk and mawk:

awk -F ',| +' '{ for (i = 2; i <= NF; ++i) { print $1, $i } }' data.in |
sort -u |
awk '{ f[$1] = (f[$1] ? f[$1] "," : "") $2 } END { for (k in f) { print k, f[k] } }'

The first awk expands the given data into

Special c1
Special c2
Special c5
Special c7
Special c1
Special c2
Special2 C6

It uses both commas and multiple spaces as field delimiter and, for each record (line) of input, it prints the first field followed by each of the other fields in turn on separate lines. This assumes that there are no other spaces or commas on the lines other than where they will be properly interpreted as delimiters.

The sort in the middle sorts it into

Special2 C6
Special c1
Special c2
Special c5
Special c7

It does a sort using the full line as the sorting key and discards any duplicate line.

The last awk recombines the data into

Special c1,c2,c5,c7
Special2 C6

It does this by using the first field as a key into an associative array and stores the comma-separated concatenation of the corresponding data in the second field as the value. At the end, all collected data is printed.

Answer 4

gawk(GNU awk) approach:

 awk '{if($2~/.*,.*/){l=split($2,a,","); asort(a); $2=a[1]; b[a[1]]++;
      for(i=2;i<=l;i++) $2=(!b[a[i]]++)? $2","a[i] : $2 }}1' file

The output:

Special c1,c2,c5,c7
Special2  C6

---

if($2~/.*,.*/) - if we have multiple comma-separated items in the second field

l=split($2,a,",") - dividing string into pieces separated by ,.
l is assigned with number of pieces

asort(a) - sorts an array of substrings

$2=(!b[a[i]]++)? $2","a[i] : $2 - considering unique items

Answer 5

Another way in one line:

while read line; do echo "$line" | awk '{print $1}' | tr '\n' ' ';  echo "$line" | awk '{print $2}' | tr ',' '\n' | sort -u | tr '\n' ',' | sed -e 's/.$//g'; echo; done < file_to_sort

It takes the first column of every line (echo $line | awk '{print $1}' | tr '\n' ' ';) and sort the second column values separated by ',' after converting it in a single column in order to apply sort and then converting it back to a single row with the original formatting (echo $line | awk '{print $2}' | tr ',' '\n' | sort -u | tr '\n' ',').

Doing line splitting as suggested by @tripleee:

while IFS=$'\t' read first second; do printf "%s\t%s\n" "$first" "$(tr ',' '\n' <<<"$second" | sort | tr '\n' ',' | sed -e 's/.$//g';)"; done < file_to_sort