Basically I want to apply if condition in lines which have same value in column2. In example first three lines have same value in column2 (Disease1 ). Now for these three line I want to apply IF condition - if 3rd column is 'High' write only that line to output file. If 3rd column doesn't have 'High', then look for 'Medium' and print it. If 'Medium' is also not there, then print line containing 'Low' value.


id1 Disease1 High
id2 Disease1 Medium
id3 Disease1 Low
id4 Disease2 Low
id5 Disease3 Medium
id6 Disease3 Low

Expected Output:

id1 Disease1 High
id4 Disease2 Low
id5 Disease3 Medium
awk 'BEGIN { FS=OFS="\t"; d["High"]=1; d["Medium"]=2; d["Low"]=3 } { print d[$3], $0 }' file |
sort -t $'\t' -k3,3 -k1,1n | sort -t $'\t' -s -u -k3,3 | cut -f 2-

Unfortunately, the words "High", "Medium" and "Low" does not sort alphabetically in this order, so instead we prepend each line with the integers 1, 2 and 3 for the corresponding words in column 3. This is done with the awk script. The integer is separated from the original content by a tab character and indicates the severity of the disease (1 is highest).

Then we sort the modified data on the disease and on this integer field, so that the records are grouped by disease and ordered by severity.

We then do a "unique sort" with only the diseases as the key (the second sort). This discards any duplicated disease and for each disease we are left with only the most severe record. The -s makes sort use a stable sorting algorithm that preserves the reordering of records with identical keys.

The cut at the end removes the integer that we previously added with awk.

The pipeline assumes the use of bash to specify the field delimiter to the sort commands. If bash is not used, insert a single quoted literal tab instead of using $'\t' (you may likely do this by pressing Ctrl+V Tab).

  • could also use awk -F'\t' '!seen[$3]++' in place of the 2nd sort. – glenn jackman Sep 21 '18 at 12:24
  • what is the usage of specifying the \t as separator for the sort command? it's the default separator for the sort and even cut – αғsнιη Sep 22 '18 at 5:55
  • @αғsнιη The default field separator for sort is any consecutive number of tabs or spaces. Assuming that, for example, the disease name may contain spaces, we need to specify tabs as the separator for the sorting to work correctly. – Kusalananda Sep 22 '18 at 7:28
  • it is working fine when i am using in command line but giving error when I am trying in shell Error: sort: multi-character tab ‘$\\t’ – LiNi Sep 24 '18 at 12:30
  • @LiNi Are you writing $'\\t' on the command line (with two backslashes)? – Kusalananda Sep 24 '18 at 12:39

If the priorities are always from High to Low for each Ids in seconds column, then that would be done easily with:

sort -u -k2,2 infile

Otherwise you could do:

sed 's/High$/1/; s/Medium$/2/; s/Low$/3/' infile \
| sort -k2,3  \
| sort -uk2,2 \
| sed 's/1$/High/; s/2$/Medium/; s/3$/Low/'

Or with justawk:

awk '{ pr[$2]=($3=="High"?$3:(pr[$2]=="High"?pr[$2]:(pr[$2]=="Medium"?pr[$2]:$3)));
       if (temp!=pr[$2]) { id[$2]=$0; temp=pr[$2] }
} END { for (key in id) print id[key] }' infile

I'd use Perl and build a parser, that would create this hash:

  • column 2 value as the key
  • first line corresponding to a key would constitute the value of that key
  • all other lines for a key would be skipped once the value is set

After the parsing's been done, I'd print the hash, sorting on the second token if required.


With just awk

awk '
    BEGIN {prio["High"]=1; prio["Medium"]=2; prio["Low"]=3}
    !($2 in p) || prio[$3] < p[$2] {p[$2] = prio[$3]; line[$2] = $0}
    END {for (key in line) print line[key]}
' file
id1 Disease1 High
id4 Disease2 Low
id5 Disease3 Medium

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.