I have a file as below:


I would like to sort each column and unique the columns individually as below:


I wonder if there is anyone who knows how to use sort or uniq to do so?

  • 1
    How big is your file, in terms of columns and rows? (I.e. would transposing rows and columns first and working on that be feasible?) Jul 3, 2015 at 7:20

2 Answers 2


You can try something like this:

paste -d'\t' <(cut -f 1 -d' ' file | sort -u) <(cut -f 2 -d' ' file | sort -u) <(cut -f 3 -d' ' file | sort -u) <(cut -f 4 -d' ' file | sort -u) >output

I put tab as delimiter of paste to be more visible the output.

  • Hi taliezin, Thanks a lot for the suggestion. Is it possible to execute it as script, for example, a shell scripts? I tried to sh the file and got error in it.
    – bison72
    Jul 6, 2015 at 8:04
  • you can run it from command line and you can put it in shell script, but it will also not work for file with 500 columns, it will work for file with 4 columns. What errors you encountered?
    – taliezin
    Jul 6, 2015 at 8:09
  • Hi @taliezin, Thanks for your comment and my apologies for late reply due to heavy workload for the past few days. I wish to execute it as a simple shell script by doing sh this_script.txt / this_script.sh but I encountered error line 1: syntax error near unexpected token '(' . Appreciate if you could kindly suggest how to avoid the error and execute as a script file. Thanks a lot.
    – bison72
    Jul 10, 2015 at 6:18
  • Oh, sh does not support process substitution e.g.: <(command). You can try with bash.
    – taliezin
    Jul 10, 2015 at 15:29
  • I have found out that the output is actually correct. However, it is strange that if I awk '{print}' a specific column, it will also take characters from the column next to it. I did it in a more manual way which I phisically separated each column into individual files and proceed to next step. Thanks for all you help anyway. :)
    – bison72
    Jul 20, 2015 at 9:30

(I like the paste/cut answer of @taliezin. Here's a more prosaic solution in case of many columns). Split the data into one file per column, sort these files, and merge back the files. I'm assuming fixed-width columns of 1 character, to cope with column 2 being shorter than column 3, for example. In this demo data0 is the initial file:

for i in {1..4}
do  awk -v i=$i '{ch = substr($0,i*2-1,1);if(ch!=" ")print ch}' </tmp/data0 |
    sort -u >/tmp/data$i
awk -v rows=$(wc -l </tmp/data0) '
       file[i] = "/tmp/data" i
             d = ""
             getline d <file[i]
             printf("%1s ",d)
        printf "\n"
}' </dev/null
  • Hi Meuh, Thanks a lot for the script. For some reason, it does not work for a file with a large number of columns. I have tried on a file with 500 columns and it returned an unexpected result. What I have done was only changing the number '4' to '500'. Is that all I need to modify to work on file with large number of columns?
    – bison72
    Jul 6, 2015 at 8:03
  • In principle changing all the 4's to 500 is all you need. However, you must be running into some max limit somewhere. If we can find out where, it may be possible to work round it. Start by looking at the data* files. Do they have the right number of lines? Then how many columns do the first output lines of the 2nd awk have?
    – meuh
    Jul 6, 2015 at 8:14
  • @bison72 I made a small change to avoid accumulating a long output line in awk. It may help. You still need to have 500 open files at a time. You can find your limit on open files with ulimit -n.
    – meuh
    Jul 6, 2015 at 9:08
  • Hi Meuh. Thanks for your comment and my apologies for late reply due to heavy workload for the past few days. I have seen results as below. For example, column have multiple "G" and a "-". By performing the script, it is expected to output a "G" and a "-". Nevertheless, it also take in the characters from other columns I suppose and end up with a "-" a "G", four "T", a "W" and a "Y". The input file has 40 rows and same for output files as well. Wonder if there is anything I can do to get a correct output files. Thanks a lot.
    – bison72
    Jul 10, 2015 at 6:11
  • The only problem I can see is when earlier columns are shorter than later columns. If your data really only is 1 character wide this is easily solved. I edited my script to work with input data of exactly 1 character. Perhaps this may help.
    – meuh
    Jul 10, 2015 at 8:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.