124

I'm facing a huge 4-columns file. I'd like to display the sorted file in stdout based on its 3rd column:

cat myFile | sort -u -k3

Is that enough to perform the trick?

  • 4
    Note that you can write this as sort -u -k3 < myFile. – gerrit Mar 24 '16 at 15:31
  • 5
    As sort -u -k3 myFile, even – Sebastian Graf Jun 15 '18 at 7:54
156
sort -k 3,3 myFile

would display the file sorted by the 3rd column assuming the columns are separated by sequences of blanks (ASCII SPC and TAB characters in the POSIX/C locale), according to the sort order defined by the current locale.

Note that the leading blanks are included in the column (the default separator is the transition from a non-blank to a blank), that can make a difference in locales where spaces are not ignored for the purpose of comparison, use the -b option to ignore the leading blanks.

Note that it's completely independent from the shell (all the shells would parse that command line the same, shells generally don't have the sort command built in).

-k 3 is to sort on the portion of the lines starting with the 3rd column (including the leading blanks). In the C locale, because the space and tab characters ranks before all the printable characters, that will generally give you the same result as -k 3,3 (except for lines that have an identical third field),

-u is to retain only one of the lines if there are several that sort identically (that is where the sort key sorts the same (that's not necessarily the same as being equal)).

cat is the command to concatenate. You don't need it here.

If the columns are separated by something else, you need the -t option to specify the separator.

Given example file a

$ cat a
a c c c
a b ca d
a b  c e
a b c d

With -u -k 3:

$ echo $LANG
en_GB.UTF-8

$ sort -u -k 3 a
a b ca d
a c c c
a b c d
a b  c e

Line 2 and 3 have the same third column, but here the sort key is from the third column to the end of line, so -u retains both. ␠ca␠d sorts before ␠c␠c because spaces are ignored in the first pass in my locale, cad sorts before cc.

$ sort -u -k 3,3 a
a b c d
a b  c e
a b ca d

Above only one is retained for those where the 3rd column is ␠c. Note how the one with ␠␠c (2 leading spaces) is retained.

$ sort -k 3 a
a b ca d
a c c c
a b c d
a b  c e
$ sort -k 3,3 a
a b c d
a c c c
a b  c e
a b ca d

See how the order of a b c d and a c c c are reversed. In the first case, because ␠c␠c sorts before ␠c␠d, in the second case because the sort key is the same (␠c), the last resort comparison that compares the lines in full puts a b c d before a c c c.

$ sort -b -k 3,3 a
a b c d
a b  c e
a c c c
a b ca d

Once we ignore the blanks, the sort key for the first 3 lines is the same (c), so they are sorted by the last resort comparison.

$ LC_ALL=C sort -k 3 a
a b  c e
a c c c
a b c d
a b ca d
$ LC_ALL=C sort -k 3,3 a
a b  c e
a b c d
a c c c
a b ca d

In the C locale, ␠␠c sorts before ␠c as there is only one pass there where characters (then single bytes) sort based on their code point value (where space has a lower code point than c).

  • columns are blank-separated that may include other characters in addition to space and tab depending on locale. – jfs Dec 11 '13 at 0:23
  • 1
    Nice, +1. Could you explain what the 3,3 does? Why not just 3? – terdon Dec 11 '13 at 1:04
  • @terdon, see expanded description with examples. – Stéphane Chazelas Dec 11 '13 at 9:49
  • @J.F.Sebastian, you're right, answer updated. – Stéphane Chazelas Dec 11 '13 at 11:56
  • Ah, to make it only sort on the 3rd, not the rest of the line, thanks. – terdon Dec 12 '13 at 3:57
4

If you understand "column" as in text file (4th character) then yes, your solution should work (or even sort -u -k3 myFile to allow sort perform some memory-saving magics with random access). If you understand "column" as in database - a whole entity of data followed by a separator, and variable column width, you'll need something fancier e.g. this sorts ls -l by size

      ls -l |awk '{print $5 " " $0;}'| sort -n | cut -d " " -f 2-

(which is equivalent to trivial ls -lS but serves the example nicely.)

  • 5
    No, by default sort columns are blank separated, they are not character columns, to sort on the 3rd character column, the syntax would be: sort -k 1.3,1.3. ls -l | sort -k5,5n to sort on the size. – Stéphane Chazelas Dec 10 '13 at 14:30
  • The awk solution is exactly what I needed-- easily modified to fit complex sorting requirements – jchook Nov 8 '17 at 2:48
2
sort -g -k column_number 

is the right command to sort any list having numeric characters using specific column

  • 1
    Using -k was covered pretty well already so it would be helpful if you explained how this command is different or better. Maybe you could also include actual column numbers to address the OP's actual question. – Jeff Schaller Jun 5 '17 at 9:52
  • This got me to use the man pages :p "-g, --general-numeric-sort, compare according to general numerical value" which was what I needed in my case. – joels Aug 17 '18 at 18:42
1

You can use the awk Velour library:

#!/usr/local/bin/velour -f
{
  q[NR] = $3
  z[NR] = $0
}
END {
  a_sort_by(q, z)
  io_puts(q)
}
0
$ sort -k 1.3,1.3 myfile

Will sort your myfile file on the third column if your file don't have any separator.

$ cat myfile 
ax5aa 
aa3ya 
fg7ds 
pp0dd 
aa1bb

$ sort -k 1.3,1.3 myfile 
pp0dd 
aa1bb
aa3ya 
ax5aa 
fg7ds 

man page of sort:

[...] -k, --key=POS1[,POS2] start a key at POS1 (origin 1), end it at POS2 (default end of line) [...] POS is F[.C][OPTS], where F is the field number and C the character position in the field; both are origin 1. If neither -t nor -b is in effect, characters in a field are counted from the beginning of the preceding whitespace. OPTS is one or more single-letter ordering options, which override global ordering options for that key. If no key is given, use the entire line as the key.

With --key=1.3,1.3, you said that there only one field (the entire line) and that you're comparing the third character position of this field.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.