direct GNU sort to use much memory and many cores

Question

I have a fairly large text file (19Gb, half a billion lines) and a fairly large server on which to sort it (16 cores, 64Gb memory) running Ubuntu 12.04.

What is the best way to use the command line sort utility (it's from GNU core utilities) to quickly sort this file?

I believe I can use --parallel 15 (leaving one core free), can use something like -T /run/shm/sortTmp to keep temp files on a ram disk (so it's fast)¹, can use -S 50G to utilise available memory and can tweak --batch-size=15 to reduce the use of temp files (here I'm not sure what a good value would be..).

Are there other options to consider? What is a good choice for --batch-size? Are there other things I should be considering?

There is nothing else that needs to be running on the machine while I'm doing the sort, so I can use all it's resources.

¹ -S in combination with the ram disk for temp files is a bad idea! The kernel will likely kill sort when it writes to the ram disk and memory is almost full (see this comment)

Answer 1

Nobody's answering, so this is what I did:

sort -T /mnt/tmp --parallel=14 -S 50G myBigFile.txt

I didn't use --batch-size as I wasn't sure what would be a good value and presumably sort is clever about it anyway.

/mnt is mounted on a big ephemeral disk (the machine is a VM in the cloud). If /tmp is on a disk with plenty of space, you could leave out -T. If you've got a very fast disk with enough space, use that.

If I remember correctly, it needs about the same space again as the size of the file being sorted - basically it divides the file into two or more chunks which it can each sort in memory, stores them sorted in the temp disk, then does a merge sort to produce the final output.