I have a fairly large text file (19Gb, half a billion lines) and a fairly large server on which to sort it (16 cores, 64Gb memory) running Ubuntu 12.04.
What is the best way to use the command line sort
utility (it's from GNU core utilities) to quickly sort this file?
I believe I can use --parallel 15
(leaving one core free), can use something like 1, can use -T /run/shm/sortTmp
to keep temp files on a ram disk (so it's fast)-S 50G
to utilise available memory and can tweak --batch-size=15
to reduce the use of temp files (here I'm not sure what a good value would be..).
Are there other options to consider? What is a good choice for --batch-size
? Are there other things I should be considering?
There is nothing else that needs to be running on the machine while I'm doing the sort, so I can use all it's resources.
1 -S
in combination with the ram disk for temp files is a bad idea! The kernel will likely kill sort
when it writes to the ram disk and memory is almost full (see this comment)