4

I would like to ask if there is a out of the box multicore equivalent for a '| sort | uniq -c | sort -n' command?

I know that I can use below procedure

split -l5000000 data.tsv '_tmp';
ls -1 _tmp* | while read FILE; do sort $FILE -o $FILE & done;
sort -m _tmp* -o data.tsv.sorted

But it tastes a bit overhelming.

8

GNU sort has a --parallel flag:

sort --parallel=8 data.tsv | uniq -c | sort --parallel=8 -n

This would use eight concurrent processes/threads to do each of the two sorting steps. The uniq -c part will still be using a single process.

As Stéphane Chazelas points out in comments, the GNU implementation of sort is already parallelised (it's using POSIX threads), so modifying the number of concurrent threads is only needed if you want it to use more or fewer threads than what you have cores.

Note that the second sort will likely get much less data than the first, due to the uniq step, so it will be much quicker.

You may also (possibly) improve sorting speed by playing around with --buffer-size=SIZE and --batch-size=NMERGE. See the sort manual.

To further speed the sorting up, make sure that sort writes its temporary files to a fast filesystem (if you have several types of storage attached). You may do this by setting the TMPDIR environment variable to the path of writable directory on such a mountpoint (or use sort -T directory).

  • Note that GNU sort works in parallel by default, so no need to specify --parallel unless you want to use fewer threads than CPU cores or more than 8 ones. It doesn't make sense to use disk storage on memory. temporary storage is used to offload memory. No need to offload in the first place if it's to still go to memory. – Stéphane Chazelas Jun 29 '18 at 10:27
  • I find that GNU sort (8.28) seems to use additional threads (with or without --parallel) only for regular files (not for pipes nor sockets, not even block devices) – Stéphane Chazelas Jun 29 '18 at 10:44
  • @StéphaneChazelas I can't see an explicit test for this in the source for 8.29. – Kusalananda Jun 29 '18 at 10:53
  • 2
    It seems it has to do with how the data is read and how the threads are spawned. It needs at least 128*1024 lines in an input buffer to spawn a new thread, and it seems that for non-regular files (where find doesn't know the size of the file), it never happens unless you raise the -S size above some threshold. 10M is not enough 100M seems to be enough for the output of seq. I've not looked in details. – Stéphane Chazelas Jun 29 '18 at 11:31
0

This is faster for me. YMMV (it definitely depends on how many duplicates there are):

parallel --lb --pipepart --block 15m -a /tmp/big.file 'sort | uniq -c' |
  awk '{ count[$2] += $1 } END { for(elem in count) print count[elem], elem }' |
  sort -n

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.