Find the frequency of unique names

Question

I have subfolders of compressed csv (.csv.gz) files containing usernames and I'm trying to find the frequency of usernames.

The following bash pipeline gives me top users.

for subfolder in folder; do gunzip -rc $subfolder | cut -d, -f1 | sort | uniq -c | sort -nr | head; done

However, it is very slow when I have many large files in the subfolders. Is there a more efficient bash script for this purpose?

Answer 1

Depending on the number of unique users, you could let awk do the heavylifting (counting repeated entries as they are read, without sorting) and then sort the result. It should go faster (but you still have to decompress everything and that takes sometime):

$ for subfolder in folder; do gunzip -rc $subfolder | awk -F, '{l[$1]++} END {for (i in l) {print l[i],i}}' | sort -rn | head;done

As a reference, you cannot make it run faster than:

$ for subfolder in folder; do gunzip -rc $subfolder > /dev/null;done