0

I have subfolders of compressed csv (.csv.gz) files containing usernames and I'm trying to find the frequency of usernames.

The following bash pipeline gives me top users.

for subfolder in folder; do gunzip -rc $subfolder | cut -d, -f1 | sort | uniq -c | sort -nr | head; done

However, it is very slow when I have many large files in the subfolders. Is there a more efficient bash script for this purpose?

2
  • Is there a reason why you're looping calling gunzip -rc one "subfolder" at a time instead of doing gunzip -rc folder without a loop? Also - "folder" is a Windows term, in Unix there are "directories" and "files".
    – Ed Morton
    Jan 4, 2021 at 17:53
  • @EdMorton Thanks for the heads up. Sometimes sorting many sorted files is faster than sorting a huge unsorted file.
    – kiasari
    Jan 4, 2021 at 20:10

1 Answer 1

1

Depending on the number of unique users, you could let awk do the heavylifting (counting repeated entries as they are read, without sorting) and then sort the result. It should go faster (but you still have to decompress everything and that takes sometime):

$ for subfolder in folder; do gunzip -rc $subfolder | awk -F, '{l[$1]++} END {for (i in l) {print l[i],i}}' | sort -rn | head;done

As a reference, you cannot make it run faster than:

$ for subfolder in folder; do gunzip -rc $subfolder > /dev/null;done
2
  • Thanks. awk makes it 5x faster and I believe the speedup is more than 5 for larger data sets.
    – kiasari
    Jan 5, 2021 at 10:23
  • @kiasari you are welcome. Please consider marking the answer as accepted if that is the case so that other people can profit from it in a similar situation. Jan 5, 2021 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.