2

Say i have a command using several pipes. This could be any command but for example lets use this simple cut and sort. e.g

cat data.tsv | cut -f1,5,8,12 | sort | uniq -c | sed -e "s/cat/dog/g" | sort -k1nr > output.tsv

How can i 'profile' this?

Is there anyway i can say that its the sed or the uniq that's slowing down my execution time?

In my ideal world i'd love to see something like:

1- cat: 15 lines per second
2- cut: 13 lines per second
3- sort: 6 lines per second
4- uniq: 6 lines per second (Limited by input)
5- sed: 6 lines per second (Limited by input)
6- sort 6 lines per second (Limited by input)
7- write 6 lines per second (Limited by input)

Any ideas?

1

I suggest to use pv to count lines through a pipe. With pv you can step by step find the bottleneck.

cat data.tsv | pv -lr > /dev/null

cat data.tsv | cut -f1,5,8,12 | pv -lr > /dev/null

cat data.tsv | cut -f1,5,8,12 | sort | pv -lr > /dev/null

cat data.tsv | cut -f1,5,8,12 | sort | uniq -c | pv -lr > /dev/null

...

Example with pv‘s output:

timeout 10 yes | sed 's/yes/foo/' | rev | pv -lr >/dev/null
[ 120k/s] 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.