Recently, we purchased a software solution to integrate into a device we're developing. There's plenty we need to modify and adapt to our needs, so today I was going through part of the code to see what would need to change and whatnot and started wondering about something.

Going through some of the scripts I saw some lines that caught my attention. For example, one was something like this:

cat file | grep ^field | head -n1 | sed 's/:/ /' | awk '{print $1}'

That seemed a little silly when you can easily do it all with a single call to awk, maybe something like:

awk -F':' '/^field/ {print $1; exit}' file

After that I started paying more attention to this and found many similar situations. Some I don't care about because are run during initialization. Others, on the other hand, are called quite frequently.

This means I have scripts spawning processes everywhere for tasks that can be accomplished with much less. Now what I actually started wondering about... does piping too much start hurting performance at some point? Particularly if there's a less "piped" alternative.

Keep in mind that I'm running an embedded Linux in a platform with much less resources that an actual PC. Although for the sake of the question, maybe that doesn't matter.

  • Spawning a subshell and a new process is cheap, but certainly more expensive than not.
    – jordanm
    Feb 2 '16 at 22:59
  • Certainly, but I'm wondering whether it'd be worth it to take the time to try to find this cases (at least the most critical ones) and re-write them, or at least profile them first.
    – elpato
    Feb 3 '16 at 0:33

Sometimes it is just easier to profile things:

I've created a sample input file:


shell script 'a.sh':

for i in `seq 1 1000`; do
        cat test.dat | grep ^field | head -n1 | sed 's/:/ /' | awk '{print $1}' >/dev/null

shell script 'b.sh':

for i in `seq 1 1000`; do
        awk -F':' '/^field/ {print $1; exit}' test.dat >/dev/null

Profile it:

time ./a.sh
real    0m10.253s
user    0m5.526s
sys 0m8.668s

time ./b.sh
real    0m3.274s
user    0m1.288s
sys 0m1.783s

(This was done on my old beloved 2008 MacBook, 2,4 GHz Intel Core 2 Duo)

So obviously your version is many times faster. However, these are the times for 1000 invocations. Depending on the frequency this shell code is executed you might only save a few milliseconds.

  • 1
    @drewbenn +1 for taking file size into the equation
    – gollum
    Feb 3 '16 at 0:13
  • Nice, I just tried it out and indeed it makes a really big difference, I'm a big fan of awk but had never noticed this... I might need to rewrite some stuff of my own =) In any case, that was just an example, I could always use both grep | awk or maybe even grep | cut. I was wondering if having large chains of pipes might cause performance issues, particularly in an embedded system which has much less resources than an actual PC. But I guess what I can take from this answer (plus comments) is that it depends entirely on the code at hand, is that right?
    – elpato
    Feb 3 '16 at 0:49

Performance is complicated. The only way to be sure is to benchmark on a real system with a real load.

Piping multiple utilities definitely has a cost. Compared with string operations, this cost is very high. However, if the amount of data is large enough, a pipe solution can be faster, because it may allow specialized tools to do their job faster and it can allow parallel data processing (if the machine is multicore and there is enough useful processing that can be done in parallel to make it worthwhile). But if the amount of data is small, the cost to start up the programs dominates. The point at which the dominant factor changes is very dependent on the system and the usage scenario.

Generally speaking, the more versatile a tool is, the slower it is. So if you have a task that grep or head can do, then usually sed can also do it, but not as fast; and awk will be even slower. This is just a rule of thumb; if you go looking you'll be able to find specific implementations and specific workloads where awk or sed beat other tools. The data volume has to be sufficiently high for the difference to be observable at all.

For small data volumes, the number of process startups is the dominant cost. Generally speaking, the more versatile a tool is, the slower it is to start up. Launching multiple tools has a performance cost in itself, because it means that more code has to be loaded into memory. However, if you use BusyBox, where all tools are grouped in a single executable, that aspect is minimized.

Starting from

cat file | grep ^field | head -n1 | sed 's/:/ /' | awk '{print $1}'

the invocation of cat is useless and cannot help performance. Using grep ^field | head -n1 | sed 's/:/ /' may have a slight advantage if the data volume is very large, but in most scenarios I'd expect

<file sed -n '/^field/ { s/:/ /p; q; }'

to be faster because it avoids having to wait for multiple processes.

As for the awk invocation, it simply isn't necessary here. If there are no leading colons, then the command is equivalent to

<file sed -n '/^field/ { s/:.*//p; q; }'

or, if grep proves to have an advantage,

<file grep '^field' | sed -e 's/:.*//' -e 'q'

And if there are leading colons, just add s/^::*// at the beginning of the sed command.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.