15
votes
2answers
386 views

Is dash or some other shell “faster” than bash?

I always thought that the only benefit of using dash instead of bash was that dash was smaller, and therefore many instances of dash would start faster at boot time. But I have done some research, ...
7
votes
2answers
454 views

Why is `tac file | grep foo' (piping) faster than `grep foo < <(tac file)' (process substitution)?

This question was motivated by "Reverse grepping", about grepping a huge file from bottom up. @chaos said: tac file | grep whatever Or a bit more effective: grep whatever < <(tac file) ...
3
votes
1answer
98 views

Performance differences between pipelines and process substitution

I tend to use pipelines in my bash scripts over process substitution in most situations, especially in cases of using multiple sets of commands as it seems more readable to do ... | ... | ... over ... ...
23
votes
5answers
3k views

Why is iterating over a file twice faster than reading it into memory and computing twice?

I'm comparing the following tail -n 1000000 stdout.log | grep -c '"success": true' tail -n 1000000 stdout.log | grep -c '"success": false' with the following log=$(tail -n 1000000 stdout.log) echo ...
3
votes
1answer
139 views

CPU is free, yet bash script not utilizing all CPU resources

I ran a simple script to generate a large (10000000 lines) csv file with 6 fields in which some fields changed in each line/row, using a while loop. The machine had all (32) CPUs free, lot of RAM (~31 ...
2
votes
2answers
2k views

Running thousands of curl background processes in parallel in bash script

I am running thounsand of curl background processes in parallel in the following bash script START=$(date +%s) for i in {1..100000} do curl -s "http://some_url_here/"$i > $i.txt& ...
23
votes
9answers
908 views

for vs find in Bash

When looping through files there are two ways: 1. use a for-loop for f in *; do echo "$f" done 2. use find find * -prune | while read f; do echo "$f" done Assuming these two loops will ...
4
votes
3answers
325 views

Why is bash's printf faster than /usr/bin/printf?

I have two ways of calling printf on my system: $ type -a printf printf is a shell builtin printf is /usr/bin/printf $ file /usr/bin/printf /usr/bin/printf: ELF 64-bit LSB executable, x86-64, ...
9
votes
2answers
1k views

echo vs <<<, or Useless Use of echo in Bash Award?

By now the Useless Use of cat Award is very well known, and there's also a mention of a Useless Use of echo (not relevant for this question). I'm wondering if there should be a "Useless Use of echo in ...
5
votes
1answer
2k views

Fast elimination of duplicate lines across multiple files

I have a huge amount of data in which each (data-)line should be unique. There are a lot of files in one folder in which this is already true. It is about 15GB splitted into roughly 170 files with ...
5
votes
3answers
1k views

Fast way to build a test file with every second listed in YYYY-mm-dd HH:MM:SS format

I want to create a large test file with lines containg dates listed by the second, but my method is taking inordinately long... (or at least, that's how it feels :) ... 43 minutes to create only ...