5

I was wondering if grep, sed, and awk were viable tools for finding data in very large files.

Lets say I have a 1TB file. If i wanted to process the text in that file, what would the time frame look like if i used the individual commands grep, sed, and awk, as well as mixing them together.

Obviously a specific answer is not possible since the results will vary based on hardware specs, but if i could get a general estimation that would be helpful.

closed as too broad by Archemar, Thomas Dickey, GAD3R, Eric Renouf, G-Man Sep 22 '16 at 20:43

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • unix.stackexchange.com/questions/303044/… might help – Sundeep Sep 22 '16 at 7:16
  • Something that can have a very large influence: if you know that the file contains only 7bit ASCII, set LANG=C (as opposed to some .utf8 string) in the environment before running the command. – marcolz Sep 22 '16 at 10:20
4

Generally I would say grep is the fastest one, sed is the slowest. Of course this depends on what are you doing exactly. I find awk much faster than sed.

You can speed up grep if you don't need real regular expressions but only simple fixed strings (option -F).

If you want to use grep, sed, awk together in pipes, then I would place the grep command first if possible.

For example this:

grep -F "foo" file | sed -n 's/foo/bar/p'

is usually faster than this:

sed -n 's/foo/bar/p' file

Although the grep in the first line seems unnecessary.

BTW you may speed up these commands using LC_ALL=C if you are dealing with simple ASCII text files.

Note all my experience is based on the gnu commands. You may also try different implementations and compare the speed.

Not the answer you're looking for? Browse other questions tagged or ask your own question.