Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have some big files that I have changed into "standardized form", removing line breaks. I want to speed up greps on those files. I know that you can optimize grep by doing grep 'partofpattern' | grep -E 'partofpattern [[:digit:]]+' -- that is, have grep search for lines that contain a more simple part of a pattern and then have grep run a second time on the few lines that contain part of the pattern. That makes me think that grep somehow works line-by-line. So will adding line breaks to big files with no line breaks speed up grep?

share|improve this question
    
If a file has no line breaks, your grep commands either print the entire file or nothing at all. I think something must be missing from the description of what you're trying to do here. –  Michael Homer Aug 5 at 0:24
    
@MichaelHomer I am using the -o option –  akh2103 Aug 5 at 0:27
1  
I think grep and sed parse intermittently gulping by line break. If you remove all the line breaks, it may well be faster; but I'd worry about buffer overflow. –  ericx Aug 5 at 0:36
2  
I'm skeptical that you can actually speed grep up by scanning lines twice. Do you have an actual benchmark which demonstrates this? –  rici Aug 5 at 1:43
    
It depends strongly on the pattern. If it's a regular expression that does a ton of backtracking, shorter lines should help a lot. For a static string it should just be a matter of memory usage. –  frostschutz Aug 5 at 23:50

2 Answers 2

You should probably just use fmt or fold or pr or one of those. Here's fmt:

until [ $((i=i+1)) -gt 10000 ]
do  printf %s\  words and more words
done | fmt

OUTPUT

words and more words words and more words words and more words words
and more words words and more words words and more words words and more
words words and more words words and more words words and more words
words and more words words and more words words and more words words
and more words words and more words words and more words words and more
#...and so on for a long time

grep's buffer is bound to give way under any serious load unless it can dump it at a newline boundary.

For example:

time (
    tr \\0 \\n < /dev/zero | 
    sed 'c words and words and words and words and words and' | 
    tr -d \\n | 
    grep -o words
)
grep: memory exhausted
49.42s user 44.93s system 229% cpu 41.070 total

It takes a while - but it's got a lot of empty RAM on this system, and still managed to go bust in under a minute.

So just:

fmt <file | grep search

More robustly though, if there is some reliable single character string delimiter in the file:

tr ${delim} \\n <file | grep search
share|improve this answer

grep definitely works line based, as does sed. It won't necessarily speed up searching, but producing the output for a matching line should be quicker.

share|improve this answer
    
I just added in line breaks every 1000 char and it speeded it up like WHOA sed -e "s/.\{1000\}/&\n/g" < temp.txt –  akh2103 Aug 5 at 0:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.