Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Imagine we have a lot of huge(30000 line)files: a.2014-05-06, a.2014-05-07, a.2014-05-08 and so on.

I know that we can grep a line with word using this command:

grep "word" a.*

First I guess this will open every file and close it before opening the second file to search for the word? is this efficient, if not is there a way more efficient?

Second How to determine the file which we find the word in? for example:
if a.2014-05-06 have:

a
bx
.
.

a.2014-05-07:

by
.
.

a.2014-05-08:

c
.
.

and we do the following:

grep "b" a.*

the output will be:

bx
by

I want output like this:

bx  a.2014-05-06
by  a.2014-05-07
share|improve this question

6 Answers 6

up vote 3 down vote accepted

First I guess this will open every file and close it before opening the second file to search for the word? is this efficient, if not is there a way more efficient?

Yes, grep will open and search every file in turn. On most setups, that's the most efficient way. Unless the regexp is extremely complex, this task is firmly I/O-bound, i.e. the performance bottleneck is reading from the disk, and your CPU will not be taxed.

On some setups, I/O can be parallelized; for example, if you have a RAID-1 or RAID-0 configuration, then the two (or more) components in the RAID array can be read from in parallel, which will save time. If you have such a setup, you can call a tool like GNU Parallel to call two instances of grep (see the manual for command examples). On most setups, calling two instances of grep in parallel will be slower, because the disk heads will keep switching between the files accessed by the two instances (with SSD, calling two instances in parallel will typically not cause a major slowdown, but it won't be faster either).

If you pass more than one file on the command line, grep outputs the file name before each match, in the format

path/to/file:line containing a match

If you're using a wildcard pattern or some other forms of generating file names and you want to display the file name even in the case when there happens to be a single matching file, tell grep to search the empty null device as well.

grep REGEX /dev/null *.txt

(grep -H REGEX *.txt is similar, but using /dev/null has the additional benefit that it works seamlessly even if the list of matching files is empty, whereas grep -H REGEX reads from standard input.)

share|improve this answer
    
+1 for introducing me to GNU Parallel, It seems very helpful indeed –  Networker Aug 13 at 1:09

Use perl:

perl -nle 'print "$1 $ARGV" if /(pattern)/' a.*

And for your question, yes, grep open each file to search then close it, open next file and so on.

$ strace -e trace=file,close grep Power 1.txt 2.txt
....
openat(AT_FDCWD, "1.txt", O_RDONLY)     = 3
1.txt:Power and signal 
1.txt:VDD Digital Power This pin provides power supply connection for the digital
1.txt:VMEASPOS Digital Power Voltage to be measured.
1.txt:VREFEXT Digital Power Reference voltage input of 1.024V %for VSENS calibration.
close(3)                                = 0
openat(AT_FDCWD, "2.txt", O_RDONLY)     = 3
2.txt:0.078362 Power
2.txt:Power
close(3)

I can't think any better way to do it. Almost text processing tools also behaves like this. The only difference I can see is what system call they use. grep use openat(), while perl, awk use open().

share|improve this answer
    
Thanks this works, and is using grep is good for performance? –  Networker Aug 12 at 6:23

From man grep:

       -H, --with-filename
           Print the file name for each match.  This is the default when there is
           more than one file to search.

It will print the filename first, followed by the match; which isn't what you've shown in your example results. But it is quick and easy if that doesn't cause a problem.

As it's default for more than one input then using it with a wildcard (as in your example) results in:

$ grep "b" a.*
a.2014-05-06:bx
a.2014-05-07:by

You didn't mention which flavour of Unix/Linux you're using, but the -H option is available in most implementations although it isn't in the POSIX spec.

share|improve this answer

To search a disk without opening every file:

dd if=/dev/${disk_device} |
grep -b 'some regex'

Actually, I like this a lot better:

sudo cat /dev/${some_disk} |
tr -c '[:print:][:space:]' '\n\n' |
grep -b 'some regex'

The -b option will provide you the byte offsets for all matches. You can afterward check with the filesystem what files exist at those offsets.

In the second form you avoid grep complaining about binary filetypes AND automatically speed along its search by providing newlines in place of irrelevant data.

P.S. - If your filesystem can be defragmented, it might be a good idea to do that first.

share|improve this answer

Two other things to keep in mind when grep-ing in large (or many) files:

  1. If you are searching for a fixed string instead of a pattern, add the -F option to grep, it will speed up your search tremendously (see Source)

  2. If you know that you are actually looking for a word, i.e. your search pattern is at bounded by non-word characters or beginning/end of the line, then add the -w option. This will speed up the search, I think.

share|improve this answer
    
+1, brilliant source –  Networker Aug 14 at 11:17

This is an 'alternative' answer to your question. I thought about comment for a while but in the end decided it might be an answer for some. Also it seems too long for a readable comment.

If your objective is searching a bunch of files for a string in a faster way and are then finding grep too slow, you might be able to try git grep. Requires git to be installed and a repository to be created.

I find this seems to be faster and easy to use.

git is a distributed version control system. It also has feature like git grep [string] for searching which I find to be blazing fast. I believe this may be because of the way data in indexed and stored.
Get git at http://git-scm.com/
Downloads for Mac/Linux/Windows/Solaris at http://git-scm.com/downloads

You can create a git repository for a current project by simply typing git init in the root folder.

Once done then you can type git grep [string]

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.