I have directory with cca 26 000 files and I need to grep in all these files. Problem is, that I need it as fast as possible, so it's not ideal to make script where grep will take name of one file from find command and write matches to file. Before "arguments list too long" issue it took cca 2 minutes to grep in all this files. Any ideas how to do it? edit: there is a script that is making new files all the time, so it's not possible to put all files to different dirs.
(but beware that unless you have a recent version of GNU grep, that will follow symlinks when descending into directories). Non-regular files won't be searched unless you add a
Very old versions of GNU
Performances are likely to be I/O bound. That is the time to do the search would be the time needed to read all that data from storage.
If the data is on a redundant disk array, reading several files at a time might improve performance (and could degrade them otherwise). If the performances are not I/O bound (because for instance all the data is in cache), and you have multiple CPUs, concurrent
For instance, if the data is on a RAID1 array with 3 drives, or if the data is in cache and you have 3 CPUs whose time to spare:
However note that if the output of
(on a recent GNU or FreeBSD system)
If it's not multi-byte character data, or if for the matching of that pattern, it doesn't matter whether the data is multi-byte character or not, then:
could improve performance significantly.
If you end up doing such searches often, then you may want to index your data using one of the many search engines out there.
26000 files in a single directory is a lot for most filesystems. It's likely that a significant part of the time is taken reading this big directory. Consider splitting it into smaller directories with only a few hundred files each.
All files in the directory