5

I have this grep command to find files with matching pattern.

grep -oiE 'pattern1|pattern2|pattern3|pattern4' *pattern_in_a_filename* | sort -u

Output:

file_one:pattern1
file_two:pattern2
file_two:pattern3

What I want is to output the pattern4 saying that is not being found like this example:

file_one:pattern1
file_two:pattern2
file_two:pattern3
pattern4 not found

I'm doing thousands of thousands of patterns and it has to be done as quick as possible because these are crucial data needed in our operations.

6
  • what if only one pattern was matched? Oct 17, 2017 at 9:12
  • @RomanPerekhrest Then it will do the same like pattern4. Oct 17, 2017 at 9:13
  • *file_name* denotes multiple files? Oct 17, 2017 at 9:17
  • yup dude! It's saying that there are bunch of different filenames with that pattern in a directory. I'll edit and clarify it for all. Oct 17, 2017 at 9:21
  • Can you not just loop over all the patterns? Oct 17, 2017 at 9:33

4 Answers 4

7

it has to be done as quick as possible

Want to get things done as quick as possible?

Discover parallel processing and apply the following GNU parallel solution:

The right way starts with placing all your patterns into file, let's say patterns.txt.

The crucial job:

cat patterns.txt | parallel -j 0 --no-notice 'grep -Hoi {} /path/to/files/* || echo "{} not found"' | sort -u
  • -j N - Number of jobslots. Run up to N jobs in parallel. 0 means as many as possible. Default is 100% which will run one job per CPU core.

The above command will search for each pattern from patterns.txt within all files in parallel.

I've made a test having only 2 CPU cores with some list of patterns and a few csv files of 641 Mb each. And I've got about 470% time speed increase in comparison with consecutive processing.
The more CPU cores your computer has the more time speed performance you reach.

Make your conclusions ...

https://www.gnu.org/software/parallel/man.html

11
  • -bash: parallel: command not found. Does it have to be installed? Unfortunately, I don't have a superuser privilege. Oct 18, 2017 at 2:41
  • your answer was very good but there's this big problem. when I execute that command, it returns the files with the patterns in it but also it returns all the pattern from pattern.txt as not found. And that is not good. A little more tweak on the command and your answer will be my best answer Oct 18, 2017 at 3:15
  • @WashichawbachaW, it's obvious that new programm should be installed beforehand. Make sure that you are using GNU parallel, check parallel --version. As I wrote - I've tested your "scheme" and it worked fine. I suppose that the issue is on your side and you missed something or made some mistakes or your actual conditions differ from those mentioned in the question Oct 18, 2017 at 6:15
  • I have this now GNU parallel 20170922 Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017 Ole Tange and Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> Oct 18, 2017 at 6:45
  • 1
    @WashichawbachaW The warning is a warning only: You should get the correct results. It is simply saying it can only run 252 in parallel. The rest will be run, when the first 252 have finished. If you want to run the 1800 processes in parallel, try the trick: gnu.org/software/parallel/…
    – Ole Tange
    Oct 18, 2017 at 20:16
6

The tool grep searches files by patterns. This means the pattern is the input and the file is the output. And this means all you can find with grep are files but not patterns.

In order to find those files, which do not contain a matching pattern, you have to invert the search with -v. This requires two calls.

Example:

$ echo a > xa
$ echo ab > xab
$ echo c > xc
$ { grep -oiE 'a|b' x*; grep -vl -E 'a|b' x*; } | sort -u
xa:a
xab:a
xab:b
xc

In order to find those patterns, which did not match, you have to make your patterns the input and the search result the pattern. The list of matches becomes the pattern and the pattern becomes the data, in which you have to search.

Lets assume this may be the pattern for the files from the preceding example.

$ pattern='a|b|d'

Then this stores the list of matching patterns in an array:

$ found=($(grep -hoiE "$pattern" x* | sort -u))

And this converts the array into the new pattern:

$ new_pattern="$(IFS='|' ; echo "${found[*]}")"

And this converts the original pattern into data:

$ data="${pattern//|/$'\n'}"

Then this is the list of patterns, which did not match:

$ grep -viE "$new_pattern" <<< "$data"
d
2
  • Can you have an example which makes the patterns inside a txt file? Oct 18, 2017 at 2:26
  • pattern=$(cat pattern.txt)
    – ceving
    Oct 22, 2017 at 18:58
3

Here is a way to achieve what you expect. Since you stated that you have thousands of patterns, so I'm assuming you feed patterns from a file, then that's much better than passing direct patterns to grep. The command is as following.

grep -oif ../patt_file file* && 
    printf "%s Not Found.\n" $(grep -vFxf <(grep -oihf ../patt_file file*) \
    <(< ../patt_file))

To avoid matching patterns file patt_file itself, you need to move it to different directory than where your input infiles* are there.

2
  • 1
    Thanks for the answer. I'll try this tomorrow when I get back to work Oct 17, 2017 at 11:44
  • this will print Not Found.\n even if all patterns were found which is not pretty good Oct 17, 2017 at 13:34
2

Here's an sh script that produces the results you need.

#!/bin/sh

grep -f /path/to/patterns.txt /path/to/*_856_2017* | sort -u > /path/to/foundFiles.txt 

while read -r LINE
do
    grep -F "$LINE" /path/to/foundFiles.txt
    if [ $? -eq 1 ]
    then
        echo "$LINE" not found
    fi
done < /path/to/patterns.txt

In this script, I assume you output the results of your grep to the file found.txt, and that you store your patterns in the file /path/to/foundFiles.txt.

As you can see, the grep in the loop will produce the same contents of the file found.txt while adding "$pattern" not found for the missing ones.

I also devised a second approach to your case:

#!/bin/sh

grep -f /path/to/patterns.txt /path/to/*_856_2017* |
    sort -u > /path/to/foundFiles.txt

comm -23 /path/to/patterns.txt /path/to/foundFiles.txt |
    xargs -L 1 -I {} echo {} not found > /path/to/notFoundFiles.txt

cat /path/to/foundFiles.txt /path/to/notFoundFiles.txt > /path/to/finalList.txt

In this case, patterns.txt needs to be already sorted for comm to work.

The comm command compares the two files returning the lines present only in patterns.txt (-23 parameter), which is the list of patterns not found by grep.

Then, xargs grabs every line (-L 1) and echoes the line ({}) with " not found" appended to it. The result of xargs is redirected to the notFoundFiles.txt file.

Finally, you simply concatenate foundFiles.txt and notFoundFiles.txt into finalList.txt.

7
  • Almost the same with my script. But as I said, it takes 20 secs more or less to execute only 1 pattern. Need things to be faster Oct 18, 2017 at 0:02
  • This loop runs on the results of your command; it should take like one second total. Did you try it?
    – cesarv
    Oct 18, 2017 at 5:14
  • Try to refer on my other question. unix.stackexchange.com/questions/397402/… My problem with my script is #!/bin/sh seems not reading correctly on *856* as * as wildcard (When it's inside the while loop) Try to run it yourself Oct 18, 2017 at 5:38
  • You will need to add the -F parameter to the grep command in my script to treat every pattern as a fixed string.
    – cesarv
    Oct 18, 2017 at 5:41
  • And also you did cat patterns.txt but you call on while read -r pattern which doesn't make sense where did you get that pattern. I think you have mistaken pattern with $LINE. Oct 18, 2017 at 5:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .