5

I have this grep command to find files with matching pattern.

grep -oiE 'pattern1|pattern2|pattern3|pattern4' *pattern_in_a_filename* | sort -u

Output:

file_one:pattern1
file_two:pattern2
file_two:pattern3

What I want is to output the pattern4 saying that is not being found like this example:

file_one:pattern1
file_two:pattern2
file_two:pattern3
pattern4 not found

I'm doing thousands of thousands of patterns and it has to be done as quick as possible because these are crucial data needed in our operations.

  • what if only one pattern was matched? – RomanPerekhrest Oct 17 '17 at 9:12
  • @RomanPerekhrest Then it will do the same like pattern4. – WashichawbachaW Oct 17 '17 at 9:13
  • *file_name* denotes multiple files? – RomanPerekhrest Oct 17 '17 at 9:17
  • yup dude! It's saying that there are bunch of different filenames with that pattern in a directory. I'll edit and clarify it for all. – WashichawbachaW Oct 17 '17 at 9:21
  • Can you not just loop over all the patterns? – Rastapopoulos Oct 17 '17 at 9:33
2

Here's an sh script that produces the results you need.

#!/bin/sh

grep -f /path/to/patterns.txt /path/to/*_856_2017* | sort -u > /path/to/foundFiles.txt 

while read -r LINE
do
    grep -F "$LINE" /path/to/foundFiles.txt
    if [ $? -eq 1 ]
    then
        echo "$LINE" not found
    fi
done < /path/to/patterns.txt

In this script, I assume you output the results of your grep to the file found.txt, and that you store your patterns in the file /path/to/foundFiles.txt.

As you can see, the grep in the loop will produce the same contents of the file found.txt while adding "$pattern" not found for the missing ones.

I also devised a second approach to your case:

#!/bin/sh

grep -f /path/to/patterns.txt /path/to/*_856_2017* |
    sort -u > /path/to/foundFiles.txt

comm -23 /path/to/patterns.txt /path/to/foundFiles.txt |
    xargs -L 1 -I {} echo {} not found > /path/to/notFoundFiles.txt

cat /path/to/foundFiles.txt /path/to/notFoundFiles.txt > /path/to/finalList.txt

In this case, patterns.txt needs to be already sorted for comm to work.

The comm command compares the two files returning the lines present only in patterns.txt (-23 parameter), which is the list of patterns not found by grep.

Then, xargs grabs every line (-L 1) and echoes the line ({}) with " not found" appended to it. The result of xargs is redirected to the notFoundFiles.txt file.

Finally, you simply concatenate foundFiles.txt and notFoundFiles.txt into finalList.txt.

  • Almost the same with my script. But as I said, it takes 20 secs more or less to execute only 1 pattern. Need things to be faster – WashichawbachaW Oct 18 '17 at 0:02
  • This loop runs on the results of your command; it should take like one second total. Did you try it? – cesarv Oct 18 '17 at 5:14
  • Try to refer on my other question. unix.stackexchange.com/questions/397402/… My problem with my script is #!/bin/sh seems not reading correctly on *856* as * as wildcard (When it's inside the while loop) Try to run it yourself – WashichawbachaW Oct 18 '17 at 5:38
  • You will need to add the -F parameter to the grep command in my script to treat every pattern as a fixed string. – cesarv Oct 18 '17 at 5:41
  • And also you did cat patterns.txt but you call on while read -r pattern which doesn't make sense where did you get that pattern. I think you have mistaken pattern with $LINE. – WashichawbachaW Oct 18 '17 at 5:42
6

it has to be done as quick as possible

Want to get things done as quick as possible?

Discover parallel processing and apply the following GNU parallel solution:

The right way starts with placing all your patterns into file, let's say patterns.txt.

The crucial job:

cat patterns.txt | parallel -j 0 --no-notice 'grep -Hoi {} /path/to/files/* || echo "{} not found"' | sort -u
  • -j N - Number of jobslots. Run up to N jobs in parallel. 0 means as many as possible. Default is 100% which will run one job per CPU core.

The above command will search for each pattern from patterns.txt within all files in parallel.

I've made a test having only 2 CPU cores with some list of patterns and a few csv files of 641 Mb each. And I've got about 470% time speed increase in comparison with consecutive processing.
The more CPU cores your computer has the more time speed performance you reach.

Make your conclusions ...

https://www.gnu.org/software/parallel/man.html

  • -bash: parallel: command not found. Does it have to be installed? Unfortunately, I don't have a superuser privilege. – WashichawbachaW Oct 18 '17 at 2:41
  • your answer was very good but there's this big problem. when I execute that command, it returns the files with the patterns in it but also it returns all the pattern from pattern.txt as not found. And that is not good. A little more tweak on the command and your answer will be my best answer – WashichawbachaW Oct 18 '17 at 3:15
  • @WashichawbachaW, it's obvious that new programm should be installed beforehand. Make sure that you are using GNU parallel, check parallel --version. As I wrote - I've tested your "scheme" and it worked fine. I suppose that the issue is on your side and you missed something or made some mistakes or your actual conditions differ from those mentioned in the question – RomanPerekhrest Oct 18 '17 at 6:15
  • I have this now GNU parallel 20170922 Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017 Ole Tange and Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> – WashichawbachaW Oct 18 '17 at 6:45
  • 1
    @WashichawbachaW The warning is a warning only: You should get the correct results. It is simply saying it can only run 252 in parallel. The rest will be run, when the first 252 have finished. If you want to run the 1800 processes in parallel, try the trick: gnu.org/software/parallel/… – Ole Tange Oct 18 '17 at 20:16
5

The tool grep searches files by patterns. This means the pattern is the input and the file is the output. And this means all you can find with grep are files but not patterns.

In order to find those files, which do not contain a matching pattern, you have to invert the search with -v. This requires two calls.

Example:

$ echo a > xa
$ echo ab > xab
$ echo c > xc
$ { grep -oiE 'a|b' x*; grep -vl -E 'a|b' x*; } | sort -u
xa:a
xab:a
xab:b
xc

In order to find those patterns, which did not match, you have to make your patterns the input and the search result the pattern. The list of matches becomes the pattern and the pattern becomes the data, in which you have to search.

Lets assume this may be the pattern for the files from the preceding example.

$ pattern='a|b|d'

Then this stores the list of matching patterns in an array:

$ found=($(grep -hoiE "$pattern" x* | sort -u))

And this converts the array into the new pattern:

$ new_pattern="$(IFS='|' ; echo "${found[*]}")"

And this converts the original pattern into data:

$ data="${pattern//|/$'\n'}"

Then this is the list of patterns, which did not match:

$ grep -viE "$new_pattern" <<< "$data"
d
  • Can you have an example which makes the patterns inside a txt file? – WashichawbachaW Oct 18 '17 at 2:26
  • pattern=$(cat pattern.txt) – ceving Oct 22 '17 at 18:58
3

Here is a way to achieve what you expect. Since you stated that you have thousands of patterns, so I'm assuming you feed patterns from a file, then that's much better than passing direct patterns to grep. The command is as following.

grep -oif ../patt_file file* && 
    printf "%s Not Found.\n" $(grep -vFxf <(grep -oihf ../patt_file file*) \
    <(< ../patt_file))

To avoid matching patterns file patt_file itself, you need to move it to different directory than where your input infiles* are there.

  • 1
    Thanks for the answer. I'll try this tomorrow when I get back to work – WashichawbachaW Oct 17 '17 at 11:44
  • this will print Not Found.\n even if all patterns were found which is not pretty good – RomanPerekhrest Oct 17 '17 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.