1

There are many files in a directory (logs from Java application - one record per line) and I would like to list all filenames that contain a searched string plus list all occurrences of that string in the particular file.

I've been using this until now. Here I get file names and bellow I get matched lines, but I do not know from what filename matched line come from.

#!/bin/bash
cd ${DIRECTORY}
clear
echo 'WARNINGS'
egrep -l "WARN" * | sort
echo ''
cat * | grep 'WARN'

I need to list all filenames (but only that that contain WARN) and for all these matched files show all lines with WARN string in the particular file.

6
  • Are you just looking for grep WARN * instead of cat * | grep WARN? Is that the output format you want?
    – terdon
    Commented May 2, 2023 at 11:59
  • @terdon I specified my request into original question.
    – jnemecz
    Commented May 2, 2023 at 12:05
  • 1
    Yes, I know that's what you want and that is what grep WARN * does, so please try it and then tell us if that is the format you want. If it isn't, show us an example of the format you want.
    – terdon
    Commented May 2, 2023 at 12:08
  • Somewhat related: Why does grep sometimes show the file name, sometimes not? Commented May 2, 2023 at 12:09
  • @terdon Yes, sorry, I overlooked that grep WARN * show filename at the very beginning of the line. This is suitable enough for my needs, thanks!
    – jnemecz
    Commented May 2, 2023 at 12:12

3 Answers 3

5

A little known fact about grep is that it can be brought to print the name of the file it (successfully) searches when more than one file is being searched. That means that if you include /dev/null into the list of files to search you will get exactly what you want:

$ grep "something" /path/to/file
something (and some other thing)

whereas:

$ grep "something" /dev/null /path/to/file
/path/to/file:something (and some other thing)

Using this you could do (a variation of) the following:

$ find /path/to/start [-name "<filename pattern>"] \
                       -exec grep '<searchstr>' /dev/null {} \;

This would give you an output like this:

/path/to/start/file1:<searchstr> foo
/path/to/start/file1:<searchstr> bar
/path/to/start/subdir/file2:foo <searchstr> bar
/path/to/start/subdir/file3:bar <searchstr> foo
...

As @ilkkachu correctly pointed out the command I gave will pass one file at a time to grep. If many files are to be searched this can be quite taxing for the system. One way to optimize the process is to write it like this:

$ find /path/to/start [-name "<filename pattern>"] \
                       -exec grep '<searchstr>' /dev/null {} +

Where find passes several filenames to grep at once, resulting a smaller load because grep is not called so often. You still need to specify /dev/null because it is possible that find only finds one file, though.

6
  • You can also use -exec grep 'pattern' /dev/null {} + here, it'll tell find to (possibly) pass multiple filenames to grep at once. You still need to add the /dev/null as it might end up with passing just one, it's just an optimization.
    – ilkkachu
    Commented May 2, 2023 at 14:43
  • @ilkkachu: you are right. I was more focused on showing the principle, though, than presenting an optimized solution. I will add your suggestion.
    – bakunin
    Commented May 2, 2023 at 14:47
  • yes, and that's ok. It's just that {} + doesn't really cost anything compared to {} \; here, and it's easier to write in a few keyboard layouts, so it's something to keep advocating for. :)
    – ilkkachu
    Commented May 2, 2023 at 14:50
  • No need for /dev/null. The OP has many files, that's kind of the point of the question.
    – terdon
    Commented May 2, 2023 at 15:19
  • 1
    @terdon: that might be the case, but why take the risk? I cannot guarantee that there will always be "many files" and I like my scripts to work (that is: produce a defined result) no matter what. If /dev/null is left out the script will produce different results if there is one file or two (or more) files.
    – bakunin
    Commented May 2, 2023 at 15:25
5

When you give multiple files to grep, it appends the file name to each line of output. Also, this way, you avoid a classic useless use of cat. While you're at it, avoid using CAPS for shell variable names since, by convention, global environment variable names are capitalized, so if you also capitalize your own shell variables, this can lead to naming collisions and bugs. Finally, egrep is deprecated in favor of grep -E (but you don't need grep -E either here) and you don't need to cd into a directory before running things in it, but if you do, you should also make sure the cd worked before doing anything. Here is an improved version of your script that does what you want:

#!/bin/sh

clear
printf 'WARNINGS found in files in directory "%s":\n' "$1"
grep 'WARN' "$1"/*

You can now run the script giving it the target directory name as an argument:

your_script /path/to/target

For example, running it on a directory named /home/terdon/foo on my system gives:

WARNINGS found in files in directory "/home/terdon/foo":
/home/terdon/foo/file2:WARNING from file2
/home/terdon/foo/file4:WARNING from file4
/home/terdon/foo/file4:WARNING2 from file4
/home/terdon/foo/file5:WARNING from file5

If you only want to show the file name without the path, you can choose to cd (but as mentioned above, make sure to exit if the cd fails):

#!/bin/sh

clear
if cd -- "$1"; then
  :
else
  echo "cd to '$1' failed!"
  exit 1
fi

printf 'WARNINGS found in files in directory "%s":\n' "$1"

grep 'WARN' *

Or you can just strip the path from the output:

#!/bin/sh

clear
printf 'WARNINGS found in files in directory "%s":\n' "$1"
grep 'WARN' "$1"/* | sed 's|.*/||'

Finally, if you want to ensure this works even when your target directory only has one file, if you are using GNU grep (the default on Linux) you can use the -H flag which tells grep to always include the file name:

#!/bin/sh

clear
printf 'WARNINGS found in files in directory "%s":\n' "$1"
grep -H 'WARN' "$1"/* | sed 's|.*/||'

If your grep doesn't support -H, use bakunin's trick of including /dev/null:

#!/bin/sh

clear
printf 'WARNINGS found in files in directory "%s":\n' "$1"
grep 'WARN' "$1"/* /dev/null | sed 's|.*/||'
0

Better use ack in your Java repository.

That will prevent to search in .git directory or similar, so it's far better, faster.

There's more options available to be better than grep in this situation.

Try (recursively):

ack WARN
ack -l WARN
1
  • None of the approaches here would search through hidden directories. They won't even search through directories at all, grep foo * only searches through files, directories will be skipped with a warning message, and cat * | grep foo also will only search for files. And since the * ignores hidden files unless the user has specifically set shopt -s dotglob, all of the approaches will ignore something like .git. And why make this recursive when the OP is explicitly not using a recursive search?
    – terdon
    Commented May 3, 2023 at 10:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .