I had asked a question a while ago about how to pull specific info from a log file, and got some useful answers, but I don't quite have what I need yet. What I have so far:


mkdir result

cd $1
ls | while read -r file;
egrep "ERROR|FAIL|WARN" > ../errors-$file
cd ..

mv errors-* result

Here's what I want in total: the script scans through a directory of logs, grabs each line of text (some are multi-line) that includes ERROR|FAIL|WARN, and outputs them in a directory called "result" with each individual file being named based on their source file.

It works when my target log directory only has one text file in it, but it doesn't work when my target directory has directories and log files at different levels within it. I know it's because the "../errors-$file" bit in the script, but I'm not sure how to change it.

Any help would be great, and I apologize in advance if it's poorly phrased.

  • You use the argument $1, what is the value of that argument when you normally run that script? – Centimane Aug 12 '16 at 18:05
  • Also, you say it doesn't work but what does that mean? What actually happens when you run your script? – Centimane Aug 12 '16 at 18:08
  • 1
    The egrep line should be egrep "ERROR|FAIL|WARN" $file > ../errors-$file – Rui F Ribeiro Aug 12 '16 at 18:17
  • @Dave I follow the ./logsifter.sh with the name of the folder I want to scan through. I'm still new to UNIX so I'm not entirely sure if that's what you're asking for though. As for when it doesn't work - the only output in the "result" folder is an empty file that takes its name from the first item inside the target directory, and which is a subdirectory. – Patremagne Aug 12 '16 at 18:22
  • @RuiFRibeiro What does that change? – Patremagne Aug 12 '16 at 18:24

I'll poke at your loop itself, and try to walk through what it's actually doing, and then look at how you might achieve what you want.

How you initiate your loop is overly complex: ls | while read -r file;. You're piping the results of ls to your while read, not a big deal, still works, though I would recommend a for loop more like:

for file in ./* ;

This will loop over the same files, but is a little more straight forward. Also the ls command can return output that is not just the filenames (ever notice that ls has different font colors for directories/executables?), and this can cause other commands grief.

Your command: egrep "ERROR|FAIL|WARN" > ../errors-$file is trying to egrep for ERROR|FAIL|WARN in nothing, and redirecting output to ../errors-$file.

The syntax for egrep is:

egrep [search pattern] [location]

So you're missing the location for egrep to look in. You can fix this by adding $file after your pattern, so it becomes:


mkdir result

cd $1
for file in $(find ./ -type f)
egrep "ERROR|FAIL|WARN" $file > ../result/errors-$(basename $file)
echo "-------------------------------" >> ../result/errors-$(basename $file)
egrep "denied" -B 1 $file >> ../result/errors-$(basename $file)
  • Thanks, this is super helpful! Does the fact that there are subfolders that also include logs within the scanned directory affect the code you supplied? – Patremagne Aug 15 '16 at 13:12
  • Also, you mentioned -h prevents the filename being included in the output. I'd actually like it in the output, as the script will be scanning several log files and ouputting the ERROR|FAIL|WARNs into separate files that are named based on their original. Can I just remove the -h? – Patremagne Aug 15 '16 at 13:18
  • @Patremagne find will look in subfolders, but {} would expand to include the directory name, so it may be better to stick to your for loop (else the find -exec becomes more complicated to include a subshell). As for the -h, even in your prior example the output from grep should only be the matching line. The -h option ensures the output is only the matching line. If you use -H then the output will be: [filename]:[matching line] in your output files. But if the files you're putting the output into are named based on the original you may not need the contents to mention the original. – Centimane Aug 15 '16 at 15:53
  • I see. I tried the script on a directory that had a text file as well as a folder (with text files in it) beneath it and only got a single "errors {}" file for output, when I would've expected (or wanted) one for each text file. Any ideas? – Patremagne Aug 16 '16 at 17:21
  • @Patremagne use your for loop instead, I'm going to edit out the find exec suggestion – Centimane Aug 16 '16 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.