I have many generated logs in /var/opt/log_loc1,/var/opt/log_loc2, /var/opt/log_loc3 from different cron jobs. Instead of looking in each log, is there a way to schedule a cron job that populates the new logfile with all logs that have Error message in them?

Basically I want to populate with all log names and the error in it a new logfile with "all_logs_%Y_%m_%d" as a name. Each day, I will have a new all_logs_%Y_%m_%d

Example: Here's a list of my logfiles in /var/opt/log_loc1

  1. InsertSales_2019_08_19.log
  2. PurgeHistory_2019_08_19.log
  3. Import11_2019_08_19.log
  4. ImportSales_2019_08_19.log
  5. Import22_2019_08_20.log
  6. Import33_2019_08_20.log
  7. InsertSales_2019_08_20.log
  8. PurgeHistory_2019_08_20.log
  9. Import11_2019_08_20.log
  10. ImportSales_2019_08_20.log
  11. Import22_2019_08_21.log
  12. Import33_2019_08_21.log
  13. InsertSales_2019_08_21.log
  14. PurgeHistory_2019_08_21.log

*Apologies if my question is incomplete

  • Since you have a vim tag, are you asking for vim $(grep -l Error /var/opt/log*) to start vim with a list of files that include the word Error?
    – icarus
    Aug 23 '19 at 6:01
  • Do you want to combine all log lines which are somehow marked as errors? In chronological order? Without reprocessing any lines you have already processed (crucial if you have a lot of logs)? Please also provide a couple example files with some data and what you expect the result to look like. Because unless you just want a naive grep Error /var/opt/log* > errors this is not a trivial task.
    – l0b0
    Aug 23 '19 at 7:04
  • @icarus sorry I am actually a DBA. I usually use vi command in linux. Aug 23 '19 at 8:45
  • @l0b0 i've edited my entry. let me know if that's enough Aug 23 '19 at 8:46
  • OK, you have shown us some typical input filenames. What is a typical output filename that you want and more importantly what goes into this new file? All the lines containing Error from the current logs? Have you considered moving the logs so that instead of /var/opt/log_loc1/InsertSales_2019_08_21.log you have /var/opt/log_loc1/2019/08/21/InsertSales.log which will reduce the number of files in any given directory?
    – icarus
    Aug 23 '19 at 11:26

What about find?, the easiest way I can think is using it, maybe it's a little rough but it must do the job

find /var/opt/ -type f -name "*.log" -exec grep -iH 'err\|warn\|fail\|whatever' {} >> total.log \;

It's not clear for me if you just want to filter all log files (from all the days) or just from today, if you need to filter with date you can use -mtime option from find or just make a "trick" with -name option to only filter, for example, logs from yesterday

find /var/opt/ -type f -name "*_`date --date yesterday '+%Y_%m_%d'`.log" -exec grep -iH 'err\|warn\|fail\|whatever' {} >> total.log \;

Not sure if this can help you... Other thing that is not clear is, what kind of syntax the logs files have? Coz find don't find in alphabetical or numerical order, so you will get a chaotic file. Usually (and must be a dogma), logs start with %Y%m%d_%H:%M or something like that, if that's the case you can sort the output of find in a way like

find /var/opt/ -type f -name "*.log" -exec grep -iH 'err\|warn\|fail\|whatever' {} \; sort -no total.log

I tried this un my /var/log and looks like it do the job, but as I said, it's a rough way.

Hope this can help you.

You have more info about the options I used with find, sort and grep in the man pages.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.