0

I have ~750 log directories I need to search for the latest log file available:

These are captured by the allLogs.txt file and each entry contains a wild card '*' so that ONLY the log with the latest date is grabbed (there can be many logs of the same name with different dates. e.g.

import_Asbeg_DA_Config3_*.log

import_test_deploy_*.log

import_Plus_DQ_Config1_*.log

e.g. I know this is not pretty but it works great to accomplish the goal in a loop.

for LATEST in `cat allLogs.txt` ;
do
    find -name "${LATEST}" -type f | sort -n | tail -1

done < infile > outfile

This finds ~350 records because all the directories might not have a log file.

"allLogs.txt" contains all the ~750 paths to the log file for that directory.

What I am trying to figure out how to run this command in my loop BUT, maintain consistency even if no log file is found. Perhaps by appending 'INVALID' to the records that DO NOT contain a log file. I've tried combinations of echo with the find command but no luck. Any ideas?

  • What do the entries in allLogs.txt look like? – Jeff Schaller Nov 28 '18 at 13:58
  • import_Asbeg_DA_Config3_.log import_test_deploy_.log import_Plus_DQ_Config1_*.log – SSDdude Nov 28 '18 at 14:09
  • Might have to edit those into the question so that the wildcards are visible... – Jeff Schaller Nov 28 '18 at 14:11
  • Hmm... not sure I understand that. I have to have the logs this way to grab only the latest log because there are many older logs of the same name in in those paths. Thoughts? – SSDdude Nov 28 '18 at 14:15
  • What I meant was, the formatting of comments makes it hard to interpret what's in your input; please edit your question to make it clearer. Thanks! – Jeff Schaller Nov 28 '18 at 14:16
1

Based on your question, I think what you are really asking is to print out the latest file in the directory, if there is any file in the directory. If there are no files in any directory, then print out INVALID, or something similar.

If the above is a correct summarization, you can add an if clause to your loop that checks to see if there is only a single line outputted when you do a find on that directory. If there is only a single line, that is the directory. Hence, there are no files. You can then skip any further commands within the loop and continue to the next item in the list.

Here is the updated script:

#!/bin/bash

for LATEST in $(cat allLogs.txt); do
    if [[ $(find ${LATEST} -print | wc -l) -eq 1 ]]; then
        echo "${LATEST} has no files"
        continue
    fi
    find ${LATEST} -type f | sort -n | tail -1
done

exit 0

I did not want to just print INVALID as you would not know which directory was printing out INVALID.

I wasn't sure why you were redirecting stdin for the loop. I did not redirect stdout, but that is easy enough to add.

  • Hey Lewis thanks so much... This looks like its working great. One question, how do I output stdout to a file? I tried adding "> outfile:" after done but no log is created. I also tried kicking off the script and indicating all should to logfile... e.g. ./script.sh > outfile but still no log. thanks! – SSDdude Nov 28 '18 at 15:00
  • Really? "./script.sh >outfile" should have worked. That's the standard. Do you get any file called "outfile"? Is there anything in it? – Lewis M Nov 28 '18 at 15:04
  • yup I get "outfile" but its blank!... really scratching my head! Thx!! – SSDdude Nov 28 '18 at 15:08
  • What is in outfile if you run the command, "w >outfile"? – Lewis M Nov 28 '18 at 15:09
  • Also I should have mentioned I cannot do this when I kick off the script... it has to be generated from with in the script. THX! – SSDdude Nov 28 '18 at 15:10
1

As in the answer you already have, I assume that you have many directories, each one containing zero or more files whose names are composed by a common part plus a number for ordering, and that the common parts of those names are listed in your file allLogs.txt.

You can use something like this to add a line to outfile for each name in allLogs.txt that has no matches found by find:

#!/bin/bash

while read -r LATEST; do
  find -name "${LATEST}" -type f | tee >( [ "$(wc -l)" = 0 ] && printf '%s\n' "${LATEST} INVALID" )  | sort -n | tail -1
done < allLogs.txt > outfile

tee and the process substitution will be executed even if find finds nothing, in which case a message is printed.

I changed the for loop into a while loop to let it manage file names with spaces. The input redirecton is changed accordingly, but this may not be what you want. Anyway, it is not relevant to your main question (how to add the "INVALID" notice).

  • Hey fra-san thanks for the suggestion. This is returning the correct number of records but it is not letting the wild card find the LATEST log file in those paths. it simply is appending "INVALID" to every line. – SSDdude Nov 29 '18 at 13:21
  • import_Asbeg_DA_Config3_*.log INVALID (and repeat ~750x. – SSDdude Nov 29 '18 at 13:23
  • This is the expected result if there is no matching file in the current directory and its sub-directories, so it's hard to tell what is going wrong. You could also try your original script, changing only the find line (the whole line). Making a small change at a time might help. – fra-san Nov 29 '18 at 13:36
  • Thx fra-san. it may be the 'expected result' but, it does not work as I need it to. I will mod my script as you suggested and keep trying. Thx! – SSDdude Nov 29 '18 at 13:54
  • Well, I meant that: 1) the loop I proposed should find the same files that your original script finds 2) if it yields no results, it probably means that it's not running from the proper path. Are you executing it in the same directory in which you used to execute your original script? Also, is that in your question just a snippet of a bigger script? If so, the rest of it might be influencing our loop (e.g. changing working directory, operating on file descriptors, ...). – fra-san Nov 29 '18 at 14:43
0

All thanks tremendously with the help!

"printf" is my buddy!

This code worked to solve my initial... outfile maintains the line integrity of ~750... outfile is a little messy but nothing that cannot be processed with some awk, sed, etc.

 for LATEST in `cat ${TMPDIR}/AllLogs.txt` ;
    do
      printf "`echo ${LATEST}` `find -name "${LATEST}" -type f | sort -n | tail -1`\n"

    done < infile > outfile

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.