0

Basically I'm looking at a log entry like this:

*** Send Command has completed Successfully 
Transmitted 508200 bytes in 1 seconds
File Transfer Complete
RemoteTransactionNumber is ***********
************Server: Transfer Mode Set To Send
: *********************************************************50 al**********b:Y 
lf:"*********************************************************" rf:"*************
*(+1)" **************************************41
2017/04/18 13:05:32 About to execute the following Send command:

I've defined a variable $LOGGER and stored the logged data in it because it's on a remote server. I've also created a variable called $currentdate and I'm storing:

currentdate=$(date +"%Y/%m/%d/ %I")

So my current time while posing this Q looks like: 2017/04/19/ 01

So let's say I want to match on "Send Command has completed Successfully" I've done the following:

ALERT=$(echo "$LOGGER" | awk "/Send Command has completed Successfully/" | grep -A 12 $currentdate)

I then am trying to grep ahead 12 lines looking for the current date & time up to the hour to make sure the send i'm matching is current because that's all I care about when this script runs.

But that last awk | grep -A 12 command isn't working... any ideas how I can look for the date and verify it matches $currentdate after hitting an awk match? This is all being done on local variables after retrieving a log scrape from an SSH pass command FYI so no actual files are being manipulated at this point.

Edit: so to give a clearer picture as too why i'm doing what i'm trying to do here is any match I get for the current hour I need to report on. And the date is within 12 lines of the "Send Successful" entry. I'm actually using SSH pass to remote into servers, grab the number of log files written to in the last hour. Then I'm going in and grabbing the last 100 lines of each and storing it in the $Logger variable like this:

COMMAND1="find /xxy -mmin -60 | wc -l"
FILENUM=`/xxy-1.05/sshpass  -p$PASS ssh -q -o StrictHostKeyChecking=no -o 
ConnectTimeout=310 $USER@$HOST "$COMMAND1"

COMMAND2="find /xxy -mmin -60 | tail -n $FILENUM | xargs tail -n100"
LOGGER=`/xxy-1.05/sshpass  -p$PASS ssh -q -o StrictHostKeyChecking=no -o 
ConnectTimeout=310 $USER@$HOST "$COMMAND2"

So given that I need to review every matches date I wasn't sure the best way to go about it and store it clean in a variable (which awk tends to do and why it was my first choice)

closed as unclear what you're asking by G-Man, phk, Archemar, Anthon, user34720 Jun 6 '17 at 18:15

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • I'm confused: why 12 lines? why don't you jsut parse the latest entry (ie, the current one) and then search in that if you find "has completed succesfully", instead of awk-ing all "completely succesfully" lines and looknig at the 12th of those?? (which is what your "awk only those linee | grep -A 12 currentdate" does: grep acts on awk's output, which is only the lines with "completed successfully" (which have no dates whatsoever) – Olivier Dulac Apr 19 '17 at 17:49
  • I'm actually grabbing the last 100 lines of multiple logs that were written to in the last hour (but they could be months old giving me older entries too) into one variable so they won't be in order by date after the first log file is written and the new one begins if you get me. It'll be all mixed in... This is because the app is using 3rd party software that starts a new log at every process restart and names the log file down to the second making regex to monitor the log statically err not feesable... forcing me to do this. But your Awk comment makes sense... I just wasn't seeing that. – JHarder51 Apr 19 '17 at 18:11
  • I corrected my answer to know look for the currentdate among any pair of "date/successfully just after it", and say if there was a match or not – Olivier Dulac Apr 19 '17 at 18:21
  • @JHarder51: Your FILENUM= and LOGGER= commands have unmatched backticks (`).  Please fix them, not by adding more backticks, but by switching to the $(…) notation.  And when you have a command that spans two lines, please indicate it clearly — e.g., by appending a backslash (\) to the first line and/or indenting the second line, or simply by putting it all on one line (it will require horizontal scrolling, as your ALERT= command does). – G-Man Apr 19 '17 at 19:05
1

When something doesn’t work, you should try pulling it apart and seeing what the pieces do, so you can figure out where it fails.  Try just this command:

echo "$LOGGER" | awk "/Send Command has completed Successfully/"

What do you get?  Just the *** Send Command has completed Successfully line.  So of course grepping that for the date doesn’t work; that line does not contain a date.

If you want an awk solution, try something like

echo "$LOGGER" | awk -vc="$currentdate" '
    /Send Command has completed Successfully/ { flag1=1 }
    $0 ~ c { flag2=1 }
END { if (flag1 && flag2) print "Yes"; else print "No" }'

which passes the shell variable currentdate into awk as awk variable c and then checks whether it and the “Send Command …” string are both present in the input.

And by the way,

  • You should always quote your shell variable references (e.g., "$currentdate") unless you have a good reason not to, and you’re sure you know what you’re doing.  Your grep … $currentdate command could never have worked without quotes as long as "$currentdate" has a space in it.
  • If you want to verify that the log entry is current up to the hour, you should not use %I in your date command — it returns the hour in the range 01..12.  So,

    • If the log entry is from 1AM (and therefore says “2017/04/18 01:##:## About to execute the following Send command:…”) and the current time is 1PM, your $currentdate variable will be 2017/04/18 01 and so it will say that the 12-hour-old message is current, but
    • if, as in your example, the log entry is from 1PM (and therefore says “2017/04/18 13:05:32 About to execute the following Send command:…”) and the current time is 1PM, your $currentdate variable will (again) be 2017/04/18 01 and so it will say that the current message is not current.

    You should use %H, which ranges (00..23).

  • I see that you’ve been changing the question while I’ve been typing, but it’s not really any clearer to me than it was before.  If my answer works for you, great.  (If you believe that it is the better/best one, please click on the checkmark to the left.)  If it doesn’t work for you, please explain why.  Please do not respond in comments; edit your question to make it clearer and more complete.  (For example, if $LOGGER might contain multiple log entries, say so, give an example, and describe what you want to happen in that case.) – G-Man Apr 19 '17 at 18:38
  • Okay, well and that's why I tried to just focus on my need to hit a pattern, and then verify another pattern (the date) is there within 12 lines. The server itself using that time standard and as you can see i'm determining what's current to the hour prior and scraping it. I'm trying to follow your awk through now and appreciate the feedback. – JHarder51 Apr 19 '17 at 19:13
  • If you just flag if you find both things (a Successfully and the current date) you will say "ok" even if the successfully was for a very different date and the currentdate have not resulted in a "successfully" result. This approach does not work (in the end just it should say" we saw both things in the whole output of LOGGER" , which is what it does and not what @JHarder51 needs). – Olivier Dulac Apr 20 '17 at 7:53
  • That thought crossed my mind, and I’m still waiting for the OP to clarify the question to confirm that that is the case. – G-Man Apr 20 '17 at 19:21
0

Edit: it seems you are not sure of the order of dates in LOGGER, so I changed my entry to match the date "anywhere"

Your exemple doesn't show several entries, so I assume they are:

2017/04/....
(several lines)
*** Send Command has completed Successfully
2017/04/.... (another time, maybe before or after, no ordering )
(several lines)
2017/04/.... (another time, maybe before or after, no ordering )
(several lines)
2017/04/.... (another time, maybe before or after, no ordering )
(several lines)
2017/04/.... (another time, maybe before or after, no ordering )
(several lines)
*** Send Command has completed Successfully
2017/04/.... (another time, maybe before or after, no ordering )
(several lines)
2017/04/.... (another time, maybe before or after, no ordering )

I believe you just need:

dateappearsas="^[0-9][0-9][0-9][0-9]/[0-9][0-9]/"
looking_for="Send Command has completed Successfully"

echo "$LOGGER" \
| egrep -i "${dateappearsas}|${looking_for}" \
|  grep -i -B 1 "${looking_for}" \
| grep -A 1 "$currentdate"
returncode="$?" # in bash, gets the latest command in the preceding pipe, ie the grep's exit code
# the egrep: outputs all dates, and interspered among those some "successfully"
# the 2nd grep: transform those into pairs of date & successfully
# the last grep find only the matching currentdate and the line after it  
if [ "$returncode" = "0" ]
then
   echo "There is a transfer that matches the current date, AND was successful"
else
   echo "There is either no date matching the current date, or it wasn't successfull"
fi

Basically: egrep to get both what you are looking for AND the context (here: date). It gives maybe several lines of context, and a few among them have also the thing you are looking for. Then the grep -B 1 'what your are looking for' will regroup those: context/lookedfor pairs together (putting together the lookedfor lines and the date just prior it), and the last grep looks if the currentdate is among those pairs

  • So the lastest pair is not actually the most important. I didn't want to go too far down the rabit hole but probably see now I should have haha. It's a rigamaroo! I'm actually grabbing the last X log files from remote servers that were written too in the past hour. So I could have 4 files and their last 100 lines stored in 1 variable given that I want any Send successful that matches the hour in that i'm running the script just getting the last entry won't do. Your input was great though thank you. I'll update the post to try and explain more. – JHarder51 Apr 19 '17 at 18:21
  • I saw that, and already corrected it to look for the pair anywhere (as you gather several different logs, maybe not in the right order) – Olivier Dulac Apr 19 '17 at 18:22
  • basically: I construct pairs of all "success and the date JUST BEFORE IT (ie the date that transfer was launched at)", then among those i look for the current date. To ensure this works, instead of grabbing 100 lines per log filse, you should grab only a few pairs of : for i in *LOGS*; do egrep -i "${dateappearsas}|${looking_for}" "$i" | grep -i -B 1 "${looking_for}" | tail -n 4 ; done > all_last_2_pairs_from_each_logfiles and then just grep "${currentdate}" all_last_2_pairs_from_each_logfiles. otherwise, here, sometimes a success could have before it another date from another logfile? – Olivier Dulac Apr 19 '17 at 18:26
  • I'm using success just for testing because it's most common entry and I see them every hour live. There will be 8 different error's i'm looking for technically so I'll have to translate it appropriately. maybe I can do that with an error array of sorts on $lookingfor I'm going to try and digest your latest revision and appreciate the input thus far. – JHarder51 Apr 19 '17 at 18:34
  • dateappearsas is only looking at the year and month looks like right? – JHarder51 Apr 19 '17 at 18:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.