1

I need to parse a log file and tell the test name if its execution time taken is more than a specific period (user will input this time, lets say script executed for more than 30mins)

start security test suite
Mon Sep 05 00:16:30 PDT 2011: creating threads...
Mon Sep 05 00:16:30 PDT 2011: starting...
Mon Sep 05 00:16:31 PDT 2011: Reporting results...
Mon Sep 05 00:16:31 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/add_role_user_security.xml
add_role_user_security.xml : Passed
Mon Sep 05 00:16:31 PDT 2011: creating threads...
Mon Sep 05 00:16:31 PDT 2011: starting...
Mon Sep 05 00:16:32 PDT 2011: Reporting results...
Mon Sep 05 00:16:32 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/privilege.xml
privilege.xml : Passed
Mon Sep 05 00:16:32 PDT 2011: creating threads...
Mon Sep 05 00:16:32 PDT 2011: starting...
Mon Sep 05 00:16:32 PDT 2011: Reporting results...
Mon Sep 05 00:16:32 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/edit_role_user.xml
edit_role_user.xml : Passed
Mon Sep 05 00:16:32 PDT 2011: creating threads...
Mon Sep 05 00:16:32 PDT 2011: starting...
Mon Sep 05 00:16:33 PDT 2011: Reporting results...
Mon Sep 05 00:16:33 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/remove_roles.xml
remove_roles.xml : Passed
Mon Sep 05 00:16:33 PDT 2011: creating threads...
Mon Sep 05 00:16:33 PDT 2011: starting...
Mon Sep 05 00:16:33 PDT 2011: Reporting results...
Mon Sep 05 00:16:33 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/role_user1.xml
role_user1.xml : Passed
Mon Sep 05 00:16:33 PDT 2011: creating threads...
Mon Sep 05 00:16:33 PDT 2011: starting...
Mon Sep 05 00:16:34 PDT 2011: Reporting results...
Mon Sep 05 00:16:34 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/role_user2.xml
bug10611.xml : Passed
security test suite
start fo test suite
Mon Sep 05 00:18:52 PDT 2011: creating threads...
Mon Sep 05 00:18:52 PDT 2011: starting...
Mon Sep 05 00:18:52 PDT 2011: Reporting results...
Mon Sep 05 00:18:52 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/setup_script.xml
setup_script.xml : Passed
Mon Sep 05 00:18:52 PDT 2011: creating threads...
Mon Sep 05 00:18:52 PDT 2011: starting...
Mon Sep 05 00:18:58 PDT 2011: Reporting results...
Mon Sep 05 00:18:58 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/bug_scripts.xml
bug_scripts.xml : Passed
Mon Sep 05 00:18:58 PDT 2011: creating threads...
Mon Sep 05 00:18:58 PDT 2011: starting...
Mon Sep 05 00:18:58 PDT 2011: Reporting results...
Mon Sep 05 00:18:58 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/loadtime_namespace.xml
[Fatal Error] loadtime_namespace.xml:11:25: The processing instruction target matching "[xX][mM][lL]" is not allowed.
Unable to parse loadtime_namespace.xml: Running diff utility to compare...
rm -f ./scripts/diffs/loadtime_namespace.xml
loadtime_namespace.xml : Passed
Mon Sep 05 00:19:01 PDT 2011: creating threads...
Mon Sep 05 00:19:01 PDT 2011: starting...
Mon Sep 05 00:19:01 PDT 2011: Reporting results...
Mon Sep 05 00:19:01 PDT 2011: Writing results to /space/builder/builds/macosx-64/HEAD/qa/scripts/results/string_script.xml
string_script.xml : Passed
fo test suite

User will be executing the script by passing the execution time taken by script

sh script_name.sh 60

And it should have the list of tests that are taking more than 60 mins to execute

The output should be like:

security test suite   add_role_user_security.xml
security test suite   privilege.xml
security test suite   remove_roles.xml
security test suite   string_script.xml
fo test suite         string_script.xml
fo test suite         setup_script.xml
  • 3
    Please edit your question to make it more readable... Add code wrappers where needed, and make it more concise. – Valentin B. Oct 27 '16 at 8:24
0

The code at the bottom of this response does what you asked, and works based on the sample you provided.

Description:

It basically works in two stages:

  • awk uses regular expressions and state variables to determine transition between files and test suite blocks. It outputs a pipe delimited list where each row has the test suite name, test file, start and end time.
  • while IFS="|" read ... part takes the pipe delimited output from awk reads it into some variables where I use date to convert timestamps into seconds since epoch and then calculate the difference and compare it the input value.

The code does not do much in the way of validation so you will have to add this yourself.

Code:

#!/usr/bin/sh

# Get the arguments
INPUT_FILENAME="$1"
FILTER_MINUTES="$2"

# Convert minutes to seconds
FILTER_SECONDS="$(( FILTER_MINUTES * 60 ))"

awk '
BEGIN {
    printf("%s,%s,%s,%s\n","TEST_NAME","FILENAME","START_TIME","END_TIME");
    TEST_NAME="";
    START_TIME="";
    END_TIME="";
  }
match($0, "^start (.+)$", arr) {
    TEST_NAME=arr[1];
    START_TIME="";
    END_TIME="";
    FILENAME="";
    /* printf("TEST_NAME START: %s\n", TEST_NAME); */
  }
match($0, /^(.+.xml) : (.+)$/, arr)  {
    FILENAME=sprintf("%s",arr[1]);
    printf("%s|%s|%s|%s\n",TEST_NAME,FILENAME,START_TIME,END_TIME);
    START_TIME="";
    END_TIME="";
    FILENAME="";
  }
match($0, /^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) ([0-9]{2}) ([0-9]{2}):([0-9]{2}):([0-9]{2}) ([^\ ]+) ([0-9]{4}):.*$/, arr) {
    if(START_TIME ~ /^$/)
       START_TIME=sprintf("%s-%s-%s %s:%s:%s %s",arr[3],arr[2],arr[8],arr[4],arr[5],arr[6],arr[7]);
    else
       END_TIME=sprintf("%s-%s-%s %s:%s:%s %s",arr[3],arr[2],arr[8],arr[4],arr[5],arr[6],arr[7]);
  }
$0 ~ "^" TEST_NAME "$" {
    /* printf("TEST_NAME END: %s\n", TEST_NAME); */
    TEST_NAME="";
    START_TIME="";
    END_TIME="";
    FILENAME="";
  }
' "$INPUT_FILENAME" | while IFS="|" read TEST_NAME FILENAME START_TIME END_TIME
do
  START_TIME_SEC=$(date -d "$START_TIME" +%s)
  END_TIME_SEC=$(date -d "$END_TIME" +%s)
  ELAPSED_SEC=$((END_TIME_SEC-START_TIME_SEC))

  # No time supplied or elapsed time exceded.
  if [ -z "$FILTER_MINUTES" -o "$ELAPSED_SEC" -gt "$FILTER_SECONDS" ]
  then
    ## Remove $ELAPSED_SEC from here to not print elapsed seconds.
    echo $TEST_NAME $FILENAME $ELAPSED_SEC
  fi
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.