0

I am running an end-to-end test suite that's quite flakey, and I'd like a way to determine which tests are the flakiest. The easiest approach it seems would be to run tests 100s of times and return a percentage success value.

My problem of course is that I'm not well-versed in bash.

This is where I'm at:

numTotal=2
numError=0
x=1

function runTests() {
  local output=$(SPEC=desktop/homepage npm test)
  local exitCode=$?
  echo "$exitCode"
}

function runTestsMany() {
  while [ $x -le $numTotal ]
  do
    local exitCode=$(runTests)
    if [ 1 -eq "$exitCode" ]
    then
      numError=$(( $numError + 1 ))
      echo "Error"
    else
      echo "Success"
    fi
    x=$(( $x + 1 ))
  done
  numSuccess=$(($numTotal - $numError))
  echo "$numSuccess / $numTotal succeeded."
}

trap runTestsMany EXIT

It's close to what I want, but it forced me to silence all output from the tests, which makes post-run investigations into failures much clunkier.

How can I still output from the tests while capturing exit codes?

Thanks in advance!

Best Michael

Edit: Also, how can I pass in the script assigned to output? Whenever I try sticking it inside a variable, I get SPEC=desktop/homepage: No such file or directory, so I'm doing something wrong.

  • If you're looking for code review, there's a site for that: Code Review. That said, please do not edit the question to incorporate changes from the answer. That invalidates the answer, since changes suggested in the answer no longer make sense. – muru Mar 6 '18 at 2:02
  • In the future I'll make my edits strictly as additions. Unfortunately due to the limited nature of comments on stack, there isn't really a better way to share code changes. – M. Herold Mar 7 '18 at 4:00
  • It's not necessary to make all changes as additions, but here it's advisable as the question/answer won't make sense otherwise. In most other cases, for example in response to comments, you can directly edit the parts. – muru Mar 7 '18 at 4:06
1

tldr; 1) output redirection; 2) don't trap.

Full answer:

  1. In your function runTests, you can redirect output to some single or multiple log files. Instead of creating a local variable output and never using it, just run SPEC=desktop/homepage npm test > uniqe_log_$x or SPEC=desktop/homepage npm test > combined_log. For this, I sneakily borrowed the variable x that you declared at the top of your script and increment every test.

  2. Also in your function runTests, there is no need to define a variable exitCode or perform an echo; simply perform return $?.

  3. I call BS on your claim that your script is close to what you want, because in your function runTestsMany, you call function runTests outside of your while, so variable exitcode will always be the same for all iterations 1..numtotal.

  4. You presume that exitcode can only be either 1 or 0, so there's no need to perform any testing of it, just add it to yor variable numerror, ie. numError=$(( $numError + $exitcode )).

  5. I don't see the point of performing the individual echos for success or error, since it doesn't add useful information, and will only clutter your output.

  6. The final variable declared, numsuccess is also unnecessary; just put $(($numTotal - $numError)) in the subsequent echo statement.

  7. What's with the trap statement? Lose it / remove it.

  • I can't remove SPEC, it's integral to how the script is called. Is there no hold an environment-variable assignment shell command in a variable? – M. Herold Mar 6 '18 at 0:07
  • My bad. updated. – user1404316 Mar 6 '18 at 0:11
  • Thanks for your help, redirecting the output makes checking logs really slick. The only remaining issue I have is that I'd like to pass "SPEC=desktop/homepage npm test" into the script, or at least store it inside a single variable so I can echo the commmand being stress tested. – M. Herold Mar 6 '18 at 0:48
  • Regarding other points: 1) Thanks, this is perfect. 2) I tried doing that, but I was getting int/string errors, couldn't figure them out. 3) Good catch! 4) Cleaner, I like it. 5) It's a progress-indicator, allows me to break early if I'd like (although I suppose I could monitor filenames now). 6) I don't hate an intermediate variable. 7) Don't know, copying other people's code as I go, never written shell scripts until today. – M. Herold Mar 6 '18 at 0:52
  • @M.Herold - passing the variable should become a separate question. I think we've already gone well beyond the scope of the original question. Vote on this one, accept the answer, close it, and open another one, but before you open another one, do make an attempt to solve it yourself or search online for the answer - it's not difficult. Good luck with it! – user1404316 Mar 6 '18 at 1:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.