Testing C++ code with bash commands

Question

I've read a programming problem and then I wrote code for it. But I thought my algorithm was not working right, so I wrote another piece of code that it was not optimal but, I think, correct.

Now I have about 100 input and output data sets and I want to give this input to the both C++ programs and compare outputs. If there are differences in the outputs, I will find out if my code is not correct.

My problem is that there are too many test cases (about 100) and the outputs are big text files and I can't check them by myself. I wonder how I can do this with bash functions and commands?

My input text files are input1.txt, input2.txt, etc. My outputs text files are similar to inputs. My programs are written in C++.

Answer 1

The best way to compare text files is with the diff command:

diff output1.txt output1.txt

For a mass comparison, you can call diff in a loop:

for x in input*.txt; do
  slow-program <"$x" >"$x.out-slow"
  fast-program <"$x" >"$x.out-fast"
  diff "$x.out-slow" "$x.out-fast"
done

If the snippet above produces any output, your fast program is buggy. In bash/ksh/zsh, you don't need to store the intermediate files on the disk. However this is not necessarily a good thing since it is likely to be useful to inspect the differing output at your leisure.

for x in input*.txt; do
  diff <(slow-program <"$x") <(fast-program <"$x")
done

It may be more convenient to put inputs and outputs in separate directories and perform a recursive diff.

for x in inputs/*; do slow-program <"$x" >"slow/${x##*/}"; done
for x in inputs/*; do fast-program <"$x" >"fast/${x##*/}"; done
diff -ru slow fast

My recommendation would be to write a makefile that runs the tests and performs the comparisons (in separate targets). (Use tabs where I put 8 spaces.)

all_test_inputs = $(wildcard input*.txt)  # relies on GNU make
%.out-slow: %.txt slow-program
        ./slow-program <$< >$@.tmp
        mv $@.tmp $@
%.out-fast: %.txt fast-program
        ./fast-program <$< >$@.tmp
        mv $@.tmp $@
%.diff: %.out-slow %.out-fast
        -diff $*.out-slow $*.out-fast >$@.tmp
        mv $@.tmp $@
# Test that all diff files are empty
test: $(all_test_inputs:%.txt=%.diff)
        for x in $^; do ! test -s "$x"; done
.PHONY: test

Run make test to process all the input files (only for the input files or the programs that have changed since the last time) and compare results. The command will be successful if and only if all the tests ran correctly and the output of both programs are identical in each case.

Answer 2

Assuming you compiled your programs as prog1 and prog2, and that they produce their output on stdout, you could do something like this:

#! /bin/bash

for input in input*.txt ; do
  ./prog1 $input > $input.out1
  ./prog2 $input > $input.out2
  if cmp $input.out1 $input.out2 > /dev/null ; then
     echo Programs disagreed on $input
  else
     echo Programs agreed on $input
  fi
done

This compares the output files byte for byte. You could also use diff for comparison.
The output of all the runs will be in files called inputX.txt.out1 or .out2, so you can inspect the cases where they do not match.