git has bisect run to figure out which revision an error was introduced.

I have a big file (100GB) and at least one of the lines is bad, but the program I have to check it with will not tell me which line.

The lines are records, so I can write a binary search using head and tail and /2 (passing the first half to the program, and if no errors, the second half), and based on that splitting the one half again.

But does an automated tool already exist that can do this without my intervention (similar to git bisect run)?

  • What kind of file? Source code? Plain text?
    – kurtm
    Oct 16 '13 at 13:50
  • Plain text: Each line is a full record.
    – Ole Tange
    Oct 16 '13 at 14:27
  • I suggest looking into git-bisect.sh from git repository instead and modifying the code to suit your need.
    – Ivan Chau
    Oct 17 '13 at 2:18

you can use split to split a file into multiple parts, and it has an option to only split at lines:

$ ls
$ split -n l/2 bigfile
$ ls
bigfile xaa xab

this really only makes sense if the file can be split and is organized in lines, which is only true for textfiles.

with this you can easily build your own bisect-tool, e.g. something like the following:



usage() {
    echo "usage: $0 <testprog> <datafile>"
    echo "     will bisect <datafile> to the single line where <testprog> exits with '0'"
    exit 1

if [ ! -x "${TESTPROG}" ]; then  usage; fi
if [ ! -e "${DATA}" ]    ; then  usage; fi

BISECTDIR=$(mktemp -d)

splitfiles() {
    split -e -n l/2 $1 ${BISECTDIR}/$2bisect_
    echo ${BISECTDIR}/$2bisect_*
cleanup() {
    rm -rf "${BISECTDIR}"
    exit 0

while [ $(head -2 "${DATA}" | wc -l) -gt 1 ]; do
  echo "testing: ${DATA} $(head -2 "${DATA}" | wc -l)" 1>&2
  files=$(splitfiles ${DATA} ${i})
  count=$(echo $files | awk '{print NF}')
  if [  ${count} -lt 2 ]; then
      cat $files
  for f in $files; do
          if ${TESTPROG} "${f}" 1>/dev/null 2>/dev/null; then
  i=$(( i+1 ))


caveat: this will put all the bisected data into /tmp (change the definition of BISECTDIR is you don't like that); also this will only cleanup the bisected data files at the end. so you might need plenty of space...

  • What I am looking for is a bigger tool - similar to 'git bisect run' that will do the splitting and testing without my intervention.
    – Ole Tange
    Oct 16 '13 at 13:14
  • @OleTange i've updated my answer to include such a tool written in bash.
    – umläute
    Oct 17 '13 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.