Why is the time to complete cmp command varying so wildly?

Question

I am using the cmp command to compare a 1GB files stored on an SD card to a reference 1GB file stored in main memory. The completion times for a single cmp command vary significantly, ranging from 17 seconds to 3.5 minutes.

The files are expected to be the same, and in all cases so far have been. I run a function (see below) that compares the 100 1GB files on the SD card to the reference 1GB file stored in main memory. Usually, all 100 cmp's in the loop will tend either fast (<20 sec) or slow for the duration of the script.

Based on the output of top, I have not observed any processes that would be causing the extended duration when it takes longer.

What could be causing the completion time of the cmp command to vary?

Also, what can be done to make sure the command is completed in a reasonable amount of time (<25 seconds)?

This is occurring on a Yocto distribution in an embedded application.

function check_files() {
    for filename in /mnt/Android/data/File_*;
    do
        echo "Checking $filename"
        result=$(cmp -l /data/1GB_File.bin $filename)
        resultlength=${#result}
        if [ $resultlength -gt 0 ]; then
            date >> /data/errors.txt
            echo $filename >> /data/errors.txt
            echo $result >> /data/errors.txt
            echo "==========" >> /data/errors.txt
        fi
    done
}

Answer 1

I suggest you redesign your method. Your method reads /data/1GB_File.bin over and over, once for each file in /mnt/Android/data/File_*.

While "disk caching" usually helps speed up disk I/O, your 1GB file size, and the fact that, the 2nd through Nth times through your loop, you are interleaving requests for cached data (/data/1GB_File.bin) and new (to be cached) data. But, since data (disk-block sized chunks of memory) is removed from cache through an "least recently used" ("oldest first") algorithm, it's a race between the new data forcing the cached data out, and the old cached data being read (changing its position on the LRU list). Additionally, normal system activity uses the disk cache, too.

Unless your disk cache is larger than "normal system usage" PLUS 2 x 1GB, you'll always have the race, and the resulting variability in timings.

Compute a checksum of each of the files and the standard. Read each file just once. Do your comparisons with the checksums.

Read man md5sum, do domething like UNTESTED:

check_files() {
    md5sum /mnt/Android/data/File_* >data.tmp
    md5dum /data/1GB_File.bin >standard.tmp
    #
    # extract the "correct" checksum 
    golden="$(cut -d" "   f1 standard.tmp)"
    #
    # do any of the suspect files not 
    # have the golden checksum?
    grep -v "$golden" data.tmp >bad.tmp
    if [[ $? -eq 0 ]]; then
        (date;cat bad.tmp;echo "==========" )>> /data/errors.txt
    fi
    #uncomment the `rm` line when you're sure it works
    # can test with adding any other filename
    # to the first `md5sum` line.
    #rm -f standard.tmp data.tmp bad.tmp 2>/dev/null
    # why not return a status from this function?
}