0

I am using the cmp command to compare a 1GB files stored on an SD card to a reference 1GB file stored in main memory. The completion times for a single cmp command vary significantly, ranging from 17 seconds to 3.5 minutes.

The files are expected to be the same, and in all cases so far have been. I run a function (see below) that compares the 100 1GB files on the SD card to the reference 1GB file stored in main memory. Usually, all 100 cmp's in the loop will tend either fast (<20 sec) or slow for the duration of the script.

Based on the output of top, I have not observed any processes that would be causing the extended duration when it takes longer.

What could be causing the completion time of the cmp command to vary?

Also, what can be done to make sure the command is completed in a reasonable amount of time (<25 seconds)?

This is occurring on a Yocto distribution in an embedded application.

function check_files() {
    for filename in /mnt/Android/data/File_*;
    do
        echo "Checking $filename"
        result=$(cmp -l /data/1GB_File.bin $filename)
        resultlength=${#result}
        if [ $resultlength -gt 0 ]; then
            date >> /data/errors.txt
            echo $filename >> /data/errors.txt
            echo $result >> /data/errors.txt
            echo "==========" >> /data/errors.txt
        fi
    done
}
7
  • 1
    Does it reliably take 3.5 minutes when you drop caches before: echo 3 > /proc/sys/vm/drop_caches?
    – A.B
    Commented Jun 5, 2023 at 20:46
  • I will give it a shot thank you.
    – mamawmaw
    Commented Jun 5, 2023 at 21:28
  • 2
    Show your complete command line, and the exit status, please. My cmp (GNU diffutils) 3.6 exits on the first difference unless you use the -l option (a feature which is not mentioned in the man page, but is in info cmp !). So 17 seconds means the first difference happened about 8% into the data. 3.5 minutes happens when there is no difference, and that is limited by the SD data rate. Commented Jun 6, 2023 at 8:41
  • 2
    Optimum strategy depends on why the files might be different. cmp is just a linear check. If the files are text, you might first check if they are different lengths -- stat %s on each, and compare. If different, no need to read any data. Then maybe check the ends if that is the most likely place for differences -- tail -c 1024 on each file and cmp those fragments. Ultimately, though, if the difference can be a single bit anywhere in the file, you have to read the whole of each file. Commented Jun 6, 2023 at 9:00
  • The files are expected to be the same, and in all cases so far have been, and I am just validating that they are. I run a shell script that compares 100 1GB files all stored on the SD card to a single reference 1GB file stored on main memory. Usually, all 100 cmp's in the loop will tend either fast (<20 sec) or slow for the duration of the script. I will add this to the original question. I haven't had a second to check if echo 3 > /proc/sys/vm/drop_caches fixes the issue. I will update.
    – mamawmaw
    Commented Jun 6, 2023 at 15:11

1 Answer 1

3

I suggest you redesign your method. Your method reads /data/1GB_File.bin over and over, once for each file in /mnt/Android/data/File_*.

While "disk caching" usually helps speed up disk I/O, your 1GB file size, and the fact that, the 2nd through Nth times through your loop, you are interleaving requests for cached data (/data/1GB_File.bin) and new (to be cached) data. But, since data (disk-block sized chunks of memory) is removed from cache through an "least recently used" ("oldest first") algorithm, it's a race between the new data forcing the cached data out, and the old cached data being read (changing its position on the LRU list). Additionally, normal system activity uses the disk cache, too.

Unless your disk cache is larger than "normal system usage" PLUS 2 x 1GB, you'll always have the race, and the resulting variability in timings.

Compute a checksum of each of the files and the standard. Read each file just once. Do your comparisons with the checksums.

Read man md5sum, do domething like UNTESTED:

check_files() {
    md5sum /mnt/Android/data/File_* >data.tmp
    md5dum /data/1GB_File.bin >standard.tmp
    #
    # extract the "correct" checksum 
    golden="$(cut -d" "   f1 standard.tmp)"
    #
    # do any of the suspect files not 
    # have the golden checksum?
    grep -v "$golden" data.tmp >bad.tmp
    if [[ $? -eq 0 ]]; then
        (date;cat bad.tmp;echo "==========" )>> /data/errors.txt
    fi
    #uncomment the `rm` line when you're sure it works
    # can test with adding any other filename
    # to the first `md5sum` line.
    #rm -f standard.tmp data.tmp bad.tmp 2>/dev/null
    # why not return a status from this function?
}
  
 
2
  • This is a good idea thank you. But I'm really just curious about why cmp's completion time varies so much.
    – mamawmaw
    Commented Jun 7, 2023 at 15:48
  • @mamawmaw Answer updated.
    – waltinator
    Commented Jun 7, 2023 at 17:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .