2

I have:

  • A checksum.txt file which contains many lines of checksums of single files from a mountpoint in a huge directory, which mounted and then it disconnected, thereby not finishing the checksum.txt (partial checksums)
  • localchecksums.txt full checksum list, containing thousands of lines of SHA256 checksums with filenames etc.

I would like to:

  • Compare the remote mount checksums and local ones with sha256sum -c checksum.txt localchecksum.txt or similar, but:
  1. I don't want to go through gigabytes of data again to get the remaining hashes
  2. I don't want to restart the whole process for checksum.txt

I generated the list by using find to recursively find single files and exec sha256sum on them.

It is possible to get the remaining hashes by comparing the two files or somehow continue checking the checksums by reading the checksum.txt file and only calculating checksums for unchecked files. The problem with the first approach is that the order is different in the files. The second approach sounds good but I don't have any idea how to start with that.

Sample of any of the checksum files:

8e2931cc1ad3adc07df115456b36b0dbd6f80f675e0a9813e20ad732ae5d4515  ./folder/8ggSHp5I7hNEl3vDCbWv6Q/wA-KzXIh1Ce3G93s20X24v_4vUeywBe3mXPhGjPt_Lg/cRf8KgbqIsqwbon3DX3PN1-oV6_Nr9Baeymaw-ZJw00
37d2dfe2315cc401536329e3fbe421384bbb50c656c3dbeb42798e5666822e6c  ./folder/8ggSHp7I7hHEl3vDCbWv6Q/wA-KzXIh1Ce3G93s2oX24v_4vUeywBe3mXPhGjPt_Lg/V02s6HKhyJ9Nyd2jQtSjWg
d0e9b95065a264db0d372ccace5d3a72f38f74ca7b44da4794dae23c91e18e57  ./folder/8ggSHp7I7hNxl3vDCbWv6Q/wA-KzXIh1Ce3G93s2oX24v_4vUeywBe3mXPhGjPt_Lg/U3fhBugX6pexYzh6qGKlW7lYWsFShWH7JwN9fmU8ay2lLZkciH2sXsiGbmIc97iJ
44a5fe29063e472857bb9a1929af06a32bb4b2394630f80c2dc732fd662620bc  ./folder/8ggSHp7I7hNEc3vDCbWv6Q/wA-KzXIh1Ce3G93s2oX24v_4vUeywBe3mXPhGjPt_Lg/gTrqUL4ZjWTWMl6BcjfwUe5bBDatscwUoYY9IFQDztc

2
  • Just use mtree. It's exactly for this kind of thing.
    – Jim L.
    Commented Apr 16 at 19:04
  • @JimL. how fast is it? would it need to download all the files for local computing just like sha256sum needs? (not that efficient) My goal is compare whether all the files and folders are identical (I omitted the folder requirement due to it being not that important in the question) Commented Apr 16 at 19:10

1 Answer 1

2

In principle,

  1. sort the list of file names already with a checksum
  2. find and sort a full list of file names
  3. use comm to obtain the set of files that aren't yet processed (files in one set that aren't in the other)
  4. generate sha256sum checksums for files in the resulting list

Demonstration (bash not sh):

based=/etc                      # Directory base
csums=/tmp/etc_checksums.txt    # Checksums file

# EXAMPLE ONLY
# TAKE CARE NOT TO OVERWRITE YOUR REAL DATAFILE WHILE TESTING !!
# Generate a partial set of 30 checksums
find "$based" -type f |
    head -n30 |
    while IFS= read -r f; do sha256sum "$f"; done >"$csums"


# USEFUL WORK
# Identify the set of differences
LC_ALL=C comm -13 <(sed 's/^[^ ]*  //' "$csums" | LC_ALL=C sort) <(find "$based" -type f | LC_ALL=C sort) |
    # For each missing item
    while IFS= read -r f
    do
        # Generate a new checksum and append it to the checksums file
        printf '%s' "$f" >&2
        sha256sum -- "$f" >>"$csums" && echo >&2
    done

This code will work for file names containing whitespace, but will fail on pathological file names containing newlines and other non-printable characters. (No doubt others can suggest how to bullet-proof the code if that's really required.)

Test carefully before modifying your original checksums file. Ensure that $based matches your original starting point so that the comm can correctly filter out duplicates.

12
  • So basically find fetches the folder and file paths for the checksums or how does it work? I'm guessing I don't need the # EXAMPLE ONLY part because I already have a partial checksum file, is that correct? Commented Apr 18 at 17:21
  • 1
    You can use the "example only" section as a try out. It generates checksums for only 30 of the files under /etc, so that the remainder of the code can generate the rest. Just ensure you're not overwriting your real checksums file, @SirMuffington Commented Apr 18 at 17:31
  • Thanks, I'll look into it soon ;-) Commented Apr 19 at 15:01
  • is it possible to restrict the amount of connections made by find to certain number? It seems to error out for me... Commented Apr 19 at 18:54
  • What's your exact find command and what is the error message? Commented Apr 19 at 22:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .