I'm currently running cfv -C -rr to generate a SFV file that contains the CRC32 checksums of all files in all subdirectories. The issue is that because there are so many hardlinks, this process is taking much longer than it should take. Is there any way to get cfv to ignore hardlinks?

cfv can be found here. The man page doesn't mention hardlinks. Are there any alternative applications?

  • Do you mean you want cfv to ignore files that are multiply linked (which I think it what you've actually asked), or that you want cfv to ignore the second and subsequent times it finds a file? – roaima May 17 '15 at 22:15
  • I want cfv to ignore the second and subsequent times it finds a file so that there is no wasted time. I want all content checksums to only be calculated once. – user115746 May 17 '15 at 22:16

I assume that file names don't contain newlines or initial or trailing whitespace (the SFV format doesn't support those anyway) and that you have GNU utilities (non-embedded Linux, Cygwin).

First generate a mapping from file names to inode numbers, and sort it by inode numbers.

tmpdir=$(mktemp -d)
find .* * -xdev -name .. -prune -o -type f -printf '%i %p\n' |
sort -k 1n > "$tmpdir/inodes.txt"

Generate a checksum file for the first file with each inode number.

<"$tmpdir/inodes.txt" awk '$1 != previous {previous = $1; sub(/^[0-9]+ /,""); print}' |
xargs cfv -C - >"$tmpdir/1.cfv"

Now complete the checksum file taking advantage of the hard links. I don't know if cfv prints the files in the order in which they are passed on the command line, if it does you can save memory at this step because the temporary CFV file and the inode file will be in ordered in the same way.

awk '
    FNR==1 {
        match($0, / +[[:alnum:]] *$/);
        cksums[substr($0, 1, RSTART-1)] = substr($0, RSTART);
    { inode = $1; sub(/^[0-9]+ /, ""); }
    previous != inode { cksum = cksums[$0] }
    { print $0, cksum }
' "$tmpdir/1.cfv" "$tmpdir/inodes.txt" >full.cfv

You can then remove the temporary files.

Warning: untested code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.