See if any of a number of zip files contains any of the original files in a directory structure

Question

I have a pretty hard problem here.

I have a photo library with a lot of photos in it in various folders.

I then started using Google Photos for my photos, I put those originals into Google Photos, and used it for 5+ years.

Now I want to move away from Google Photos. I have done a Google Takeout of all my photos, and downloaded all the Zip files, ~1.5TB worth of them (150 x ~10GB files).

Now I want to keep my original directory structure, and delete all the files that are duplicated in Google Photos. After this operation, I basically want to have two directories left over each with unique files in them. I can then merge this by hand later.

I have started extracting all the files and then I will run rmlint to detect duplicates and purge from Google Drive. The problem is I don't have enough space to maneuvre all this around, so I have to extract say 30 archives, then run rmlint, purge, extract another 30, run rmlint again, purge, etc. This rescans my original files over and over, and it's going to take a really long time to do. I already use the --xattr flag for rmlint to try and speed up subsequent runs. See appendix for full rmlint command.

How can I do this WITHOUT having to first extract all the archives? Is there a way to just use the file checksums in the zip files and compare to those?

Thanks!

Appendix

rmlint \
        --xattr \
        -o sh:rmlint-photos.sh \
        -o json:rmlint-photos.json \
        --progress \
        --match-basename \
        --keep-all-tagged \
        --must-match-tagged \
        "/mnt/f/GoogleTakeout/" \
        // \
        "/mnt/e/My Documents/Pictures/" \

Answer 1

In zsh or bash and with libarchive's bsdtar and GNU tar, you could do something like:

LC_ALL=C find . -name '*.zip' -type f -print0 |
  while IFS= read -rd '' archive; do
    bsdtar -cf - "@$archive" |
      ARCHIVE=$archive tar -xf - --to-command='
        case $TAR_FILETYPE$TAR_FILENAME in
          (f*.jpg | f*.JPG)
            sha1sum | {
              IFS= read -r sum rest &&
                printf "%s\n" "$sum:$ARCHIVE:$TAR_FILENAME"
            }
        esac' > sums.txt

(assuming none of the archive paths nor members contain newline nor : characters) to get a list of <checksum>:<archive>:<file-in-archive> for each of the archives (will take a while as it runs one sha1sum per image).

Then, you could extract only one file for each unique checksum with something like:

perl -F: -slane '
  ($sum, $archive, $file) = @F;
  if (!$seen{$sum}++) {
    push @{$files{$archive}}, $file;
  }
  END {
    for $archive (keys %files) {
      open EXTRACT, "|-", "bsdtar", "-C", $dest, "-T", "/dev/stdin", "-xvnf", $archive;
      for (@{$files{$archive}}) {
        s/[][?*\\]/\\$&/g; # escape wildcards
        print EXTRACT;
      }
    }
  }' -- -dest='/mnt/e/My Documents/Pictures/' sums.txt

(you may want to add some more error handling).

It would likely be more efficient to do both checksum and extracting in perl at the same time though:

(LC_ALL=C find "$PWD" -name '*.zip' -type f -print0 | {
  cd '/mnt/e/My Documents/Pictures' &&
    perl -MArchive::Zip -MDigest::SHA=sha1 -0lne '
      my $zip = Archive::Zip->new();
      if ($zip->read($_) == AZ_OK) {
        for $member ($zip->membersMatching(".*\.jpe?g")) {
          $zip->extractMember($member)
            unless (!$seen{sha1(zip->contents($member))}++);
        }
      }'
)

(warning: this is all untested).