1

I have a pretty hard problem here.

I have a photo library with a lot of photos in it in various folders.

I then started using Google Photos for my photos, I put those originals into Google Photos, and used it for 5+ years.

Now I want to move away from Google Photos. I have done a Google Takeout of all my photos, and downloaded all the Zip files, ~1.5TB worth of them (150 x ~10GB files).

Now I want to keep my original directory structure, and delete all the files that are duplicated in Google Photos. After this operation, I basically want to have two directories left over each with unique files in them. I can then merge this by hand later.

I have started extracting all the files and then I will run rmlint to detect duplicates and purge from Google Drive. The problem is I don't have enough space to maneuvre all this around, so I have to extract say 30 archives, then run rmlint, purge, extract another 30, run rmlint again, purge, etc. This rescans my original files over and over, and it's going to take a really long time to do. I already use the --xattr flag for rmlint to try and speed up subsequent runs. See appendix for full rmlint command.

How can I do this WITHOUT having to first extract all the archives? Is there a way to just use the file checksums in the zip files and compare to those?

Thanks!

Appendix

rmlint \
        --xattr \
        -o sh:rmlint-photos.sh \
        -o json:rmlint-photos.json \
        --progress \
        --match-basename \
        --keep-all-tagged \
        --must-match-tagged \
        "/mnt/f/GoogleTakeout/" \
        // \
        "/mnt/e/My Documents/Pictures/" \

1
  • Your workflow is badly designed. You should use zip to list the contents of each .zip file, select the desired files, and, in a second pass, have zip extract only those files. rmlint should not be needed. Rather than cleaning up junk, don't make junk in the first place. Read man hardlinks.
    – waltinator
    Jul 28 at 3:59

1 Answer 1

1

In zsh or bash and with libarchive's bsdtar and GNU tar, you could do something like:

LC_ALL=C find . -name '*.zip' -type f -print0 |
  while IFS= read -rd '' archive; do
    bsdtar -cf - "@$archive" |
      ARCHIVE=$archive tar -xf - --to-command='
        case $TAR_FILETYPE$TAR_FILENAME in
          (f*.jpg | f*.JPG)
            sha1sum | {
              IFS= read -r sum rest &&
                printf "%s\n" "$sum:$ARCHIVE:$TAR_FILENAME"
            }
        esac' > sums.txt

(assuming none of the archive paths nor members contain newline nor : characters) to get a list of <checksum>:<archive>:<file-in-archive> for each of the archives (will take a while as it runs one sha1sum per image).

Then, you could extract only one file for each unique checksum with something like:

perl -F: -slane '
  ($sum, $archive, $file) = @F;
  if (!$seen{$sum}++) {
    push @{$files{$archive}}, $file;
  }
  END {
    for $archive (keys %files) {
      open EXTRACT, "|-", "bsdtar", "-C", $dest, "-T", "/dev/stdin", "-xvnf", $archive;
      for (@{$files{$archive}}) {
        s/[][?*\\]/\\$&/g; # escape wildcards
        print EXTRACT;
      }
    }
  }' -- -dest='/mnt/e/My Documents/Pictures/' sums.txt

(you may want to add some more error handling).

It would likely be more efficient to do both checksum and extracting in perl at the same time though:

(LC_ALL=C find "$PWD" -name '*.zip' -type f -print0 | {
  cd '/mnt/e/My Documents/Pictures' &&
    perl -MArchive::Zip -MDigest::SHA=sha1 -0lne '
      my $zip = Archive::Zip->new();
      if ($zip->read($_) == AZ_OK) {
        for $member ($zip->membersMatching(".*\.jpe?g")) {
          $zip->extractMember($member)
            unless (!$seen{sha1(zip->contents($member))}++);
        }
      }'
)

(warning: this is all untested).

1
  • 1
    +1. The %seen hash in the perl part of the one-liners could keep the pathname of the 1st seen sha1 instead of a counter, then it could use link(old,new) to make a hard-link for dupes. Another enhancement for the last version would be to allow it to be re-run by saving the %seen hash (using Data::Dump or some other serialisation module) on exit, and load it when it first starts up. Maybe use a second hash with key=filename, val=sha1. Also, the : separated format in the two stage version could use NUL instead, and the perl part could read in three NUL-separated records at a time.
    – cas
    Jul 29 at 0:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .