4

Recently, I was backing up a directory tree using cp -r, when I ran out of space in the receiving drive. I had to carry on with the backup, but to a different destination. Normally, to resume a cp command, you would ask cp to only copy the file if it isn't in the destination. You can see the problem here.

Here was my solution:

  1. Screw up.

    # cp -rv sourcedir destdir
    error: no space left on device.
    
  2. Make a list of all files that had been copied successfully.

    # cd destdir
    # find . >/tmp/alreadycopied
    
  3. Write a script that could take any list A and a blacklist B and return a new list C = B \ A, containing every element in A that is not in B. I called it setminus. ***generate list A*** | setminus listB returns C to stdout.

  4. Use find and setminus to copy all remaining files to the new destination.

    # cd sourcedir
    # find . -type f | setminus /tmp/alreadycopied | xargs -d '\n' -I file cp -v --parents file overflowdestdir
    

It worked, but I figure this set minus of lists is a common enough problem that the standard UNIX tools must somehow cover this use case, making my script unnecessary. Have any of you run into this problem, and if so, how did you solve it?

The setminus script:

#!/usr/bin/env python3

# ***generate list*** | setminus blacklist
# Performs a set minus on its inputs and returns the result. Specifically,
# filters a newline-separated list in stdin using blacklist---also a
# newline-separated list. If an element of stdin is found in blacklist, it is
# excluded from the output. Otherwise, the element is returned to stdout. Very
# useful in conjunction with find commands.

from sys import *

try:
    blacklistfile = argv[1]
except IndexError:
    stderr.write('usage: ***generate list*** | setminus blacklist.\n')
    exit(1)

# A dict is used instead of a list to speed up searching. This blacklist could potentially be very large!
blacklist = {}
for line in open(blacklistfile):
    blacklist[line] = True

for line in stdin:
    inblacklist = False
    try:
        inblacklist = blacklist[line]
    except KeyError:
        pass

    if not inblacklist:
        stdout.write(line)
4

If your lists are sorted you could use comm -23 to get the unique elements of the first list. If they are not, you could use grep like

find -type f | grep -vFxf /tmp/alreadyCopied
  • -v will find all the lines without a match
  • -F tells it to use the strings as fixed strings, not as patterns
  • -x matches the whole line instead of the string anywhere in the line
  • -f /tmp/alreadyCopied read the lines to match from the given file

You'll have to make sure the paths match though, so if find is producing ./dir1/file1 that needs to be the same string in /tmp/alreadyCopied

Do note though, that this general approach will have problems if, say, you have a filename with \n in it. You could probably redo the whole thing within find with something like

find . -type f -exec test ! -f destdir/{} \; -exec cp -v --parents {} overflowdestdir \;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.