Advanced file filtering

Question

I have 3 distinct folders: history, inbox, backup.

I need to copy all the files from 'history' to 'inbox', only if they are not present in 'backup'.

How to do this?

Answer 1

Just an example, is there any subfolder in history ?

for x in history/*;
do
  [[ -f backup/"$(basename "$x")" ]] || cp "$x" inbox
done

This script would loop through all possible files in history folder, and extract the basename of it (e.g the basename of /bin/ls is ls), and check if the file exists in backup folder; if not, do the copy procedure.

Answer 2

This simple script works only if the three directories don't contain any subdirectories (a more complicated recursive algorithm would be required to handle that case).

Warlock's answer is good, but the advantage of a script like this over Warlock's one-liner is that by using xargs we don't fork an instance of cp for every file that we copy. We only fork one instance of cp for every time xargs generates a command line that fills the shell's command-line length limit. That tends to be huge on modern systems, so it's quite likely we'd only run one instance of cp total, even for hundreds or possibly thousands of files in history/

#! /bin/sh

# use tempfile(1) if available, otherwise use:
# EXCLUDEFILE="/tmp/excl.$$"
EXCLUDEFILE=$(tempfile -p excl)

# generate an exclude file, changing backup/ to history/    
find  backup/ -type f | sed -e 's:^backup/:history/:' > "$EXCLUDEFILE"

find history/ -type f -print0 | \
  grep -z -Z -F -v -x -f "$EXCLUDEFILE" | \
  xargs -0r -i{} cp {} inbox/

rm "$EXCLUDEFILE"

FYI, the long-form of the grep options used here are:

grep --null-data --null --fixed-strings --invert-match \
      --line-regexp --file "$EXCLUDEFILE"

and an even more compact form is:

grep -zZFvxf "$EXCLUDEFILE"

Most of the options are commonly used or self-explanatory, but three may require some explanation:

-z or --null-data tells grep to expect null-terminated input (from find -print0)
-Z or --null tells grep to output null-terminated lines (to pipe into xargs -0)
-x or --line-regexp tells grep to match only on the whole full lines