Copy with Find Creates Duplicate Files

Question

I'm trying to copy the latest files and directories from one location to another. I'm using the find command with a reference file. Any files newer than the reference file get copied.

find  /root/test/BACKUP/   -newer /root/test/BACKUP//reference -exec cp -r  '{}' /root/test/backup_170213_0328 ';'

The issue is if the /root/test/BACKUP/ has an sub directories these get copied separately in addition to getting copied as a sub directory. This creates duplicate files in the destination.

[root@localhost S3_Backup]# tree /root/test/BACKUP/
/root/test/BACKUP/
├── Chrysanthemum.jpg
├── Desert.jpg
├── Hydrangeas.jpg
├── Jellyfish.jpg
├── level2
│   ├── Lighthouse.jpg
│   ├── Penguins.jpg
│   ├── teddy.jpg
│   └── Tulips.jpg
└── reference

1 directory, 9 files
[root@localhost S3_Backup]# tree /root/test/backup_170213_0328
/root/test/backup_170213_0328
├── BACKUP
│   ├── Chrysanthemum.jpg
│   ├── Desert.jpg
│   ├── Hydrangeas.jpg
│   ├── Jellyfish.jpg
│   ├── level2
│   │   ├── Lighthouse.jpg
│   │   ├── Penguins.jpg
│   │   ├── teddy.jpg
│   │   └── Tulips.jpg
│   └── reference
└── level2
    ├── Lighthouse.jpg
    ├── Penguins.jpg
    ├── teddy.jpg
    └── Tulips.jpg

3 directories, 13 files

Answer 1

If you're on non-embedded Linux, or more generally if your cp command is from GNU coreutils, then you can use cp -u to copy only files that are newer than the destination. If you don't already have files under /root/test/backup_170213_0328 that are more recent than the files under /root/test/BACKUP/, then this is equivalent to what you're doing now, except that files won't be copied twice.

You should also pass the -p option to cp. Most of the time, and especially when making backups, you should preserve permissions. And you should pass -d if you want to copy symbolic links as such, rather than copy their targets. -rdp can be abbreviated as -a.

find  /root/test/BACKUP/   -newer /root/test/BACKUP//reference -exec cp -a  {} /root/test/backup_170213_0328 ';'

However, it's unclear to me why you would want to copy a whole directory tree if its root has been modified. For example, creating a file directly under /root/test/BACKUP would cause the whole tree to be copied. You should probably not copy directories.

find  /root/test/BACKUP/ ! -type d -newer /root/test/BACKUP//reference -exec cp -RPp {} /root/test/backup_170213_0328 ';'

Furthermore all files are getting copied under the root directory, instead of reproducing the directory tree. You can fix that by using a tool whose main purpose is to copy directory trees, rather than a tool whose main purpose is to copy individual files. For example, with pax:

find  /root/test/BACKUP/ ! -type d -newer /root/test/BACKUP//reference -exec pax -rw -pe {} /root/test/backup_170213_0328 ';'

Or with rsync:

find  /root/test/BACKUP/ ! -type d -newer /root/test/BACKUP//reference -exec rsync -a {} /root/test/backup_170213_0328 ';'

But if you want to make incremental backups, I suspect that what you're really after is rsync's --link-dest. Point it at your previous backup, and it'll make a backup where files that already existed are hard-linked to the previous backup.

rsync -a --link-dest=/root/test/backup_170213_0228 /root/test/BACKUP /root/test/backup_170213_0328