5

Context

I have user uploaded content that needs to be backed up. The content is over 3 separate servers at /var/www/domain/media/ (on each server it is in the same place). The backup is a NFS mounted RAID at /var/www/domain/bak/.

media/ is owned by a different user than bak/, basically the webapp can write to media/ but can only read bak/ (users can delete their upload only until it is backed up at 00:00 GMT).

This has resulted in two issues: the user can force the same filename to overwrite the file in the backup, and a file in media/ can end up on two different servers (exactly the same file if the user upload it two times and it gets served by a distinct server).

All this runs on 4 CenOS 7 (web X 3 + backup X 1). The "web" servers have limited disk space, and moving things to the backup server is needed to keep them from filling the disks.

There are no race conditions so that is something we do not need to care about. The backup is done from the single backup machine, by executing commands through ssh over the other three machines, sequentially.


Current solution

The "move" of the files to backup is done after purging duplicates:

find /var/www/domain/media/ -type f | > media
find /var/www/domain/bak/ -type f | awk '{a=gensub("bak","media",1); print a}' > bak
cat bak media | sort | uniq -d > dupes
cat dupes | xargs rm
cp -r /var/www/domain/media/* /var/www/domain/bak/
rm -rf /var/www/domain/media/*

The problem with using mv is that /var/www/domain/media/ has subdirectories per user. For example:

media/user13/myvideo.webm
media/user13/walk-in-the-park.webm
media/user16/cat-video.webm
media/user17/presentation-may-2016.webm

bak/user13/mountai-trip.webm
bak/user13/walk-in-the-park.webm
bak/user14/reax-the-dog.webm

The command must create directories for user16 and user17, whilst it must avoid overwriting bak/user13/walk-in-the-park.webm.


Issue with the current solution

I would like to keep the duplicates on media/ instead of deleting them. Copying them to another place falls into the same problem since new files will come during the day and i'll need to sync the dupes with their copies.

How can i move all files media/ that are not in bak/ whilst keeping the directory structure and not removing the files already present.

In other words i'm looking for a move that will perform:

source      | destination         | action
----------- | ------------------- | ----------------------------------
file exists | file does not exist | move (`mv`), source -> destination
file exists | file exists         | do nothing, both files stay as they are
no file     | file exists         | do nothing (will not trigger)
no file     | file does not exist | do nothing (well, there's nothing to do something with!)

Attempts at a more elegant solution

I believe rsync shall be able to perform this. I'm aware of --remove-source-files but i cannot find a way for it not to check timestamps, checksums, filesize, everything.

I'm keeping and checking checksums as a completely separate process.

I only care about the filenames. I'm aware that it might lead to file corruption but i'm afraid that it is much easier to get a corrupted file on the normal disk rather than on the RAID server.

Non-rsync solutions are welcome. I though of writing a shell-script to perform the move (extend the script from the Current solution section). Yet, once i though how error prone it would be i gave up.

I also tried:

tar -cf /var/www/domain/media | (cd /var/www/domain/bar; tar -kxf -)

But it is both too slow for media files (which might be rather big) and keeps all files at media/ (which has limited disk space).

6
  • Would cp -lrn (creating hard links) followed by a removal of the source files do the trick? Can the source files have multiple hard links? Jun 4 '16 at 22:35
  • @Gilles - unfortunately hard links are out of question, the move always happens between different disks: one on the machine and one NFS mount. I also added a little clarification about race conditions that, i believe, has been badly written in the question.
    – grochmal
    Jun 5 '16 at 22:00
  • 1
    Oh, I thought you wanted to move rather than copy+delete because it was on the same filesystem. So what's wrong with rsync --remove-source-files then? I don't understand what you want to do when a file exists in the source and in the destination: overwrite the destination? ignore the file? remove the file from the source? do one of these based on some extra information? Jun 5 '16 at 22:10
  • @Gilles - my only issue with rsync --remove-source-files is that it will copy the source file if it is never than the destination (i.e. when the destination file exists). The files that are on backup shall never be overwritten. I need to admit that i do not use rsync much, and might have overlooked something completely trivial.
    – grochmal
    Jun 5 '16 at 22:28
  • 1
    --ignore-existing? Jun 5 '16 at 22:39
3
+50

To do nothing if the file already exists in the destination tree (regardless of any metadata), pass the option --ignore-existing to rsync.

rsync -a --remove-source-files --ignore-existing /var/www/domain/media/ /var/www/domain/bak/

For completeness, here's a solution based on find and mv for the case when the source and the destination are on the same filesystem (in this case, rsync is not a good solution because it copies then removes files, rather than simply moving them to the target directory).

cd /var/www/domain/media
find -type f -exec '
  for x; do
    if ! [ -e "/var/www/domain/bak/$x" ]; then
      mkdir -p "/var/www/domain/bak/${x%/*}" &&
      mv -- "$x" "/var/www/domain/bak/$x"
    fi
  done
' sh {} +
4
  • Good solution, but the problem is this can be Extremely Slow for a large directory tree, as it is copying every file instead of simply moving (inodes)
    – Rucent88
    Aug 20 '21 at 14:28
  • @Rucent88 The question has an explicit assumption that the source and destination are on different filesystem. But for the sake of visitors who might find this question when looking for a similar case with the source and the destination are on the same filesystem, I've added a solution that renames the files. Aug 20 '21 at 21:55
  • @Rucent88 The question (from 2016, like this answer!) states that the backup directory is on an NFS-mounted RAID. Therefore, we cannot think about moving inodes.
    – Kaz
    Aug 20 '21 at 23:42
  • @Kaz I didn't read fully. My apologies
    – Rucent88
    Aug 21 '21 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.