3

If I have some number of source directories, e.g. dir1, dir2... dir5, how can I create an automatically syncing "merged" destination directory that has all the files and directories of the source directories but doesn't involve duplicating files?

For example, I have the following source directory structure...

dir1/
  - a
  - b

dir2/
  - b   <-- Note the duplicate name, this one is more recent than dir1\b
  - c
  - dir2.1/   <-- Subdirectories present too.
      - z

dir3/
  - d

... which when "merged" would look like this:

merge/
 - dir2.1/
     - z
 - a
 - b   <-- Which one to show is based on modified time; most recent first.
 - c
 - d

(Assume there could be dozens of directories with thousands of files and subdirectories.)

All the files and directories in the source list should remain in place and unchanged and the "merged" destination directory should take up no addition storage; i.e. It's possibly just all symlinks managed by some daemon using inotify. The source directories are also having files and subdirectories added and removed frequently and this needs to be reflected, as soon as possible, in the "merged" directory too.

Some usage examples:

I create a new file, dir3/e, and it automatically and immediately (or within a few seconds) appears in the merge directory.

I remove the file dir1/a and it automatically and immediately (or within a few seconds) disappears from the merge directory.

I edit dir3/d by opening merge/d

I call touch on dir1/b so it it has a newer modified date than dir2/b and so merge/b automatically updates to point to dir1/b as it is the most recent.

I remove dir1/b and now merge/b will point to the older file dir2/b.

I attempt to create a file in merge but get an error because doing so makes no sense!

  • 1
    What if a directory has the same name as a file? – gollum Mar 20 '16 at 0:26
  • @gollum An intelligent solution would show both the file and the directory and know the difference... can Linux file systems can do this? If not I suppose it should pick something and warn me or pick nothing and tell me the error. – x-x Mar 20 '16 at 1:06
  • 2
    Dynamically combining the content of several directories is exactly what a union mount (or union filesystem) is about. But I can't find one that resolves conflicts based on mtime, they all use the same preference rules for all files (e.g. dir1 always takes precedence over dir2). – Gilles Mar 20 '16 at 15:43
3

Dynamically combining the content of several directories is exactly what a union mount (or union filesystem) is about. There are several implementations on Linux, but the usual ones resolve duplication between branches by always preferring one particular branch (e.g. dir2/file always has precedence over dir1/file if both exist). However I've just discovered mergerfs which has more flexible policies, and in particular allows picking the newest file on a file-by-file basis.

The setup:

mkdir dir1 dir2 merge
echo 1 >dir1/b; echo 2 >dir2/a
sleep 1
echo 1 >dir1/a; echo 2 >dir2/b; echo 2 >dir2/c

Now we mount. We set the newest policy for file access (search) and metadata modification (action), and forbid creation (create) (you could make creation pass down to one of the branches, either always the same, or based on available disk space, or even randomly!).

$ mergerfs -o category.action=newest,category.search=newest,category.create=erofs dir1:dir2:dir3 merge
$ head merge/?
==> merge/a <==
1

==> merge/b <==
2

==> merge/c <==
2

$ rm dir2/c
$ echo merge/?
a b

To unmount: fusermount -u merge

0

This is my hack using bash. Drawback of the solution is that during the script run it creates symlinks for every copy but as the last filename (with the same name) is the one with newest timestamp the symlink to it will remain.

Be sure to change MDIR (merge dir) and BDIR variables (initial directory/directories) before you will run it.

MDIR=/tmp
(
BDIR="/tmp/ssh /etc/ssh"
for entry in $(find $BDIR -type f -exec stat --printf "%n,%Y\n" {} \;);do

  fname=$(echo $entry | cut -f 1 -d ,)
  tstamp=$(echo $entry | cut -f 2 -d ,)
  bname=$(basename $fname)

  echo $bname,$fname,$tstamp

done
) | sort -t, -nk 3 | \
while read entry;do
  bname=$(echo $entry | cut -f 1 -d ,)
  fname=$(echo $entry | cut -f 2 -d ,)
  tstamp=$(echo $entry | cut -f 3 -d ,)
  ln -svf $fname $MDIR/$bname
done
0

Use ln to create symbolic (-s) links which allow multiple pointers to the same actual file will answer your main question. Your usage example of automatic addition however are now met by this (but this may be useful to others reading your title and header).

ln -s src/of/actual/file merge_directory/pointer_link

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.