52

Having migrated to Linux from Windows, I would like to find an alternative software to Winmerge or rather learn command line tools to compare and sync two folders on Linux. I would be grateful if you could tell me how to do the following tasks on the command line... (I have studied diff and rsync, but I still need some help.)

We have two folders: "/home/user/A" and "/home/user/B"

Folder A is the place where regular files and folders are saved and folder B is a backup folder that serves as a complete mirror of folder A. (Nothing is directly saved or modified by the user in folder B.)

My questions are:

  • How to list files that exist only in folder B? (E.g. the ones deleted from folder A since the last synchronization.)

  • How to copy files that exist in only folder B back into folder A?

  • How to list files that exist in both folders but have different timestamps or sizes? (The ones that have been modified in folder A since last synronization. I would like to avoid using checksums, because there are tens of thousands of files and it'd make the process too slow.)

  • How to make an exact copy of folder A into folder B? I mean, copy everything from folder A into folder B that exists only in folder A and delete everything from folder B that exists only in folder B, but without touching the files that are the same in both folders.

  • Why not use a proper backup program for this? Duplicity is one example. – Qudit May 16 '15 at 21:57
73

This puts folder A into folder B:

rsync -avu --delete "/home/user/A" "/home/user/B"  

If you want the contents of folders A and B to be the same, put /home/user/A/ (with the slash) as the source. This takes not the folder A but all of it's content and puts it into folder B. Like this:

rsync -avu --delete "/home/user/A/" "/home/user/B"
  • -a Do the sync preserving all filesystem attributes
  • -v run verbosely
  • -u only copy files with a newer modification time (or size difference if the times are equal)
  • --delete delete the files in target folder that do not exist in the source

Manpage: https://download.samba.org/pub/rsync/rsync.html

  • 6
    rsync: run rsync app, -a: do the sync preserving all filesystem attributes, -v: run verbosely, -z: compress the data during the sync (transport the data in compressed mode), --delete: delete the files in target folder that do not exist in the source, /home/user/A: source folder, /home/user/B: target folder – SonicARG Jul 9 '16 at 19:25
  • Hi SonicARG, I totally forgot to get back to this and put the explanation, thanks submitting the explanation, I put yours in the answer, hope you don't mind. – TuxForLife Jul 11 '16 at 5:43
  • 5
    Rsync is primarly meant to copy files between different computers, as explained here it can serve the purpose to sync directories as well. So the -z option is interesting to reduce network traffic and thus enhance the performance of an rsync between 2 computers: ( read data from disk -> compress) ===network===> (uncompress->write to disk) Using -z to sync 2 directories on the same host is a bit silly and waste of cpu cycles as you would get (read data from disk -> compress -> uncompress -> write to disk) – GerritCap Mar 29 '17 at 8:51
  • @GerritCap, I made an edit, thanks for your valuable input – TuxForLife Jul 6 '18 at 15:48
  • I've tried the command but it create a sub-dir /home/user/B/A instead of overwrite A's content to B's content. Could you help me to have a look on it? – Luke Sep 19 '18 at 3:24
9

You could unison tool developed by Benjamin Pierce at U Penn.

Let us assume you have two directories,

/home/user/Documents/dirA/ and /home/user/Documents/dirB/

To synchronize these two, you may use:

~$unison -ui text /home/user/Documents/dirA/ /home/user/Documents/dirB/

In output, unison will display each and every directory and file that is different in the two directories you have asked to sync. It will recommend to additively synchronize (replicate missing file in both locations) on the initial run, then create and maintain a synchronization tree on your machine, and on subsequent runs it will implement true synchronization (i.e., if you delete a file from .../dirA, it will get deleted from .../dirB as well. You can also compare each and every change and optionally choose to forward or reverse synchronize between the two directories.

Optionally, to launch graphical interface, simply remove the -ui text option from your command, although I find the cli simpler and faster to use.

More on this: Unison tutorial at Unison user documentation.

1

The answer from TuxForLife is pretty good, but I strongly suggest you use -c when syncing locally. You can argue that it's not worth the time/network penalty to do it for remote syncs, but it is totally worth it for local files because the speed is so great.

-c, --checksum
       This forces the sender to checksum every regular file using a 128-bit  MD4
       checksum.   It  does this during the initial file-system scan as it builds
       the list of all available files. The receiver then checksums  its  version
       of  each  file  (if  it exists and it has the same size as its sender-side
       counterpart) in order to decide which files need to be updated: files with
       either  a  changed  size  or a changed checksum are selected for transfer.
       Since this whole-file checksumming of all files on both sides of the  con-
       nection  occurs  in  addition to the automatic checksum verifications that
       occur during a file's transfer, this option can be quite slow.

       Note that rsync always verifies that each transferred file  was  correctly
       reconstructed  on  the receiving side by checking its whole-file checksum,
       but that automatic after-the-transfer verification has nothing to do  with
       this  option's  before-the-transfer  "Does  this file need to be updated?"
       check.

This shows how having the same size and time stamps can fail you.

The setup

$ cd /tmp

$ mkdir -p {A,b}/1/2/{3,4}

$ echo "\___________from A" | \
      tee A/1/2/x  | tee A/1/2/3/y  | tee A/1/2/4/z  | \
  tr A b | \
      tee b/1/2/x  | tee b/1/2/3/y  | tee b/1/2/4/z  | \
      tee b/1/2/x0 | tee b/1/2/3/y0 >     b/1/2/4/z0

$ find A b -type f | xargs -I% sh -c "echo %; cat %;"
A/1/2/3/y
\___________from A
A/1/2/4/z
\___________from A
A/1/2/x
\___________from A
b/1/2/3/y
\___________from b
b/1/2/3/y0
\___________from b
b/1/2/4/z
\___________from b
b/1/2/4/z0
\___________from b
b/1/2/x
\___________from b
b/1/2/x0
\___________from b

The rsync that copies nothing because the files all have the same size and timestamp

$ rsync -avu A/ b
building file list ... done

sent 138 bytes  received 20 bytes  316.00 bytes/sec
total size is 57  speedup is 0.36

$ find A b -type f | xargs -I% sh -c "echo %; cat %;"
A/1/2/3/y
\___________from A
A/1/2/4/z
\___________from A
A/1/2/x
\___________from A
b/1/2/3/y
\___________from b
b/1/2/3/y0
\___________from b
b/1/2/4/z
\___________from b
b/1/2/4/z0
\___________from b
b/1/2/x
\___________from b
b/1/2/x0
\___________from b    

The rsync that works correctly because it compares checksums

$ rsync -cavu A/ b
building file list ... done
1/2/x
1/2/3/y
1/2/4/z

sent 381 bytes  received 86 bytes  934.00 bytes/sec
total size is 57  speedup is 0.12

$ find A b -type f | xargs -I% sh -c "echo %; cat %;"
A/1/2/3/y
\___________from A
A/1/2/4/z
\___________from A
A/1/2/x
\___________from A
b/1/2/3/y
\___________from A
b/1/2/3/y0
\___________from b
b/1/2/4/z
\___________from A
b/1/2/4/z0
\___________from b
b/1/2/x
\___________from A
b/1/2/x0
\___________from b
0

This is not completely the same as what you ask for, but you could considered using a version-control tool. Tools like Git do everything you ask for, and more, especially if you do not work in folder B directly it could be interesting to take a look at it. you can find some more information on git here

  • 2
    This only works if you're willing to add everything to version control. It also forces every change ever committed to be permanently stored, which may be undesirable. – Qudit May 16 '15 at 21:58
  • @Qudit, that is true, although it is possible via cloning to limit the history, but limiting the history is not (yet ?) implemented in Git by default. – switch87 May 16 '15 at 22:07
  • @switch87 Yes, I know you can delete old commits. Version control is not really an appropriate solution for generic backups imo though, especially if there are large binary files. – Qudit May 16 '15 at 22:10
  • His question is for local backup, but if you use it for remote backup you can still use git annex for the bigger files. for local backup this is not a problem. – switch87 May 16 '15 at 22:22
  • 2
    @switch87 This really should've been a comment to the Q and not an answer since it doesn't explain how you'd use git to do backups. – slm May 17 '15 at 11:35
0

You can use it this way:

rsync -avu --delete /home/user/A/* /home/user/B/

This way you will copy the content of folder A into folder B, not the content of folder A itself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.