1

We want to copy ~800GB of data from a source server to destination server using rsync. I was wondering if its safe to use rsync without shutting down source server application? Can rsync handle real time data inflow on source while it copies/syncs the data back to destination server?

1

For backups and real-time synchronization I use glusterfs. Is a very friendly and robust sync & backup package.

See more about glusterfs in here.

1

Rsync "handles" it by ignoring it. Does that count? Not really... Any changes that get made after rsync copies that portion of the file are lost and not copied. This can mean that different parts of the same file contain data from different points in time.

As you suspected, this can be a problem.

A workaround I often use on linux (saw your tag) is to create an LVM snapshot, mount that read-only, then copy the snapshot data. An LVM snapshot is guaranteed to be unchanging for the lifetime of the snapshot, but it must be sized to hold enough of the changed data (writes.) When an LVM snapshot fills, it becomes unavailable.

For example:

# lvcreate --snapshot /dev/vgName/lvName --size 1G --name snapNameDate
# mkdir /snaps
# mkdir /snaps/lvName/Mount/Point
# mount -o ro /dev/vgName/snapNameDate /snaps/lvName/Mount/Point
# cd /snaps/lvName/Mount/Point
# rsync --archive --verbose --progress . user@destination:/destination/Point/.
# cd /
# umount /snaps/lvName/Mount/Point
# lvremove /dev/vgName/snapNameDate
  • Unfortunately, we have to copy data from a VFS shared drive which does not support taking snapshot of it :( Hence rsync seems to be the only way out at the moment. – Himanshu Feb 4 '16 at 15:57
  • Isn't VFS just an abstraction layer (en.wikipedia.org/wiki/Virtual_file_system)? What is underneath the VFS that's actually storing your data? Note that any process that read at the file level will have the same problem as rsync does with getting different data at different times out of the same changing file, this includes cp, dd, tar, etc. – Steve Bonds Feb 4 '16 at 16:18
  • Thanks Steve for your inputs! I totally agree with your point on VFS but we have external team internally which manages the same and taking a snapshot is under their realms. – Himanshu Feb 4 '16 at 19:18
  • 1
    On rsync, I did one mock test to copy about 500MB of data from source server to destination server using "rsync -av --progress --sparse" but was little surprised by the fact that the destination server directory size is much bigger (about 574 MB)than actual source directory data (500MB). Could this be due to sparse files at destination? If Yes, is there any impact of having sparse files on application? Any way to suppress those sparse files (if that is at first the reason) ls | wc -l on both directories are giving up same number of files. – Himanshu Feb 4 '16 at 19:19
  • 1
    The --sparse (-S) flag will help if it really is due to sparse files. The only impact you'll have with non-sparse files is the increase in disk allocation. (You can recover the sparseness with cp or rsync at a later date, when copying to another medium.) – roaima Feb 5 '16 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.