2

I'm trying to use rsyn with the --dry-run option to see if there is enough space for the real synchronization. For test purposes I'm trying to synchronize one directory Documents. The size of the directory is

  x@x:~$ du Documents
  ...
  640760    Documents/

The size of the file container I'm trying to rsync it with is

  x@x:~$ df /media/veracrypt2
  Filesystem             1K-blocks  Used Available Use% Mounted on
  /dev/mapper/veracrypt2      9928  1191      8737  12% /media/veracrypt2

Then I run the rsync command with:

x@x:~$ rsync -ar --dry-run --stats Documents/ /media/veracrypt2

Number of files: 665 (reg: 560, dir: 105)
Number of created files: 664 (reg: 560, dir: 104)
Number of deleted files: 0
Number of regular files transferred: 560
Total file size: 649,731,108 bytes
Total transferred file size: 649,731,108 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 19,429
Total bytes received: 2,115

sent 19,429 bytes  received 2,115 bytes  43,088.00 bytes/sec
total size is 649,731,108  speedup is 30,158.33 (DRY RUN)

I don't understand why there were only 19,429 bytes sent? The container is empty so all files of the directory Documents should be transferred, menaing 649,731,108!?

I have also tried it with a smaller directory Scripts which size is

 du -h Scripts/
 32K    Scripts/test/Logs
 56K    Scripts/test
 116K   Scripts/Logs
 264K   Scripts/color_schemes
 580K   Scripts/

Here the entire directory should be able to be copied. When running rsync for that directory I get the output:

sending incremental file list
./
after_install.sh
install-crafter.sh
install-eclipse.sh
mk_autostart_app.sh
package_backup.sh
pandora.sh
sync_script.sh
trackpoint_speed_sens.sh
wallpaper.sh
Logs/
Logs/LOG_SYNC.log
Logs/LOG_SYNC.log~
Logs/LOG_WALLPAPER.txt
Logs/Log_sync.log
Logs/PANDORA.log
Logs/test
color_schemes/
color_schemes/kile.kateschema
test/
test/sync_script.sh
test/Logs/
test/Logs/Log_sync.log

Number of files: 25 (reg: 20, dir: 5)
Number of created files: 22 (reg: 18, dir: 4)
Number of deleted files: 0
Number of regular files transferred: 18
Total file size: 360,658 bytes
Total transferred file size: 353,166 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 900
Total bytes received: 95

sent 900 bytes  received 95 bytes  1,990.00 bytes/sec
total size is 360,658  speedup is 362.47 (DRY RUN)

So why is the sent data here only 928 bytes?? Shouldn't that value indicate how many bytes need to be copied to the destination?

migrated from stackoverflow.com May 17 '16 at 18:27

This question came from our site for professional and enthusiast programmers.

  • You can add --verbose to get the full list of changes that would be done. – ivan_pozdeev May 15 '16 at 12:22
  • You also didn't show the contents of the source/dest path (or the aforementioned list of changes) to back your claim that "full data is gonna be transferred". – ivan_pozdeev May 15 '16 at 12:23
  • My guess is that rsync only calculates hash differences for file chunks but doesn't transfer actual data. Source code inspection is needed to say for sure (or maybe there's some debug output switch - then the outputs can be diffed). – ivan_pozdeev May 15 '16 at 15:01
  • Btw your question is actually two in one: 1)the explanation between the numbes; 2)calculating the space needed for sync. I answered both, but the 2nd one probably warrants a separate question for a full answer (code and such). – ivan_pozdeev May 15 '16 at 15:59
2

In --dry-run mode, rsync determines which files need to be transferred but doesn't actually transfer their data - 'cuz there's no use to do that, naturally. This implies it doesn't perform delta matching - because it's a part of data transfer logic.

The reason for the numbers reflecting that is... well... that the code shows actual statistics, not some "would-be" statistics.

Here's a diff between rsync -avvv --log-file=rsync.log --no-whole-file --stats doc doc2 for the doc folder in rsync source tree, between --dry-run and normal sync (with stripped timestamps, PID differences and heap stats):

 delta-transmission enabled
 recv_generator(doc,1)
 recv_generator(doc,2)
+set modtime of doc to (1463404939) Mon May 16 16:22:19 2016
 recv_generator(doc/README-SGML,3)
 recv_generator(doc/profile.txt,4)
 recv_generator(doc/rsync.sgml,5)
 send_files(2, doc)
 cd+++++++++ doc/
 send_files(3, doc/README-SGML)
+send_files mapped doc/README-SGML of size 672
+calling match_sums doc/README-SGML
+sending file_sum
+false_alarms=0 hash_hits=0 matches=0
+>f+++++++++ doc/README-SGML
+sender finished doc/README-SGML
 send_files(4, doc/profile.txt)
+send_files mapped doc/profile.txt of size 1935
+calling match_sums doc/profile.txt
+sending file_sum
+false_alarms=0 hash_hits=0 matches=0
+>f+++++++++ doc/profile.txt
+sender finished doc/profile.txt
 send_files(5, doc/rsync.sgml)
+send_files mapped doc/rsync.sgml of size 11843
+calling match_sums doc/rsync.sgml
+sending file_sum
+false_alarms=0 hash_hits=0 matches=0
+>f+++++++++ doc/rsync.sgml
+sender finished doc/rsync.sgml
 recv_files(1) starting
 recv_files(doc)
 recv_files(doc/README-SGML)
+got file_sum
+set modtime of doc/.README-SGML.hkH0u5 to (1463404939) Mon May 16 16:22:19 2016
+renaming doc/.README-SGML.hkH0u5 to doc/README-SGML
 recv_files(doc/profile.txt)
+got file_sum
+set modtime of doc/.profile.txt.Wdf4x9 to (1463404939) Mon May 16 16:22:19 2016
+renaming doc/.profile.txt.Wdf4x9 to doc/profile.txt
 recv_files(doc/rsync.sgml)
+got file_sum
+set modtime of doc/.rsync.sgml.JSte5H to (1463404939) Mon May 16 16:22:19 2016
+renaming doc/.rsync.sgml.JSte5H to doc/rsync.sgml
 generate_files phase=1
+set modtime of doc to (1463404939) Mon May 16 16:22:19 2016
 send_files phase=1
 recv_files phase=1
 generate_files phase=2
 send_files phase=2
 send files finished
-total: matches=0  hash_hits=0  false_alarms=0 data=0
+total: matches=0  hash_hits=0  false_alarms=0 data=14450
 Number of files: 4 (reg: 3, dir: 1)
 Number of created files: 4 (reg: 3, dir: 1)
 Number of deleted files: 0
 Number of regular files transferred: 3
 Total file size: 14,450 bytes
 Total transferred file size: 14,450 bytes
-Literal data: 0 bytes
+Literal data: 14,450 bytes
 Matched data: 0 bytes
 File list size: 0
 File list generation time: 0.001 seconds
 File list transfer time: 0.000 seconds
-Total bytes sent: 153
-Total bytes received: 793
-sent 153 bytes  received 793 bytes  378.40 bytes/sec
-total size is 14,450  speedup is 15.27 (DRY RUN)
-[sender] _exit_cleanup(code=0, file=main.c, line=1196): about to call exit(0) (DRY RUN)
+Total bytes sent: 14,723
+Total bytes received: 1,435
+sent 14,723 bytes  received 1,435 bytes  4,616.57 bytes/sec
+total size is 14,450  speedup is 0.89
+[sender] _exit_cleanup(code=0, file=main.c, line=1196): about to call exit(0)

The following is a diff between the initial dry run and a dry run after I 1)did a real sync, 2) edited one file, rsync.sgml. It shows that delta matching is not done in a dry run:

 received 3 names
 recv_file_list done
 get_local_name count=4 doc2
-created directory doc2
 delta-transmission enabled
 recv_generator(doc,1)
 recv_generator(doc,2)
 recv_generator(doc/README-SGML,3)
+doc/README-SGML is uptodate
 recv_generator(doc/profile.txt,4)
+doc/profile.txt is uptodate
 recv_generator(doc/rsync.sgml,5)
 send_files(2, doc)
<...>
 Number of files: 4 (reg: 3, dir: 1)
-Number of created files: 4 (reg: 3, dir: 1)
+Number of created files: 0
 Number of deleted files: 0
-Number of regular files transferred: 3
-Total file size: 14,450 bytes
-Total transferred file size: 14,450 bytes
+Number of regular files transferred: 1
+Total file size: 14,476 bytes
+Total transferred file size: 11,869 bytes
 Literal data: 0 bytes
 Matched data: 0 bytes
 File list size: 0
-File list generation time: 0.010 seconds
+File list generation time: 0.001 seconds
 File list transfer time: 0.000 seconds
-Total bytes sent: 153
-Total bytes received: 793
-sent 153 bytes  received 793 bytes  1,892.00 bytes/sec
-total size is 14,450  speedup is 15.27 (DRY RUN)
+Total bytes sent: 157
+Total bytes received: 830
+sent 157 bytes  received 830 bytes  658.00 bytes/sec
+total size is 14,476  speedup is 14.67 (DRY RUN)
 [sender] _exit_cleanup(code=0, file=main.c, line=1196): about to call exit(0) (DRY RUN)

Now, regarding your task - to check whether it is enough space on destination for real syncronization.

As you can see from the diff,

  • files are copied one by one,
  • each file is written to a temporary name, then moved over the original one

So, the amount of space needed at the destination is:

  • sum( max(existing_size,new_size) for all files to be synced) + max( (new_size) for all files to be synced)

The first term is the worst case scenario for the amount of "final" data at any moment, the second one is the space for a temporary copy.

Each file size can be padded up to a multiple of a storage unit size to allow for wasted space (if applicable for the target FS and the total expected amount is high enough to make a difference).

  • That sees quite complicated though, isn't there any other command to check if there is enough space for a synchronization? – wasp256 May 15 '16 at 17:29
  • @wasp256 well, you can get a crude conservative approximation: total size of the local folder times the maximum expected ratio of new_size/old_size (no more than 1.1-1.2 probably) plus the size of the largest file. It's more than enough in virtually all cases: if you're running out of disk space so badly that you need more precision, it's probably a sign that you're doing it wrong. – ivan_pozdeev May 15 '16 at 17:38
  • Okay thx for the answer – wasp256 May 15 '16 at 17:44
  • One last question, I just saw the line Total transferred file size: 353,166 bytes in the stats output, aren't those the bytes that would need to be transmitted in the real transmission? – wasp256 May 15 '16 at 18:53
  • @wasp256 No, it's the "total sum of all files sizes for just the transferred files": lbackup.org/backup_statistics . The net unmatched data size is "Literal data" which is 0 for a dry run. Anyway, why are you interested in transmission volume? You were interested in the disk space required at destination (which is a different thing, as you can see from the answer). – ivan_pozdeev May 16 '16 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.