0

On "source side" I am running a:

rsync -azP file.txt username@target:/root/

that I had to break once (CTRL+C), but then, issued the same command, so rsync is continuing, where it left (afaik).

On "target side" I can see these files:

username@target:/root> ls -lah|grep -i file.txt
-rw-r--r-- 1 username group 120G May  3 16:32 file.txt
-rw------- 1 username group 342G May  8 12:47 .file.txt.yISk8R
username@target:/root> 
  • file.txt: I think (?) it is the already transferred part, before breaking rsync once.

  • .file.txt.yISk8R: I think (?) it is the part that was/is transferred after the one-time rsync break happened.

  • Using rsync-3.1.0-12.1.x86_64 on SLES12

The q: what exactly are these two files on target side?

So when the rsync will finish, it will append the ".file.txt.yISk8R" to the "file.txt"?

Or is it more complicated?:

I can see that the start of the ".file.txt.yISk8R" looks same to the original "file.txt", so the rule could be something different..:

source:/root # head -1 file.txt|strings|head -1
A_.o
source:/root #

username@target:/root> head -1 .file.txt.yISk8R|strings|head -1
A_.o
username@target:/root>

username@target:/root> head -1 file.txt|strings|head -1
f/o*d
username@target:/root> 

I can see that the ".file.txt.yISk8R" is growing during the still ongoing transfer.

it isn't more clear even from https://linux.die.net/man/1/rsync

-a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
-z, --compress              compress file data during the transfer
-P                          same as --partial --progress
->
    --partial               keep partially transferred files
    ->
        --partial
            By default, rsync will delete any partially transferred file if the transfer is 
            interrupted. In some circumstances it is more desirable to keep partially 
            transferred files. Using the --partial option tells rsync to keep the partial 
            file which should make a subsequent transfer of the rest of the file much faster.

Original purpose is to copy these files forward from the "target" to another, "target2" while they are still being copied to the "target". I should start copying the ".file.txt.yISk8R" file? Then why is "file.txt" 120G sized? or data is duplicated somehow?

UPDATE on 2019 May 13: my god, I can see these are almost duplicated datas... so when you are copying a 600 GByte file and break at ex.: 500 GByte, you will need at least 1100 GByte storage on target side, since rsync with "-azP" will create almost duplicates files (.file.txt.xxxxx vs. file.txt). Why isn't it doing some hashing of smaller chunks and then continue based on what last chunk doesn't have the same hash?

1

Some background for the process can be read at https://rsync.samba.org/how-rsync-works.html (especially in "The Receiver" section).

what exactly are these two files on target side?

The first file was already there before you started (the most recent) sync. Perhaps you had already synced previously? Assuming default options, it will stay there until rsync believes the sync has completed.

The second file is the temporary file that was used during the recent sync. With default options, it will pick up at that point on the next run.

So when the rsync will finish, it will append the ".file.txt.yISk8R" to the "file.txt"?

No. It will delete the existing "file.txt", and will then rename the temporary file to be the new "file.txt".

I can see that the ".file.txt.yISk8R" is growing during the still ongoing transfer.

Yes. File transfer isn't atomic. That's why a hidden filename is used. Under normal circumstances you wouldn't want to access that file since it may be changing or incomplete.

If you're coping onward, you should ignore the temporary file. Just copy "file.txt". That file should always be consistent and complete (if not always the most up-to-date).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.