2

Deep and complex directory structure can be modified: files can be moved (prefix changed) and simultaneously content of the files can be changed partially.

I want to decrease an amount of data to transmit over network.

rsync works with chunks of data of 4k size (if I remember correctly).

Can I use some kind of concatenation (tar-like) preserving meta information about file system structure and attributes, which places the file's content starting from offsets of multiple 4k bytes for each next file to enable rsync algorithm advantages?

The problem I want to solve is omission of rsync: it unable to detect if file content changed and file is moved at the same time between synchronizations to use matching blocks of destination files to reduce data transfer.

  • @ilkkachu I want to transmit only new blocks of all the files using rsync. This means, that each concatenated file should start from rsync-specific block offset inside resulting archive file – Tomilov Anatoliy Dec 13 '17 at 15:40
  • Have a look at inosync – Patrick Mevzek Dec 13 '17 at 15:46
  • @ilkkachu Let's assume I can make such an imaginary tar archive. At initial step I just transfer the directory by rsync. Then I make such an archive on destination side locally. After update of source directory I create archive on the source side and synchronize it with remote using rsync. Then on remote side I just unpack whole the updated archive replacing the destination directory. In the case tar is responsible on archiving the file system attributes. – Tomilov Anatoliy Dec 13 '17 at 16:39
  • @ilkkachu rsync can find different blocks that differs in content inherently. So-called "delta-transfer algorithm" is the main feature of rsync. – Tomilov Anatoliy Dec 13 '17 at 16:55
  • I can imagine the solution: create mirror of the source directory w/ files filled w/ zeros, make compressed archive of them, then using dd with bs=4k and increasing seek I fill blob with contents of real files. Small compressed tar is transmitted and big blob is rsync-ed. On remote side one do inverse two-step procedure. – Tomilov Anatoliy Dec 14 '17 at 5:57
2

Based on your suggestion in a comment (which should really be in your question), this seems to be what you want

cd /path/to/directory
tar cf /var/tmp/directory.tar .
rsync -azv /var/tmp/directory.tar remote:/var/tmp/directory.tar
ssh remote 'cd /path/to/destination && tar xf /var/tmp/directory.tar'

You need enough space to store directory.tar on both sides.


I've been asked whether this apparently trivial solution can work in the situation where a small amount, such as a single byte, is added (or removed) from the beginning of the tar file.

Hopefully this sample will illustrate how good rsync is at handling such situations. It works best if you have an equivalence (certificate key) login for the remote server so that no time is spent entering a password.

# Generate some data
dd iflag=fullblock bs=1M count=200 </dev/urandom >200M.dat

# See how long it takes to transfer
time rsync -av 200M.dat remote:

# See how long it takes to transfer "nothing"
time rsync -av 200M.dat remote:

# Generate one byte of data and prepend it to another data file
dd bs=1 count=1 </dev/urandom >1b.dat
cat 1b.dat 200M.dat >200M1b.dat

# Copy the new file across to the original target
time rsync -av 200M1b.dat remote:200M.dat

# Clean up
rm 1b.dat 200M.dat 200M1b.dat
ssh remote rm 200M.dat

If the algorithm can handle the single byte inserted at the beginning of the data stream the transfer should take but a few moments. If it can't you would expect the transfer time to be broadly similar to the first.

  • This is not the solution. Imagine, that some single file in the beginning of the tar archive has size 0 byte and after update became of size 1 byte. 4k-blocks related to all the next files in a tar archive layout now cannot be recognized by rsync as identical to counterparts on remote side even if they are not changed at all. This is because of different offset. – Tomilov Anatoliy Dec 14 '17 at 5:52
  • Do you really think, that answer can be so trivial? – Tomilov Anatoliy Dec 14 '17 at 5:52
  • 1
    @Orient rsyncs algorithm is not dependent on the data’s position in the file, iirc. It chops the source into chunks, checks wether this chunk exists in the target file. The algorithm would be rather useless if a one byte offset would cause the whole file to be retransferred. – Markus W Mahlberg Dec 14 '17 at 8:14
  • @MarkusWMahlberg I'll read the article about rolling checksum. But intuitively it seems, that it is not possible, what you say. – Tomilov Anatoliy Dec 14 '17 at 8:48
  • @Orient I have added a proof of concept to my answer for you. See what you think. – roaima Dec 14 '17 at 9:58
1

Here's another suggestion for you. The hrsync tool, which I've found on GitHub, appears to be pretty good at maintaining a memory of files when you rename them or move them between directories of a source tree.

  • It can track moves and edits to files within the source tree
  • It is a shell script and does not require administrative privileges to install on the source system, although putting it into /usr/local/bin does have advantages
  • It requires both local and remote systems to have a filesystem capable of handling hard links
  • It is not able to track changes to a file where that file is renamed and replaced (i.e. deleted and then recreated, rather than edited in place)

Example

hrsync /path/to/directory/ remote:/path/to/destination/
0

I found the solution using bash and command-line utilities only. It is possible to optimize the solution: sort files by size in ascending order and place as many small files as possible to every chunk (knapsack problem here =), but it would be overengineering):

pack.bash:

#! /usr/bin/env bash

set -e

[[ -d "$1" ]]
[[ -d "$( dirname '$2' )" ]]

BLOB="$2.blob"
FSIZES="$2.sizes"
OFFSET=0

shopt -s globstar
for f in "$1"/* "$1"/**/*
do
    if [[ -f "$f" ]]
    then
        SIZE=$( stat -c %s "$f" )
        echo "$SIZE" >> "$FSIZES"
        COUNT=$(( ($SIZE + 4096 - 1) / 4096 ))
        dd if="$f" of="$BLOB" bs=4096 seek=$OFFSET count=$COUNT conv=notrunc
        OFFSET=$(( $COUNT + $OFFSET ))
    fi
done

cp --recursive --archive --attributes-only "$1" "$2.dir"
XZ_OPT="-9e --threads=$(( $( nproc ) + 1 ))" tar cpJf "$2.tar.xz" -C "$2.dir" .
rm --recursive "$2.dir"

unpack.bash:

#! /usr/bin/env bash

set -e

BLOB="$2.blob"
FSIZES="$2.sizes"

[[ -f "$BLOB" ]]
[[ -f "$FSIZES" ]]

mkdir --parents "$1"
[[ ! "$( ls -A '$1' )" ]]

tar xpJf "$2.tar.xz" -C "$1"

SIZES=($( < "$FSIZES" ))

i=0
OFFSET=0
shopt -s globstar
for f in "$1"/* "$1"/**/*
do
    if [[ -f "$f" ]]
    then
        SIZE=${SIZES[i]}
        dd if="$BLOB" of="$f" bs=4096 skip=$OFFSET count=$SIZE iflag=count_bytes
        OFFSET=$(( $OFFSET + ($SIZE + 4096 - 1) / 4096 ))
        i=$(( $i + 1 ))
    fi
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.