4

I have over 100 microservices which are first built on a local machine and then later rsynced into destination machine and started.

All microservices use a shared fat.jar file, rename it and store in their distribution folder.

/serviceA
  /a.jar
/serviceB
  /b.jar
...

When we rsync this to the server, rsync won't figure out that all jar files (which together account for 99% of the distribution) are the exact same fat.jar. So, if rsync was smarter, it could actually only transfer one a.jar and then copy it over for all others (since the size and hash of those will be exactly the same).

Is this possible to do with rsync or shall I look for another solution? This can reduce deployment speed significantly, especially when I have poor internet connectivity!

5

Don't rename the original fat.jar on each server.

If something has to access the file with another name, then create a symbolic link to the file.

For serviceA:

ln -s fat.jar a.jar

For serviceB:

ln -s fat.jar b.jar
4
  • i have to avoid symbolic links in this case (since each app distribution must be self contained) and some other reasons... But if i dont rename them rsync will figure it out? so the only problem rsync does not detect this is because of the rename? – vach May 29 '18 at 6:11
  • @vach Yes, if the file has the original name (and timestamp, and permissions etc.). But it strikes me now that if you keep the a.jar name and rsync a second time, then rsync should notice that a.jar on your local machine is the same as a.jar on the remote machine, provided that you used rsync -a both times. – Kusalananda May 29 '18 at 6:20
  • because the fat.jar is one jar that has all changes/dependencies from all 100 services, so if i change anything in binary of any app, all binaries technically change... they are deployed to few machines but there are 100 copies of these jars... so you see – vach May 29 '18 at 6:35
  • btw it seems rsync is smart enough to send part of the changed jar, apparently some of the binaries are unchanged (say first 40%) then it wont send it somehow... maybe chunks down to 10 pieces and computes hashes... – vach May 29 '18 at 6:40
1

There are some deduplication tools that can do this for you. If you install zbackup, which is probably available as a package for your system, on the local and remote machines, you can feed it a tar of your files and it will find the parts that are duplicated, and not keep these copies.

You do not have to change your source at all, by renaming, hard linking, or soft linking. Here is an example script which creates a large file and copies it to 3 directories A, B, C. It then tars the directories (uncompressed) into zbackup. We compare the size of the resulting repository, and what an conventional compressed tar would be. Normally, at this stage the respository would now be copied to the remote, and unpacked on the remote, but the script just unpacks it via tar into a new directory so we can compare with the original.

ZB=/tmp/zrepo
cd /tmp/; mkdir try; cd try
dd count=5000 if=/dev/urandom of=file
for dir in A B C
do  mkdir $dir
    date >$dir/a
    cp file $dir/b$dir
done
ls -l /tmp/try/*/*
zbackup init --non-encrypted $ZB
tar cf - A B C  | zbackup backup --non-encrypted $ZB/backups/x
du -bs $ZB
tar czf - A B C | wc -c
cd /tmp; mkdir copy; cd copy
zbackup restore --non-encrypted $ZB/backups/x | tar xf -
ls -l /tmp/copy/*/*

Here's some of the output. As you can see the repository takes only 2632045 bytes, compared with a compressed tar of 7682010 bytes, showing that the 3 copies of the large file have been deduplicated to 1 copy.

-rw-r--r-- 1 meuh      30 Jun  2 12:35 /tmp/try/A/a
-rw-r--r-- 1 meuh 2560000 Jun  2 12:35 /tmp/try/A/bA
-rw-r--r-- 1 meuh      30 Jun  2 12:35 /tmp/try/B/a
-rw-r--r-- 1 meuh 2560000 Jun  2 12:35 /tmp/try/B/bB
-rw-r--r-- 1 meuh      30 Jun  2 12:35 /tmp/try/C/a
-rw-r--r-- 1 meuh 2560000 Jun  2 12:35 /tmp/try/C/bC

4       /tmp/zrepo/info
4       /tmp/zrepo/index/2e0ec29dfd5742005a477525009cfa3a6677f28cffaf2ae5
4       /tmp/zrepo/backups/x
2052    /tmp/zrepo/bundles/e0/e0a14717771602304b480202e05a4f796e8346b7033c231e
2052    /tmp/zrepo/bundles/e0
520     /tmp/zrepo/bundles/3c/3cf381e405fc278c4336ae331c5ea6a9d67b3147792567bc
520     /tmp/zrepo/bundles/3c

2632045 /tmp/zrepo # du -bs of repo

7682010            # size of tar z

-rw-r--r-- 1 meuh      30 Jun  2 12:35 /tmp/copy/A/a
-rw-r--r-- 1 meuh 2560000 Jun  2 12:35 /tmp/copy/A/bA
-rw-r--r-- 1 meuh      30 Jun  2 12:35 /tmp/copy/B/a
-rw-r--r-- 1 meuh 2560000 Jun  2 12:35 /tmp/copy/B/bB
-rw-r--r-- 1 meuh      30 Jun  2 12:35 /tmp/copy/C/a
-rw-r--r-- 1 meuh 2560000 Jun  2 12:35 /tmp/copy/C/bC
0

yes, it is because you rename the files, so it is a different file each time for rsync. rsync is not intended to find duplicates. It is just a fast file copy tool. If you are aware of the files you won't copy multiple times, just exclude them with a rsync filter rule and deal with it on a separate way.

Exmpl. rsync -uva --filter "- a.jar" /somedir/ /otherdir/, will copy everything from /somedir to /otherdir except a.jar

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.