2

I want to backup my home directory to an external SSD drive using rsync.

I'm on Arch Linux. My home is ext4 (251G), the SSD is NTFS-3G mounted as fuseblk (512G).

The exact rsync invocation is:

rsync -aSh --info=progress2 --delete --exclude=/me/.cache /home/me /run/media/me/Samsung_T5/

Eventually, it fails with this being its last words:

        218.76G  99%   25.08MB/s    2:18:36 (xfr#2093188, ir-chk=1368/2286507)
rsync: write failed on "/run/media/me/Samsung_T5/me/a_file": No space left on device (28)

So, rsync allegedly copied around 218G of data and couldn't go furhter due to my SSD being full.

When I ask du how much data is there on my SSD rsync destination, it says 466G.

$ du -hs /run/media/me/Samsung_T5/me
466G    /run/media/me/Samsung_T5/me

This is weird. rsync tried to copy 281G, but it copied 218G and failed because it actually copied 466G.

What am I getting wrong here?

I do know that NTFS and ext4 are different. But are they different enough to make my files more than 2x larger? Am I copying more than I actually have in my home?

What would be the correct rsync procedure to back up my ~280G home to my SSD as something comparable in size with my home?

UPDATE [Thanks to the comments below]:

I have a large number of small files in my source directory and a certain amount of sparse files. For example, there is a file 4K big in the source and 128K big in the destination. There is also a sparse file that is 12K in the source and 128K in the destination.

Also, I do have 244 hard links to different executables (e.g., shared libraries). Some of those hard links point to some relatively large files. For example, a version of binutils linker (ld) is around 7M and I have 4 hard links to it.

Improve this question
15
  • 1
    In addition to sparse files, do you have a lot of very small files (say less than 100 bytes)? The minimum amount of space that a file uses can vary between filesystems. – icarus Nov 3 '19 at 22:40
  • 1
    It may work better to backup the system into a compressed tar archive file. This way the ownership and permissions will be saved and can be restored when extracted into a linux file system. And the small files and sparse files should not occupy too much space. -- If you still want to use rsync, you had better copy into a linux file system, e.g. 'ext4`. This way it should work to copy hard links, small files and sparse files in a correct way and avoid that the backup size is bigger than the soulrce. – sudodus Nov 4 '19 at 23:33
  • 1
    @sudodus The fact is that I do want to have an incremental backup of my home. Is there a technique to build tarballs incrementally (as rsync syncs only changed files)? – foki Nov 5 '19 at 14:30
  • 2
    There are probably other tools, but you might look at duplicity and its gui deja-dup. It does incremental backups using tar files, optionally encrypted, optionally to a remote server. – meuh Nov 5 '19 at 14:45
  • 1
    Yes, there are tools for incremental backup using tar. It is easiest to use such tools, but you can also look at the basics, for example at this link, where the crucial additonal option for incremental backup is --listed-incremental=snapshot-file – sudodus Nov 5 '19 at 15:30
2

You might look at duplicity and its gui deja-dup. It does incremental backups using tar files, optionally encrypted, optionally to a remote server.

It uses librsync and its rolling-checksum algorithm so that each incremental archive holds only the changed parts of files.

The home page says it handles Unix permissions, symbolic links, fifos, and device files, but does not preserve hard links. If you have many large hard-linked files it may be sub-optimal in the archive, but more importantly, you may also want to note separately which files are interlinked so that if you need to restore them you can put back the link. If possible, converting to symbolic links would solve this problem.

You can look for hard links with something like

find /home/me -links +1 -type f -printf '%n %i %D %p\n' | sort -n

where the format string shows %n the number of links, %i the inode number, %D the device the file is on, and %p the pathname. Lines with the same inode number and device are hard links. The device is only useful if you have mount points within the directory tree (as the same inode on a different device is not the same file). Of course, hard links to files outside the tree cannot be handled, even by rsync.

Improve this answer
1
  • I agree, that it is a good idea to convert hard links to symlinks in order to keep the size of the backup as small as possible when using a system that makes hard links into separate files. – sudodus Nov 5 '19 at 20:31
1

Backup of a directory tree in an ext4 file system

With an NTFS file system where you intend to write the backup, it may work better to backup the system into a compressed tar archive file. This way the ownership and permissions will be saved and can be restored when extracted into a linux file system. And the small files and sparse files should not occupy too much space.

If you still want to use rsync, you had better copy into a linux file system, e.g. ext4. This way it should work to copy hard links, small files and sparse files in a correct way and avoid that the backup size is bigger than the source.

The fact is that I do want to have an incremental backup of my home. Is there a technique to build tarballs incrementally (as rsync syncs only changed files)?

Yes, there are tools for incremental backup using tar

It is easiest to use such a tool, as described in the answer by meuh, but you can also look at the basics, for example at this link, where the crucial additonal option for incremental backup is

--listed-incremental=snapshot-file
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.