3

I have a fairly large directory that I need to sync to an external hard drive for both backup and portability reasons.

/work

I try to run the following rsync command in order to copy this folder to my external hard drive:

rsync -avz /work /media/extern_drive --max-size '4G'

Which seems to work fine, EXCEPT that it does not copy any file with a : in it.

This post gives a solution for a single file: rsync: colon in file names, but the problem is that I have so many of these files scattered in different directories that I cannot do it manually.

Is there any way to recursively rsync any files with a colon in the filename?

  • Why do your files use a colon? If it's not needed I would write a script to find all files containing a colon, then rename them, and then rsync. – eyoung100 Oct 13 '14 at 13:59
  • They are named by their timestamp (astrophysical data), and unfortunately they cannot be renamed due to blackbox programs that assume their filename structure. – Manila Thrilla Oct 13 '14 at 14:18
  • What filesystem does the external hard drive use? Is it FAT or some other filesystem that doesn't allow colons in file names, by any chance? – Gilles Oct 13 '14 at 21:28
  • @ManilaThrilla - can you please let us know what the filesystem is of your /media/extern_drive? We cannot help further without this info. – slm Oct 14 '14 at 0:40
  • Yes, actually the problem was that the filesystem was FAT, an easy thing to overlook. I have chosen Gilles' answer as it addresses this, as well as suggests how to workaround this. However, I have not actually tried these solutions yet (away on a work trip), and went the route of creating an EXT4 partition to rsync to, which brings on a new set of problems as I use OS X on my laptop. – Manila Thrilla Oct 15 '14 at 17:51
2

I surmise that your external drive uses a filesystem such as VFAT which doesn't allow colons in file names.

A simple option would be to back up your files as archives (zip, 7z, tar.xz, whatever catches your fancy). This way you wouldn't be limited by any characteristic of the filesystem other than the maximum file size.

Another possibility would be to use rdiff-backup, which takes care of translating file names that don't fit on the destination filesystem, as suggested by poolie.

A generic approach to unsupported characters is to leverage the filesystem layer to transform the file names. The FUSE filesystem posixovl transforms file names into names that Windows's VFAT supports.

mkdir ~/mnt
mount.posixovl -S /media/extern_drive ~/mnt
rsync -a /work ~/mnt
fusermount -u ~/mnt

See How can I substitute colons when I rsync on a USB key? for more details, and check that thread for any new solution that may come up.

  • Yes it ended up being because the destination was formatted as FAT. I am not quite sure if will use rdiff-backup (I do not really want to install software on my work computer) or just archive the directory (hundreds of GB). The posixovl option does not seem a good option for me (not on Ubuntu, and not really wanting to install software). – Manila Thrilla Oct 15 '14 at 18:07
  • Hey, thanks! This was useful to me when I was rsyncing to mounted samba share. – domen Jul 31 '16 at 18:53
1

I would try explicitly including the pattern of files with colons in their names like this:

$ rsync -avz --include '*:*' /work /media/extern_drive --max-size '4G'

Examples

$ mkdir 1 2
$ touch 1/file{1..5}
$ touch 1/file:{1..5}
Explicitly excluding them
$ rsync -avz --exclude '*:*' 1/ 2/
sending incremental file list
./
file1
file2
file3
file4
file5

sent 313 bytes  received 114 bytes  854.00 bytes/sec
total size is 0  speedup is 0.00
Explicitly including them
$ rm -fr 2/*

$ rsync -avz --include '*:*' 1/ 2/
sending incremental file list
./
file1
file2
file3
file4
file5
file:1
file:2
file:3
file:4
file:5

sent 573 bytes  received 209 bytes  1,564.00 bytes/sec
total size is 0  speedup is 0.00
  • I don't understand what you hope to achieve. Manila doesn't exclude filenames containing colons in the first place, adding --include won't change anything. --include='*:*' is no more useful than --include='*a*'. – Gilles Oct 13 '14 at 23:46
  • @Gilles - I know that rsync normally does not exclude colons and I suspect that he has some sort of configuration that's causing this. By using --include I was hoping that it would shed light on what was going on. – slm Oct 13 '14 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.