4

I have a fairly large directory that I need to sync to an external hard drive for both backup and portability reasons.

/work

I try to run the following rsync command in order to copy this folder to my external hard drive:

rsync -avz /work /media/extern_drive --max-size '4G'

Which seems to work fine, EXCEPT that it does not copy any file with a : in it.

This post gives a solution for a single file: rsync: colon in file names, but the problem is that I have so many of these files scattered in different directories that I cannot do it manually.

Is there any way to recursively rsync any files with a colon in the filename?

6
  • Why do your files use a colon? If it's not needed I would write a script to find all files containing a colon, then rename them, and then rsync.
    – eyoung100
    Commented Oct 13, 2014 at 13:59
  • 1
    They are named by their timestamp (astrophysical data), and unfortunately they cannot be renamed due to blackbox programs that assume their filename structure. Commented Oct 13, 2014 at 14:18
  • What filesystem does the external hard drive use? Is it FAT or some other filesystem that doesn't allow colons in file names, by any chance? Commented Oct 13, 2014 at 21:28
  • @ManilaThrilla - can you please let us know what the filesystem is of your /media/extern_drive? We cannot help further without this info.
    – slm
    Commented Oct 14, 2014 at 0:40
  • Yes, actually the problem was that the filesystem was FAT, an easy thing to overlook. I have chosen Gilles' answer as it addresses this, as well as suggests how to workaround this. However, I have not actually tried these solutions yet (away on a work trip), and went the route of creating an EXT4 partition to rsync to, which brings on a new set of problems as I use OS X on my laptop. Commented Oct 15, 2014 at 17:51

2 Answers 2

4

I surmise that your external drive uses a filesystem such as VFAT which doesn't allow colons in file names.

A simple option would be to back up your files as archives (zip, 7z, tar.xz, whatever catches your fancy). This way you wouldn't be limited by any characteristic of the filesystem other than the maximum file size.

Another possibility would be to use rdiff-backup, which takes care of translating file names that don't fit on the destination filesystem, as suggested by poolie.

A generic approach to unsupported characters is to leverage the filesystem layer to transform the file names. The FUSE filesystem posixovl transforms file names into names that Windows's VFAT supports.

mkdir ~/mnt
mount.posixovl -S /media/extern_drive ~/mnt
rsync -a /work ~/mnt
fusermount -u ~/mnt

See How can I substitute colons when I rsync on a USB key? for more details, and check that thread for any new solution that may come up.

2
  • Yes it ended up being because the destination was formatted as FAT. I am not quite sure if will use rdiff-backup (I do not really want to install software on my work computer) or just archive the directory (hundreds of GB). The posixovl option does not seem a good option for me (not on Ubuntu, and not really wanting to install software). Commented Oct 15, 2014 at 18:07
  • Hey, thanks! This was useful to me when I was rsyncing to mounted samba share.
    – domen
    Commented Jul 31, 2016 at 18:53
1

I would try explicitly including the pattern of files with colons in their names like this:

$ rsync -avz --include '*:*' /work /media/extern_drive --max-size '4G'

Examples

$ mkdir 1 2
$ touch 1/file{1..5}
$ touch 1/file:{1..5}
Explicitly excluding them
$ rsync -avz --exclude '*:*' 1/ 2/
sending incremental file list
./
file1
file2
file3
file4
file5

sent 313 bytes  received 114 bytes  854.00 bytes/sec
total size is 0  speedup is 0.00
Explicitly including them
$ rm -fr 2/*

$ rsync -avz --include '*:*' 1/ 2/
sending incremental file list
./
file1
file2
file3
file4
file5
file:1
file:2
file:3
file:4
file:5

sent 573 bytes  received 209 bytes  1,564.00 bytes/sec
total size is 0  speedup is 0.00
2
  • I don't understand what you hope to achieve. Manila doesn't exclude filenames containing colons in the first place, adding --include won't change anything. --include='*:*' is no more useful than --include='*a*'. Commented Oct 13, 2014 at 23:46
  • @Gilles - I know that rsync normally does not exclude colons and I suspect that he has some sort of configuration that's causing this. By using --include I was hoping that it would shed light on what was going on.
    – slm
    Commented Oct 13, 2014 at 23:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .