Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I mounted Box.com cloud storage using davfs according to these instructions. I mounted my Box.com account under /home/me/Cloud/Box

I can access the mounted filesystem via Dolphin as well as via the terminal. It is a little slow, but I can list (ls) and navigate the entire directory structure with no errors.

Next I attempted to run rsync as follows:

rsync -auvz  --max-size=250M --exclude '.*' /home/me/Music/ /home/me/Cloud/Box/Music

I also tried:

rsync -auv  /home/me/Music/A /home/me/Cloud/Box/Music

and other variations of rsync commands. The command is copying my Music from my local file system (/home/me/Music/) to the Box cloud (/home/me/Cloud/Box/Music) via the davfs mount.

I always get a lot of errors of this form:

rsync: mkstemp <filename> failed: Invalid argument (22)

A specific example is:

rsync: mkstemp "/home/me/Cloud/Box/Music/VariousArtists/.01_Track_1.mp3.YVmFI9" failed: Invalid argument (22)

This is all happening on Kubuntu 12.04 LTS 64 bit, server grade hardware, with a fast/reliable cable modem connection (12 Mb/s upload speeds).

share|improve this question

2 Answers 2

up vote 3 down vote accepted

The problem occurs because of rsync making temporary files with filenames that the box.com and/or davfs do not understand. Thus the file .01_Track_1.mp3.YVmFI9 does not exist on your system, but is a temporary artefact of rsync. Some guesswork from my side: if you don't get the error on all files, you probably only get the errors on files that were already uploaded (and changed).

It used to be impossible to switch off this temporary file generation, but you might nowadays have more luck by adding the option --in-place. However the advantages of using rsync if you are not talking to a rsync-daemon (which you are not if you are using davfs), are unclear to me.

Therefore, as an alternative, you could try cp --update, which only copies a file when the source is newer than the destination. New files and any files with changes in ID3 tags will get copied, others will not.

Or if you need to have more control use find:

cd /home/me/Music/
find * -size -250M -print0 | cpio -pdmv0 /home/me/Cloud/Box/Music

this preserves the hierarchy structure and cpio does not overwrite existing files that are not older.

share|improve this answer
    
Very informative. Thanks. I will test this momentarily. –  MountainX Jun 11 '13 at 4:28
    
One reason for using rsync is that I already know how to limit it to copying files less than 250MB and files or directories that are not hidden. I use --max-size=250M --exclude '.*' . I'm sure cp can be made to do this... maybe by piping the output of find into cp? But I don't know how to do that yet. If we come up with a solution, I'll try cp -ru. Thanks –  MountainX Jun 11 '13 at 4:39
    
In that case you don't need cp -u at all, use cpio (see updated answer) –  Anthon Jun 11 '13 at 4:53
    
Thank you. I believe this will solve it for me. You educated me in the process, which I appreciate. (FYI, in my case I want files less than 250MB, not greater than.) –  MountainX Jun 11 '13 at 5:20
    
That is what ... -size -250M does, otherwise you would use +250M (there was briefly a wrong version here, but I realised it before the edit was fixed) –  Anthon Jun 11 '13 at 5:25

1. Issues with special characters in filenames

Are there any special characters in the filenames? Depending on the filesystem you're writing these files to, they may not allow you to prefix files with a dot (.) for example.

2. Problems with rsync modification times and webdav2

I came across this blog post where an issue is described with rsync having a problem writing/tracking file modification times across webdav2 mounted box.com directories.

The problem shows up like this in the mounted file system:

david@sydney:~/Pictures$ ls -l /mnt/box/bwca/08/09/IMG_3084.CR2
-rw-r--r-- 1 david david 12564061 Aug 14 16:08 /mnt/box/bwca/08/09/IMG_3084.CR2
david@sydney:~/Pictures$ ls -l 2012/08/09/IMG_3084.CR2
-rw-rw-r-- 1 david david 12564061 Aug  9 13:00 2012/08/09/IMG_3084.CR2

That same article showed a workaround:

$ rsync -avhP --size-only --bwlimit=64 2012/08 /mnt/box/bwca/

This is an OK way to use rsync, but it's only comparing files based on their size now, not their checksums.

3. Issues with davfs2 (WebDAV)

I came across this thread titled: rsync via davfs2? in the WebDAV (davfs) forum over on sourceforge. Someone was inquiring about a similar situation where they wanted to use WebDAV to mount an online storage provider and perform rsync's to the mounted storage via WebDAV. This is what one of the developers (Werner Baumann) of WebDAV had to say about this topic.

excerpt of Werner's response

  • davfs2 will only upload complete files. It can not do the incremental stuff rsync usually does, and that makes rsync very efficient.

  • davfs2 uses a local cache on disk. This will make it more responsive and your application should profit from this too. But it needs local disk space for this. You should allow for a large cache size, so rsync can do most of its work with the local cache, and davfs2 will upload most of the files in the background, when rsync has already finished.

Werner goes on to suggest the following

This could be a disadvantage in this case. When rsync reads a file on the remote host, it must be transfered by davfs2 into the local cache first (if it is not already there). This could make the process really and unnecessary slow. As rsync only works as a sophisticated copy-program in your case, it might be better to use cp instead. cp has an options (-u) to copy only files that are newer than the ones in the davfs2 file system (= smartdrive) and it would not need to read the files, but only reads file meta data like mtime.

A command like "cp -pru directory/to/backup dav/" might do the job. It shold not download files (like rsync might do, but I am not sure) (please look at the manuals of cp and rsync).

Options?

So as @Anthon has suggested, you can use the cp -u method to copy the files. Realizing that this method solely looks at a file's size as a factor in comparison, so it's not completely reliable.

You should not use anything that only looks at modification times when comparing files, cp -pru. Werner explains why in this thread:

excerpt on issue with mod times

When you unmount a davfs2 file system and mount it again at some later time, file times may have changed according to the time information from the server. Tools like cp -pu and rsync can not rely on these times to determine what files have changed.

So given the various issues surrounding modification times an approach using purely checksums seems like a better fit:

$ rsync -avvz --omit-dir-times --checksum --human-readable --progress <local dir> <remote dir>
share|improve this answer
    
Hi. There are no special characters in the file names. My filename on disk is simply "01_Track_1.mp3". The dot prefix is not part of the normal filename, nor is the extra extension (e.g., .mp3.YVmFI9). Also note that not every file fails. The failures don't seem to have a pattern. –  MountainX Jun 10 '13 at 20:45
    
is this enough info? got file_sum recv_files(Some_Music_[1999]/Some_Music_[1999]Zuni_SongVarious_Artists24.mp3) rsync: mkstemp "/home/me/Cloud/Box/Music/Some_Music_[1999]/.Some_Music_[1999]Zuni_SongVarious_A‌​rtists24.mp3.1aWWRY" failed: Invalid argument (22) got file_sum –  MountainX Jun 11 '13 at 0:46
    
another example: got file_sum recv_files(Carlos/Dance/05-Tapestry-Carlos.mp3) rsync: mkstemp "/home/me/Cloud/Box/Music/Carlos/Dance/.05-Tapestry-Carlos.mp3.WeGbGS" failed: Invalid argument (22) got file_sum recv_files(Carlos/Dance/06-Dance-Turquoise_Dance-Carlos.mp3) –  MountainX Jun 11 '13 at 0:48
    
@MountainX - see if my update is your issue. Also what version of rsync are you using? rsync --version. –  slm Jun 11 '13 at 1:50
    
rsync version 3.0.9 protocol version 30 and I'm testing your updated solution now. Thanks! –  MountainX Jun 11 '13 at 4:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.