Recursive scp without following links or creating a giant tar file?

Question

So I did a recursive scp on my remote fileserver (in another state) and it created an infinite loop of links on my remote web directory...

http://www.linuxquestions.org/questions/linux-general-1/recursive-scp-w-o-following-links-658857/ says that I can can try creating a giant tar file. There is a problem with this though - I'm running the recursive scp on a Linux machine in my office, and I'm copying the files all to my external hard drive, which is in FAT32 format (because I need something that's readable by both UNIX and Windows). FAT32 doesn't support large filesizes. So I would have to try something different.

There's also a rsync option but the Linux machine in my office is very primitive (it's igel) so it doesn't have rsync...

Answer 1

I would not recommend using scp for transferring large file trees directly, because it does not handle neither hard nor soft links properly, also the stream is not compressed.

I'd recommend cpio with (de)compression on the fly:

ssh user@host "cd /path/to/files && find . | cpio -ov | bzip2 -c" | bunzip2 -c | cpio -ivd

Also, find can handle additional conditions, like "files must be less than 4G"

find . -size -4G | ...

To make cpio more space-friendly (to handle spaces in file names properly) use

find . -print0 | cpio -0 -ivd | ...

Answer 2

Another option is to use sftp -r as sftp manual says:

Recursively copy entire directories when uploading and downloading. Note that sftp does not follow symbolic links encountered in the tree traversal.

So

sftp -r source@surce_server.com:/source/dir .

does exactly what you want to do.

Answer 3

You don't need to create the tar file before sending it. You can make it on the fly:

cd /source/dir
tar -cf - . | ssh 'cd /destination/directory && tar -xf -'

This does require that the remote server accept ssh shell connections, not just scp connections. In theory, it is possible to send arbitrary file trees to the remote side that don't correspond to anything like the local file structure, but I don't know of any existing tool to do this.

If symbolic links are a problem, you can make a copy of a directory tree that doesn't include symbolic links and that uses up negligible space, assuming the source tree is on a filesystem that supports hard links (so any native unix filesystem, or NTFS, but not FAT). I'm assuming GNU utilities here:

cd /source/dir
mkdir ../regular-files-only
cp -al . ../regular-files-only
find ../regular-file-only \! -type f \! -type d -delete

You could also use sftp. sftp -r does not follow symbolic links, unlike scp -r.