Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

So I did a recursive scp on my remote fileserver (in another state) and it created an infinite loop of links on my remote web directory...

http://www.linuxquestions.org/questions/linux-general-1/recursive-scp-w-o-following-links-658857/ says that I can can try creating a giant tar file. There is a problem with this though - I'm running the recursive scp on a Linux machine in my office, and I'm copying the files all to my external hard drive, which is in FAT32 format (because I need something that's readable by both UNIX and Windows). FAT32 doesn't support large filesizes. So I would have to try something different.

There's also a rsync option but the Linux machine in my office is very primitive (it's igel) so it doesn't have rsync...

share|improve this question

2 Answers 2

I would not recommend using scp for transferring large file trees directly, because it does not handle neither hard nor soft links properly, also the stream is not compressed.

I'd recommend cpio with (de)compression on the fly:

ssh user@host "cd /path/to/files && find . | cpio -ov | bzip2 -c" | bunzip2 -c | cpio -ivd

Also, find can handle additional conditions, like "files must be less than 4G"

find . -size -4G | ...

To make cpio more space-friendly (to handle spaces in file names properly) use

find . -print0 | cpio -0 -ivd | ...
share|improve this answer

You don't need to create the tar file before sending it. You can make it on the fly:

cd /source/dir
tar -cf - . | ssh 'cd /destination/directory && tar -xf -'

This does require that the remote server accept ssh shell connections, not just scp connections. In theory, it is possible to send arbitrary file trees to the remote side that don't correspond to anything like the local file structure, but I don't know of any existing tool to do this.

If symbolic links are a problem, you can make a copy of a directory tree that doesn't include symbolic links and that uses up negligible space, assuming the source tree is on a filesystem that supports hard links (so any native unix filesystem, or NTFS, but not FAT). I'm assuming GNU utilities here:

cd /source/dir
mkdir ../regular-files-only
cp -al . ../regular-files-only
find ../regular-file-only \! -type f \! -type d -delete

You could also use sftp. sftp -r does not follow symbolic links, unlike scp -r.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.