How to copy files from linux to windows using winscp from a folder which contains millions of files

Question

I need to copy files from a linux machine to a windows machine where the only ports which can be open are for SSH (22).
I can connect to the linux machine using WinSCP but the problem is once I try to navigate to the desired folder WinSCP gets stuck since the folder contains millions of records.
Basically I don't really care which files I copy and I would be glad to find a solution which enables me to just copy the latest 200 files.
Any ideas?
I've tried using ls -f | less but that did not do the trick.

Answer 1

If you can give up the WinSCP requirement and are willing to use Cygwin instead, this two-liner will do the trick:

$ ssh linuxbox 'ls -t1 /path/to/files | head -200' > files.list
$ rsync -ave ssh --files-from=files.list linuxbox:/path/to/files local/dest

This will work where WinSCP will not because it is not trying to build up a GUI picture of millions of files. It merely requests a list of the 200 most recently-touched files from the remote box, then transfers them to the local machine by name.

rsync and ssh aren't in the default Cygwin install. Be sure to select them when you get to the package selection screen in the Cygwin installer.

Answer 2

Just to make sure I understood your question the right way:
You are only able to sit in front of the Windows machine and not the Linux machine, right?

Though I'm not familiar with specialties of WinSCP, it should be possible to specify additional SCP/SSH options somewhere in the settings of WinSCP.

I guess, for listing the files contained in a directory, WinSCP is first connecting via ssh and then listing them with ls.
Make sure, the ssh connection uses compression. This can be done with the ssh parameter -C: ssh -C
In most cases, this decreases the volume to be transferred significantly and saved me a lot of trouble in the past.

Enabling compression for SCP is unfortunately a little tricky. scp has got the parameter -o, which defines a key value pair of the options defined by ssh_config.
In your case you can use Compression yes as the default is no.

However, compression reduced the bandwidth but increases the load on the processors on both sides (host and client). In case the millions of files you want to transfer are uncompressed plain text files, I would give the SCP compression a try. Are they binary files or already compressed archives, I would not use SCP compression.

Answer 3

When transferring a large number of files, I'd consider using tar or some other archiving tool to create one big file (which you could also compress) on the UNIX side of things, and then transfer that bigger file instead of the many smaller files. I'm not familiar enough with Windows to know what archiving software is available, but there ought to be something there which understands the tar format.