Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to copy files from a linux machine to a windows machine where the only ports which can be open are for SSH (22).
I can connect to the linux machine using WinSCP but the problem is once I try to navigate to the desired folder WinSCP gets stuck since the folder contains millions of records.
Basically I don't really care which files I copy and I would be glad to find a solution which enables me to just copy the latest 200 files.
Any ideas?
I've tried using ls -f | less but that did not do the trick.

share|improve this question

3 Answers 3

If you can give up the WinSCP requirement and are willing to use Cygwin instead, this two-liner will do the trick:

$ ssh linuxbox 'ls -t1 /path/to/files | head -200' > files.list
$ rsync -ave ssh --files-from=files.list linuxbox:/path/to/files local/dest

This will work where WinSCP will not because it is not trying to build up a GUI picture of millions of files. It merely requests a list of the 200 most recently-touched files from the remote box, then transfers them to the local machine by name.

rsync and ssh aren't in the default Cygwin install. Be sure to select them when you get to the package selection screen in the Cygwin installer.

share|improve this answer
    
Thanks for the answer but I actually think that it won't cut it. I connected via putty to the machine itself and ran the ls you suggested. Had to stop it after a few minutes. Same thing with find -mmin. I think my solution will need to be applicative by having the guys which output the file split them according to date so I only have a half a million per folder. Thanks. –  Ittai Sep 7 '11 at 8:18

Just to make sure I understood your question the right way:
You are only able to sit in front of the Windows machine and not the Linux machine, right?

Though I'm not familiar with specialties of WinSCP, it should be possible to specify additional SCP/SSH options somewhere in the settings of WinSCP.

I guess, for listing the files contained in a directory, WinSCP is first connecting via ssh and then listing them with ls.
Make sure, the ssh connection uses compression. This can be done with the ssh parameter -C: ssh -C
In most cases, this decreases the volume to be transferred significantly and saved me a lot of trouble in the past.

Enabling compression for SCP is unfortunately a little tricky. scp has got the parameter -o, which defines a key value pair of the options defined by ssh_config.
In your case you can use Compression yes as the default is no.

However, compression reduced the bandwidth but increases the load on the processors on both sides (host and client). In case the millions of files you want to transfer are uncompressed plain text files, I would give the SCP compression a try. Are they binary files or already compressed archives, I would not use SCP compression.

share|improve this answer
    
I can sit in front of the linux machine. What I'll probably do is write a small script which just takes the latest 200 files and copies them to a different folder. once that is run I can view it via winscp. –  Ittai Sep 6 '11 at 11:21
    
@Ittai: You definitely should look into the WinSCP settings. I'm pretty sure, that SSH compression is somehow possible to enable and not too hard to achieve. –  Torbjoern Sep 6 '11 at 11:51
1  
I think the problem is not with the SSH but with trying to ls millions of files. Even when you do ls -f | head 100 you still have to wait some time when on the machine. –  Ittai Sep 6 '11 at 12:28

When transferring a large number of files, I'd consider using tar or some other archiving tool to create one big file (which you could also compress) on the UNIX side of things, and then transfer that bigger file instead of the many smaller files. I'm not familiar enough with Windows to know what archiving software is available, but there ought to be something there which understands the tar format.

share|improve this answer
    
Thanks but I think tar would also take a considerable amount of time since I want to only take the last 200 modified files and that requires linux to sort the folder. –  Ittai Sep 7 '11 at 11:52
    
In that case, your suggestion to split the folder into many sub-folders (possibly in multiple levels) is probably better. Any folder with more than a thousand files or so will be unwieldy to work with on many systems (not sure about Linux). See also superuser.com/questions/291803/… –  Kusalananda Sep 7 '11 at 12:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.