2

I have multiple files to be pulled from remote server. For further processing of the files in the local server, I need to merge (concatenate) them into single file, which can't be done in the remote file though.

I am not sure how scp work internally, but for the best performance I believe instead of writing those files into local directory and then merge, I feel, I should merge them on the fly and then write into single file. Can you please let me know if merging (appending) the files on fly during scp from remote to local files possible?

If not any better idea?

5

Use SSH directly instead of scp and run cat. Where you would do:

scp remote:{file1,file2...} local-dir

Instead do:

ssh remote cat file1 file2 ... > locale-file
  • Thanks Muru, How does this work internally? Does it first create files on local and then executes cat or directly cat before writing in local? – Betta Dec 15 '17 at 12:11
  • The local-file will be created before SSH is started, after that the output of cat should be dumped directly into it. – muru Dec 15 '17 at 12:13
  • so we are doing ssh and cat. I feel this is equivalent to ssh and cp, not as secure as scp. Am I wrong? – Betta Dec 16 '17 at 6:12
  • @Betta why do you imagine ssh and cat is not as secure as scp? – muru Dec 16 '17 at 6:16
  • i know that both scp and ssh>cat uses secure channel supported by ssh, but in that case what is the significance of scp? shh>cp should have been sufficient? Pardon my ignorance! – Betta Dec 16 '17 at 6:30
0

Below steps to performed being in local server only

I tested both steps it worked fine

First step


ssh username@remoteserverip "cat file1 file2 file3 >> /remoteserverpath/Mergedfile"

Second step You are copying merged file from remote server to local server You can do this by rsync or scp


I prefer rsync

rsync -avzh username@remoteserverip:/remoteserverpath/Mergedfile localserverpath_where_you_want_to_save

  • I said, I can't merge on the remote server! :) – Betta Dec 16 '17 at 6:32
  • I achieved to copy the single file content over rysnc in localserver. i am trying to do for multiple files too below is command used to copy the single file content of remote server in local server rsync -avzh username@remoteserverip:/path/remoteserverfile /localserverpath/Mergedfile Merged file contains remoteserverfile content – Praveen Kumar BS Dec 18 '17 at 18:22
0

This is silly, but it seems you can actually do this with just scp, by copying the remote files to a local fifo and piping them out of it:

$ mkfifo p
$ while :; do cat p >> output ; done  &
$ scp somehost:test/\* p
bar       100%    4    10.9KB/s   00:00    
doo       100%    4     8.6KB/s   00:00    
foo       100%    4    13.6KB/s   00:00  
$ kill %1
# output contains the files concatenated

(tested with OpenSSH 7.4p1-10+deb9u2 on Debian)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.