5

I have multiple files to be pulled from remote server. For further processing of the files in the local server, I need to merge (concatenate) them into single file, which can't be done in the remote file though.

I am not sure how scp work internally, but for the best performance I believe instead of writing those files into local directory and then merge, I feel, I should merge them on the fly and then write into single file. Can you please let me know if merging (appending) the files on fly during scp from remote to local files possible?

If not any better idea?

3 Answers 3

6

Use SSH directly instead of scp and run cat. Where you would do:

scp remote:{file1,file2...} local-dir

Instead do:

ssh remote cat file1 file2 ... > locale-file
6
  • Thanks Muru, How does this work internally? Does it first create files on local and then executes cat or directly cat before writing in local?
    – Betta
    Dec 15, 2017 at 12:11
  • The local-file will be created before SSH is started, after that the output of cat should be dumped directly into it.
    – muru
    Dec 15, 2017 at 12:13
  • so we are doing ssh and cat. I feel this is equivalent to ssh and cp, not as secure as scp. Am I wrong?
    – Betta
    Dec 16, 2017 at 6:12
  • @Betta why do you imagine ssh and cat is not as secure as scp?
    – muru
    Dec 16, 2017 at 6:16
  • i know that both scp and ssh>cat uses secure channel supported by ssh, but in that case what is the significance of scp? shh>cp should have been sufficient? Pardon my ignorance!
    – Betta
    Dec 16, 2017 at 6:30
1

This is silly, but it seems you can actually do this with just scp, by copying the remote files to a local fifo and piping them out of it:

$ mkfifo p
$ while :; do cat p >> output ; done  &
$ scp somehost:test/\* p
bar       100%    4    10.9KB/s   00:00    
doo       100%    4     8.6KB/s   00:00    
foo       100%    4    13.6KB/s   00:00  
$ kill %1
# output contains the files concatenated

(tested with OpenSSH 7.4p1-10+deb9u2 on Debian)

0

Below steps to performed being in local server only

I tested both steps it worked fine

First step


ssh username@remoteserverip "cat file1 file2 file3 >> /remoteserverpath/Mergedfile"

Second step You are copying merged file from remote server to local server You can do this by rsync or scp


I prefer rsync

rsync -avzh username@remoteserverip:/remoteserverpath/Mergedfile localserverpath_where_you_want_to_save

2
  • I said, I can't merge on the remote server! :)
    – Betta
    Dec 16, 2017 at 6:32
  • I achieved to copy the single file content over rysnc in localserver. i am trying to do for multiple files too below is command used to copy the single file content of remote server in local server rsync -avzh username@remoteserverip:/path/remoteserverfile /localserverpath/Mergedfile Merged file contains remoteserverfile content Dec 18, 2017 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.