Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

So I'm trying to transfer a bunch of files via SCP. Some of these are too large to be stored on the recipient (Android phone, 4GB file size limit).

The sender is almost out of space, so I can't create intermediate files locally.

I'd like to tar up the bunch and stream it through split so that I can get smaller segments that'll be accepted by the phone, i.e. local command:

tar -cvf - ~/batch/ | split --bytes=1024m - batch.tar.seg

But I'm not sure how I'd pipe that into scp to get it the phone. According the comment on this post, it's possible, but I first of all don't quite get what he's saying, second of all I'm not sure how to accomplish this as there'll be multiple files output from split.

Any ideas?

share|improve this question
add comment

3 Answers

up vote 4 down vote accepted

If the recipient has split, then you can do:

tar -cvf - ~/batch/ | gzip |
  ssh recipient 'cd /destination &&
    split --bytes=1024m - batch.tar.gz.seg'
share|improve this answer
    
It does! And it works! What a great way to solve it :) Thank you! –  DanielSmedegaardBuus Feb 8 '13 at 8:27
add comment

While the comment you are mentioning does refer to scp and ssh reading from the STDIN, this is not exactly your case here. The reason is the split: it does not write to STDOUT. So even if you add more pipes, you will not get the data there. Stephane suggests the right thing in his answer above: don't store anything on the source as it has lack of free space, just package, gzip and transfer the data; and then do the splitting on the receiving end, where the chunk size matters.

share|improve this answer
    
Thank you for that extra info - I hadn't thought of that, of course you're right :) –  DanielSmedegaardBuus Feb 8 '13 at 8:28
    
You're welcome, DanielSmedegaardBuus! –  Leonid Feb 8 '13 at 10:11
add comment

Your referenced post describes the problem well: As you are using a pipe you do not copy files but rather redirect standard streams. And scp is not the right tool for that (it is for copying files), ssh is the right program: You redirect the stream to ssh standard input and this will output that input onto the remote host and feed it to the standard input of a specified command:

local$ tar -cvzf - ~/batch/ | ssh target 'split --bytes=1024m - batch.tar.seg'

Note that I added the compression flag 'z' to the tar options, maybe then splitting is not needed at all.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.