14

I want to tar and send a ~700GiB directory to a remote drive I don't control. I don't have the HDD space locally to create the tarball and then copy that over. The remote is also protected by rssh - you cannot ssh into the server and run arbitrary commands.

However, I can use scp, rsync, or sftp. So, my question is specific to one of those 3 commands. Is there a way for scp, rsync, or sftp to copy a pipe, without being able to run arbitrary commands on the other side?


Update: The below solutions would not have worked for us at our secondary storage. We are restricted to running certain commands. We cannot ssh - commands that use ssh are filtered/controlled with rssh.

In the end, we canceled all our secondary accounts, and paid a lot of money to double our primary storage size (our primary host could do this without migration, which was cool). We then created a temporary archive and sent it as per normal. (Which we could not do with our initial amount of storage.)

In the future, we will be using a secondary VPS with its own dedicated larger storage, so that this does not occur again. It's clear that what I was asking for is not possible.

1
  • I've answered this over at the sister site -- in short, lftp is capable of doing this over the SFTP protocol.
    – Bob
    Sep 22, 2017 at 8:07

2 Answers 2

6

It is not exactly "pipe", but you can basically tell scp to copy specific FD (which can be pipe) from your host to the other. Simple bash command like this:

(scp does not work as it needs a size in advance):

scp <(tar cz files to compress) host:/path/to/new.file

but it can work with pure ssh:

tar cz files to compress | ssh host "cat > /path/to/new.file"
3
  • 2
    Did you try this? It doesn't work for me (OpenSSH scp 6.6). The scp protocol requires sending the filesize ahead of the contents, which isn't suited to sending a stream of an unknown length.
    – Kenster
    Jun 15, 2016 at 17:48
  • @Kenster Yes, you are right. scp will not work. But it should work with pure ssh. See edited answer. I didn't try that, but the conceptually it should do the job.
    – Jakuje
    Jun 15, 2016 at 17:54
  • 1
    @Jakuje but the point of this question is that rssh prevents invoking cat, it has to go through one of the three mentioned commands.
    – b0fh
    Nov 15, 2016 at 9:09
0

RSYNC behavior over SFTP

Without even compressing anything, you can replicate the directory structure to the other side using LFTP (name is misleading) and the mirror subsystem of LFTP. LFTP supports SFTP and the mirror subsystem supports nearly all the capabilities of rsync. It can also split up the transfer into as many connections as you want, to make it go faster. I do this all the time.

Example

Also check out this post: https://serverfault.com/questions/135618/is-it-possible-to-use-rsync-over-sftp-without-an-ssh-shell

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .