1

I'm trying to concatenate multiple input files/streams into one stream (using the imaginary command stream-cat), pipe that stream into ssh and on the remote host separate it back into individual files/streams (stream-sep), like in this example, which is for demonstration purposes only:

stream-cat <( zfs send tank/vm@snapshot ) somefile.txt | ssh user@host "stream-sep >( zfs receive tank/vm@snapshot ) somefile.txt"

Explanation of the example: zfs send outputs a large stream of data whose size isn't known in advance (that's why tar can't handle it). That data stream is concatenated with the content of a regular file somefile.txt. The resulting stream is piped into ssh, where it is separated again. The first stream gets piped into zfs receive, while the second is written to a regular file.

Such a program should be straightforward to implement by reading non-seekable streams in chunks and always writing the chunk size followed by the data, until the end of the stream is reached. The overhead would be minimal.

Does such a program exist already?

6
  • Why would you put a zfs send stream into a tar archive? That makes no sense, tar archives are for files! I think you probably have a good intention there, but I don't quite get it. Couldn't you just have two ssh connections? Jun 9, 2023 at 23:06
  • Well, it's an example. Yes, I could transfer the files individually using multiple SSH connections, but it seemed "inelegant". Jun 9, 2023 at 23:12
  • 2
    But the thing is that ssh is exactly that multiplex you're looking for and can share one control master connection. That's exactly what you ask for - kind of the opposite of inelegant :) Jun 10, 2023 at 0:12
  • I didn't know about that feature of SSH! Thanks for pointing it out. I'm not sure yet how to use it in an automated way, though. Jun 10, 2023 at 4:55
  • 1
    There's really nothing special to do, just for example use ssh to send that ZFS stream whilst using another ssh to copy that file; e.g. zfs send | zstd -10 | ssh user@host sh -c 'zstd -d | zfs receive tank/vm@snapshot' & cat somefile.txt | ssh user@host sh -c 'cat > somefile.txt'. Jun 10, 2023 at 7:46

2 Answers 2

2

What you're describing is multiplexing; something that requires a protocol (i.e., a formal specification on how to deal with data).

There's a lot of approaches to that. For example, you'll notice that your computer can perfectly well download multiple files at the same time via HTTP – even from the same server. That feature is foremost brought to you by TCP, which, as the transport protocol, allows different streams to be sent and "taken apart" at the receiving end.

So, TCP already offers that functionality, and you can simply start two concurrent SSH connections and make use of it!

zfs send | zstd -10 | ssh user@host 'zstd -d | zfs receive tank/vm@snapshot' &
# ^         ^    ^     ^             ^     ^   ^                             ^
# |         |    |     |             |     |   |                             |
# |         |    |     |             |     |   |     Tell your own shell to run
# |         |    |     |             |     |   |     this in the background and
# |         |    |     |             |     |   |     not block
# |         |    |     |             |     |   |                         
# |         |    |     |             |     |   |                            
# |         |    |     |             |     |   \---- Program to receive in the end
# |         |    |     |             |     |
# |         |    |     |             \-----+-------- use zstd to decompress
# |         |    |     |                             received data
# |         |    |     |            
# |         |    |     |
# |         |    |     \---------------------------- our first ssh invocaton
# |         |    |
# |         \----+---------------------------------- use zstd to compress at medium
# |                                                  high compression level (10)
# |
# \------------------------------------------------- the first program whose output
#                                                    we send

cat somefile.txt | ssh user@host 'cat > somefile.txt'
#                                                    Second SSH connection

Of course, instead of the slightly inelegant cat somefile.txt | ssh … > somefile.txt, you would probably just use scp somefile.txt user@host:somefile.txt (which uses SSH under the hood, but doesn't do a shell connection, but uses the built-in SCP layer in SSH to copy the file).

You can make the connection establishment for the second connection faster by adding the following to your ~/.ssh/config file:

ControlMaster auto
ControlPath /tmp/.ssh-socket-%h_%p_%r

This will tell SSH to try and re-use the one SSH session to send multiple encrypted streams at the same time (this works with any combination of scp and ssh as well).

2
  • Except in very recent versions of openssh where scp has switched to using sftp, scp does do a "shell connection" (invokes the login shell of the remote user to run a scp command in receiving mode here for upload). There is no "scp subsystem" like there is a "sftp subsystem". Jun 10, 2023 at 13:47
  • Having those control sockets in a word-writable area such as /tmp sounds like a very bad idea. I would put them in ~/.ssh or anywhere in your $HOME or /run/user directory. Jun 10, 2023 at 13:52
2

Crude implementations of those stream-cat, stream-sep could easily be written in perl:

stream-cat() {
  perl -ne 'BEGIN{$/ = \0x7fff}
            print pack("n", $c|length()<<1), $_;
            $c = !eof' -- "$@"
}
stream-sep() {
  perl -e 'while($/ = \2, $_ = <STDIN>) {
             $n = unpack "n";
             open OUT, shift@ARGV unless $n & 1;
             if ($n>>=1) {$/ = \$n; $_ = <STDIN>; print OUT}
           }' -- "$@"
}

or same as #! /bin/sh - scripts instead of sh functions.

(error handling left as an exercise to the reader :-).

stream-cat sends records up to 32767 byte large prefixed by a network-encoded (big-endian) short whose lowest bit indicates whether it's the start of a new stream (0) or the continuation, and the remaining bits are the size.

And then, for example:

$ cat a
test
$ stream-cat 'seq 10|' a | stream-sep '|wc -l' '>b'
10
$ cat b
test

So in your case:

stream-cat 'zfs send tank/vm@snapshot|' somefile.txt |
  ssh user@host 'stream-sep "|zfs receive tank/vm@snapshot" ">somefile.txt"'

A rare case where the unsafe form of open (here also used by <> as used by -n), which allows <file, >file to open a files in readonly or write-only mode or |cmd, cmd| to pipe to/from a command is actually useful.

Using those |cmd / cmd| scales better than your <(cmd) / >(cmd) as only one is open at a time, so you could send thousands of separate streams without issues.

2
  • Regarding your comment about |cmd and cmd|: This is something from Perl, not from Bash, right? Jun 11, 2023 at 16:38
  • @DavidScherfgen, yes. See perldoc -f open for details. Jun 11, 2023 at 16:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .