Quick version (TL;DR)
Update: For those that just want a quick-and-dirty with minimal reading.
Know:
- This actually had nothing to do with Termux - it affects any Linux using OpenSSH 8.8+. I also ran into it on Fedora 35 with OpenSSH 8.7p1
- While
sftp will work in a pinch, it handles copying of symlinks differently than scp so don't just throw it in a script as a replacement. Better to use rsync.
Recursively copy a directory from remote to local:
# deprecated / insecure (see CVE link below) but works
scp -O -rp -P <port> "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"
# secure / recommended by openssh - also works
rsync -saLPz --port <port> -e ssh "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"
Recursively copy a directory from local to remote:
# deprecated / insecure (see CVE link below) but works
scp -O -rp -P <port> "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"
# secure / recommended by openssh - also works
rsync -saLPz --port <port> -e ssh "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"
This works fine for multiple paths too, e.g.
# copy multiple paths
rsync -saLPz --port <port> -e ssh "<SRC_PATH_1>" "<SRC_PATH_2>" "<SRC_PATH_3>" "<user>@<host_ip>:<DST_PATH>"
# copy multiple paths from array
arrPaths=("<SRC_PATH_1>" "<SRC_PATH_2>" "<SRC_PATH_3>");
rsync -saLPz -e ssh "${arrPaths[@]}" "<user>@<host_ip>:<DST_PATH>"
Root Cause
After a lot of hunting online, I finally found an answer here - credit / thanks to manjaro forums users zbe and canyue980 for finding the initial fix and cluing me into what was actually going on.
To quote canyue980:
Thanks for all guys that offer helps! I just found the solution in upper link:
Note: Since OpenSSH 8.8 the scp utility uses the SFTP protocol by default. The -O option must be used to use the legacy SCP protocol.
by using -O option and it works！
scp -O file_name target_dir
and of course you can also add an alias for it in~/.zshrc like this:
alias scp="scp -O"
Indeed, changed my command from:
scp -rp -P 1234 "[email protected]:/home/desktop-user/Pictures/test.jpg" .
To:
scp -rpO -P 1234 "[email protected]:/home/desktop-user/Pictures/test.jpg" .
and it worked like a champ. Will leave this up in hopes it helps someone else.
Security Implications of -O
Security Implications of using -O (that's "Oh" as in OLD - not zero): according to the release notes for openssh 8.8, it sounds like the main risk is that:
Legacy scp/rcp performs wildcard expansion of remote filenames (e.g.
"scp host:* .") through the remote shell. This has the side effect of
requiring double quoting of shell meta-characters in file names
included on scp(1) command-lines, otherwise they could be interpreted
as shell commands on the remote side.
I don't personally see this as a big risk for people used to the old command and running manually or via trusted scripts on home computers. But your risk-assessment / use-cases may be different - especially for business systems or setups that are using unproved / untrusted scripts. This appears to be the case for some as RHEL 9 is deprecating the scp protocol, citing that:
We're making this change because the SCP protocol is decades old, and carries multiple security risks and issues that have no straightforward solutions. New issues are being reported frequently (CVE-2020-15778 is the most recent as of this writing, but we can’t be sure it will be the last) and it is rather difficult to fix them all properly because the protocol is inherently trustworthy of authenticated sessions.
From the above release notes and links, as well as a note on the arch wiki, it seems the intention is to completely deprecate SCP (e.g. the protocol) in favor of alternatives like sftp and rsync (see below). I am unclear of what the intended future of the scp command is, especially in the short-term, but it seems plausible or even likely that it may also get deprecated in the future.
Recommended replacements
To use sftp and rsync as drop-in replacements in scripts, probably rsync is the easier drop-in replacement as it doesn't require any changes on the host whereas at least as of Fedora 35 sftp does.
The commands below should work for sftp as long as you can connect with sftp <user>@<host_ip> just like you would with ssh (connecting this way will put you in the sftp "shell". To exit, use exit). Otherwise, you may have to enable it on the host. For Fedora 35, that process was as follows:
sudo cp -a /etc/ssh/sshd_config /etc/ssh/sshd_config.$(date +'%F_%T').baksudo sed -Ei 's~^(Subsystem[ \t]+sftp[ \t]+)(/usr/lib/openssh/sftp-server)~#\1\2\n\1internal-sftp~g' /etc/ssh/sshd_configsudo systemctl restart sshd
when done you should see something like this (e.g. the sftp server line should be commented out and a new line with internal-sftp add to take its place) :
sudo grep Subsystem /etc/ssh/sshd_config
#Subsystem sftp /usr/lib/openssh/sftp-server
Subsystem sftp internal-sftp
You can also restrict this using groups and chroot if you need increased security, as shown here and here.
Note: scp by default copies the contents of symlinks instead of them as a symlink. Rsync lets you control this behavior by using -l to copy as symlink or -L to copy what is referenced by the symlink. The -a (archive) option will default to -l, unless -L is also specified. sftp will copy the symlink as a link (e.g. the contents pointed to are not copied) and does not provide an option for controlling this behavior as of time of writing. You can verify this by looking at the documentation for the -r option under man scp and man sftp respectively.
Here are a few examples of roughly equivalent commands, which should all be going over SSH and picking up your ~/.ssh/config settings and identities:
Recursively copy a directory from remote to local:
scp -O -rp -P <port> "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"
rsync -saLPz --port <port> -e ssh "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"
Recursively copy a directory from local to remote:
scp -O -rp -P <port> "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"
rsync -saLPz --port <port> -e ssh "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"
Corrections for sftp syntax
Edit: 2022-Sep-13: Noticed today while retesting that the commands I had for sftp were giving errors, so I am moving them out from the above and adding some disclaimers. I don't personally recommend sftp as a replacement for scp anyway due to the differences I mentioned above in how they handle copying when symlinks are involved.
I was specifically interested in using this as a replacement in scripts. If you are only needing interactive mode, don't mind making a "batch" instruction file for sftp, or have rsync installed on all of the machines you'll be using then you probably won't need to go this far. My notes below are partially based this answer, with some additional testing done by me.
My biggest complaint here (other than sftp using different syntax for upload vs download) is the very poor documentation. Looking at man sftp, the syntax isn't documented at all (meaning that while the tool can be used non-interactively, it is only documented for use in batch and interactive modes, and stepping outside those use-cases, while supported, is not documented). The man page gives NO examples of common usage or even a website for more detailed documentation as many of the GNU tools do. And sftp -h provides less than the man page. Literally, if it weren't for search engines and sites like this and I would have completely given up on sftp as an option altogether.
1. Local pull (e.g. from local terminal, download files from remote to local)
This is mostly straight-forward and works somewhat similarly to scp other than the differences noted above (e.g. symlinks) and slightly different syntax. And that one of the variants will result in an interactive prompt.
# Download a single file with same name to current dir
sftp -pC -P <port> "<user>@<host_ip>:<SRC_PATH>"
# or
sftp -pC -P <port> "<user>@<host_ip>:<SRC_PATH>" .
# Download a single file to name and path in "<DST_PATH>"
sftp -pC -P <port> "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"
# Download a folder and all of its contents,
# keep the same name as the remote and save to current dir
# note that any symlinks will copy only the symlink itself
# and NOT the data the link references
sftp -rpC -P <port> "<user>@<host_ip>:<SRC_PATH>" .
# Now if you omit "<DST_PATH>" entirely like you can with
# the single-file version (first command in this code-block)
# you instead drop to an interactive prompt and have to type
# type 'exit' to get back to your bash prompt
# e.g. AVOID THIS IN NON-INTERACTIVE SCRIPTS!!!
sftp -rpC -P <port> "<user>@<host_ip>:<SRC_PATH>"
Connected to <host_ip>.
Changing to: <SRC_PATH>
sftp>
sftp> exit
# Download a folder and all of its contents,
# to name and path in "<DST_PATH>"
# note that any symlinks will copy only the symlink itself
# and NOT the data the link references
sftp -rpC -P <port> "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"
2. Local push (e.g. from local terminal, upload files from local to remote)
Firstly, if you just try to reverse the parameters, like what my original answer had, then you will get a connection error. Why doesn't sftp make the upload syntax consistent with the download syntax? Your guess is as good as mine... I for one would have been a huge fan of that approach. But if you try it, here's what happens (note: this output is from 2 machines running Fedora 35 with correctly configured ssh where scp and rsync ... -e ssh work just fine and do not prompt for authentication).
# single file
sftp -pC -P <port> test.txt "<user>@<host_ip>:<DST_PATH>"
ssh: Could not resolve hostname test.txt: Name or service not known
Connection closed.
Connection closed
# dir
sftp -rpC -P <port> "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"
ssh: Could not resolve <SRC_PATH>: Name or service not known
Connection closed.
Connection closed
Per the answer I linked to above, the correct way to do this is instead as follows
# push a single file from local to remote
sftp -pC -P <port> "<user>@<host_ip>:<DST_PATH>" <<< 'put test.txt'
# or
echo 'put test.txt' | sftp -pC -P <port> "<user>@<host_ip>:<DST_PATH>"
# push a entire dir (and its contents) from local to remote
# notice the inner single quotes to escape <SRC_PATH>
# for enclosing paths containing whitespace
sftp -rpC -P <port> "<user>@<host_ip>:<DST_PATH>" <<< "put '<SRC_PATH>'"
# or
echo "put '<SRC_PATH>'" | sftp -rpC -P <port> "<user>@<host_ip>:<DST_PATH>"
# or
printf 'put "%s" "%s"\n' "<SRC_PATH>" "<DST_PATH>" | \
sftp -rpC -P <port> "<user>@<host_ip>"
I found in my scripting that using a similar syntax for downloads, e.g.
# pull a entire dir (and its contents) from remote to local
# notice the inner single quotes to escape <SRC_PATH>
# for enclosing paths containing whitespace
printf 'get "%s" "%s"\n' "<SRC_PATH>" "<DST_PATH>" | \
sftp -rpC -P <port> "<user>@<host_ip>"
was less confusing but all of the above should work.
To copy multiple paths at once, an array works (or you could create a batch file listing each put command on a separate line and pass that to sftp using the -b <batchfile> option).
# copy multiple paths from array
arrPaths=("<SRC_PATH_1>" "<SRC_PATH_2>" "<SRC_PATH_3>");
printf 'put %s\n' "${arrPaths[@]}" | sftp -rpC -P <port> "<user>@<host_ip>:<DST_PATH>"
2022-Oct-10 Update: After running into some issues with rsync copies that contained spaces, I have updated the commands to use the -s option (explained in this answer)
However, one observation I have noticed where rsync differs from scp and sftp (in regards to handling spaces) is that scp was always fine with paths that contained inner quotes such as
scp -rp "remoteuser@remotehost:'/tmp/rsync-test/stuff from server/the data.txt'" "."
Whereas, if you do that in rsync, you'll get an error:
$ rsync -saXULz --progress -e ssh \
"remoteuser@remotehost:'/tmp/rsync-test/stuff from server/the data.txt'" "."
receiving incremental file list
rsync: [sender] change_dir "/home/remoteuser/'/tmp/rsync-test/stuff from server" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1839) [Receiver=3.2.5]
rsync: [Receiver] write error: Broken pipe (32)
The fix for this is quite easy but maybe not intuitive if you are using to using the inner quoted path syntax from scp like I was: just use -s and get rid of the inner quotes. rsync is smart enough to handle it and it just works.
So the above command that had an error, would just be changed to:
$ rsync -saXULz --progress -e ssh \
"remoteuser@remotehost:/tmp/rsync-test/stuff from server/the data.txt" "."
And all should be good (assuming scp works fine for the same path).
