9

Quick version:

ssh & scp work from my desktop and laptop computers. From termux on my android phone, ssh works but scp does not. In this case, I get an error about "Connection closed".


Expanded version

Originally, when I had asked this question, I had been encountering it while running scp from termux on my android phone while connecting to my Fedora Linux box. And I was quite stumped as ssh had been working so I had provided a lot of debug info. Since I later learned that this issue had nothing at all to do with termux and everything to do with OpenSSH versions, I have attempted to cleanup a lot of the extraneous info dumps and make this more accessible while still preserving relevant error messages for better seo matching.

devices:

  • device A (computer): Fedora 35 running openssh.x86_64 8.7p1-3.fc35
  • device B (computer): Fedora 35 running openssh.x86_64 8.7p1-3.fc35
  • device C (phone): termux on android (I failed to capture the openssh version originally, presumably it was v8.8 or later)

Computers A and B had essentially identical sshd_config files, all 3 essentially the same ~/.ssh/config file contents.

issue:

ssh connections between A->B, B->A, C->A, and C->B all worked fine. I was not interested in connecting to my phone from either computer and never configured sshd on it, so did not test that scenario.

However, for scp running over ssh, I was encountering the following: A->B and B->A worked while C->A and C->B failed with the following error:

$ scp -rp "[email protected]:/home/desktop-user/Pictures/test.jpg" .
scp: Connection closed
 

debugging:

  • LOTS of verifying ssh file permissions and ownership, ~/.ssh/config settings, etc. No obvious issues.
  • I had (temporarily) disabled SELinux and firewalld on both A and B but was still getting the same issue.
  • Based on this post, I renamed my custom my id_rsa (and id_rsa.pub) filenames to using the default names and edited my corresponding ~/.ssh/config definitions. No change.
  • Instead of (from C) trying to receive/pull a file from the remote, I attempt to push/send a file to the remote. No change.
  • From C, I made sure all packages were up-to-date. No change.
  • From C, I ran the scp command with -v flag to give verbose debug info. After redacting hashes and such I added to pastebin. The main thing I saw was that the keys appeared to be accepted but last line indicated that "debug1: Exit status 127". Results online seemed to indicate that scp did not exist on host... but in my case, it was. So still no change.

sshd config of hosts

Again, A and B had essentially the same config

# grep -Pv '^\s*(#|$)' /etc/ssh/sshd_config
Include /etc/ssh/sshd_config.d/*.conf
Port 22
Ciphers [email protected],aes256-ctr
LoginGraceTime 1m
PermitRootLogin no
MaxAuthTries 4
MaxSessions 6
PermitEmptyPasswords no
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
UseDNS yes
AcceptEnv LANG LC_*
Subsystem   sftp    /usr/lib/openssh/sftp-server
Protocol 2
DenyUsers root docker-user
MACs [email protected],hmac-sha2-512,[email protected],hmac-sha2-256
KexAlgorithms diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256
2
  • considering the helpful detailed answer you provided, it may help to simplify the question (like removing the references to Termux).
    – Uriel
    Oct 10, 2022 at 19:22
  • @Uriel thanks for the feedback. I have attempted to clean it up / remove references to termux from the title/tags and reduce its mention in the question body to only a historical / minor detail while also mentioning up-front that while that was what I ran into the issue on, it had no relevance to the final solution. Hope that is easier to read and less confusing ;-)
    – zpangwin
    Oct 14, 2022 at 22:50

1 Answer 1

15

Quick version (TL;DR)

Update: For those that just want a quick-and-dirty with minimal reading.

Know:

  1. This actually had nothing to do with Termux - it affects any Linux using OpenSSH 8.8+. I also ran into it on Fedora 35 with OpenSSH 8.7p1
  2. While sftp will work in a pinch, it handles copying of symlinks differently than scp so don't just throw it in a script as a replacement. Better to use rsync.

Recursively copy a directory from remote to local:

# deprecated / insecure (see CVE link below) but works
scp -O -rp -P <port> "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"
 
# secure / recommended by openssh - also works
rsync -saLPz --port <port> -e ssh "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"

Recursively copy a directory from local to remote:

# deprecated / insecure (see CVE link below) but works
scp -O -rp -P <port> "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"
 
# secure / recommended by openssh - also works
rsync -saLPz --port <port> -e ssh "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"

This works fine for multiple paths too, e.g.

# copy multiple paths
rsync -saLPz --port <port> -e ssh "<SRC_PATH_1>" "<SRC_PATH_2>" "<SRC_PATH_3>" "<user>@<host_ip>:<DST_PATH>"
 
# copy multiple paths from array
arrPaths=("<SRC_PATH_1>" "<SRC_PATH_2>" "<SRC_PATH_3>");
rsync -saLPz -e ssh "${arrPaths[@]}" "<user>@<host_ip>:<DST_PATH>"

Root Cause

After a lot of hunting online, I finally found an answer here - credit / thanks to manjaro forums users zbe and canyue980 for finding the initial fix and cluing me into what was actually going on.

To quote canyue980:

Thanks for all guys that offer helps! I just found the solution in upper link:

Note: Since OpenSSH 8.8 the scp utility uses the SFTP protocol by default. The -O option must be used to use the legacy SCP protocol.

by using -O option and it works!

scp -O file_name target_dir

and of course you can also add an alias for it in~/.zshrc like this:

alias scp="scp -O"

Indeed, changed my command from:

scp -rp -P 1234 "[email protected]:/home/desktop-user/Pictures/test.jpg" .

To:

scp -rpO -P 1234 "[email protected]:/home/desktop-user/Pictures/test.jpg" .

and it worked like a champ. Will leave this up in hopes it helps someone else.


Security Implications of -O

Security Implications of using -O (that's "Oh" as in OLD - not zero): according to the release notes for openssh 8.8, it sounds like the main risk is that:

Legacy scp/rcp performs wildcard expansion of remote filenames (e.g. "scp host:* .") through the remote shell. This has the side effect of requiring double quoting of shell meta-characters in file names included on scp(1) command-lines, otherwise they could be interpreted as shell commands on the remote side.

I don't personally see this as a big risk for people used to the old command and running manually or via trusted scripts on home computers. But your risk-assessment / use-cases may be different - especially for business systems or setups that are using unproved / untrusted scripts. This appears to be the case for some as RHEL 9 is deprecating the scp protocol, citing that:

We're making this change because the SCP protocol is decades old, and carries multiple security risks and issues that have no straightforward solutions. New issues are being reported frequently (CVE-2020-15778 is the most recent as of this writing, but we can’t be sure it will be the last) and it is rather difficult to fix them all properly because the protocol is inherently trustworthy of authenticated sessions.

From the above release notes and links, as well as a note on the arch wiki, it seems the intention is to completely deprecate SCP (e.g. the protocol) in favor of alternatives like sftp and rsync (see below). I am unclear of what the intended future of the scp command is, especially in the short-term, but it seems plausible or even likely that it may also get deprecated in the future.


Recommended replacements

To use sftp and rsync as drop-in replacements in scripts, probably rsync is the easier drop-in replacement as it doesn't require any changes on the host whereas at least as of Fedora 35 sftp does.

The commands below should work for sftp as long as you can connect with sftp <user>@<host_ip> just like you would with ssh (connecting this way will put you in the sftp "shell". To exit, use exit). Otherwise, you may have to enable it on the host. For Fedora 35, that process was as follows:

  1. sudo cp -a /etc/ssh/sshd_config /etc/ssh/sshd_config.$(date +'%F_%T').bak
  2. sudo sed -Ei 's~^(Subsystem[ \t]+sftp[ \t]+)(/usr/lib/openssh/sftp-server)~#\1\2\n\1internal-sftp~g' /etc/ssh/sshd_config
  3. sudo systemctl restart sshd

when done you should see something like this (e.g. the sftp server line should be commented out and a new line with internal-sftp add to take its place) :

sudo grep Subsystem /etc/ssh/sshd_config
#Subsystem  sftp    /usr/lib/openssh/sftp-server
Subsystem   sftp    internal-sftp

You can also restrict this using groups and chroot if you need increased security, as shown here and here.

Note: scp by default copies the contents of symlinks instead of them as a symlink. Rsync lets you control this behavior by using -l to copy as symlink or -L to copy what is referenced by the symlink. The -a (archive) option will default to -l, unless -L is also specified. sftp will copy the symlink as a link (e.g. the contents pointed to are not copied) and does not provide an option for controlling this behavior as of time of writing. You can verify this by looking at the documentation for the -r option under man scp and man sftp respectively.

Here are a few examples of roughly equivalent commands, which should all be going over SSH and picking up your ~/.ssh/config settings and identities:

Recursively copy a directory from remote to local:

scp -O -rp -P <port> "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"
 
rsync -saLPz --port <port> -e ssh "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"

Recursively copy a directory from local to remote:

scp -O -rp -P <port> "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"
 
rsync -saLPz --port <port> -e ssh "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"

Corrections for sftp syntax

Edit: 2022-Sep-13: Noticed today while retesting that the commands I had for sftp were giving errors, so I am moving them out from the above and adding some disclaimers. I don't personally recommend sftp as a replacement for scp anyway due to the differences I mentioned above in how they handle copying when symlinks are involved.

I was specifically interested in using this as a replacement in scripts. If you are only needing interactive mode, don't mind making a "batch" instruction file for sftp, or have rsync installed on all of the machines you'll be using then you probably won't need to go this far. My notes below are partially based this answer, with some additional testing done by me.

My biggest complaint here (other than sftp using different syntax for upload vs download) is the very poor documentation. Looking at man sftp, the syntax isn't documented at all (meaning that while the tool can be used non-interactively, it is only documented for use in batch and interactive modes, and stepping outside those use-cases, while supported, is not documented). The man page gives NO examples of common usage or even a website for more detailed documentation as many of the GNU tools do. And sftp -h provides less than the man page. Literally, if it weren't for search engines and sites like this and I would have completely given up on sftp as an option altogether.

1. Local pull (e.g. from local terminal, download files from remote to local)

This is mostly straight-forward and works somewhat similarly to scp other than the differences noted above (e.g. symlinks) and slightly different syntax. And that one of the variants will result in an interactive prompt.

# Download a single file with same name to current dir
sftp -pC -P <port> "<user>@<host_ip>:<SRC_PATH>"
# or
sftp -pC -P <port> "<user>@<host_ip>:<SRC_PATH>" .
 
# Download a single file to name and path in "<DST_PATH>"
sftp -pC -P <port> "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"
 
# Download a folder and all of its contents,
# keep the same name as the remote and save to current dir
# note that any symlinks will copy only the symlink itself
# and NOT the data the link references
sftp -rpC -P <port> "<user>@<host_ip>:<SRC_PATH>" .
 
# Now if you omit "<DST_PATH>" entirely like you can with
# the single-file version (first command in this code-block)
# you instead drop to an interactive prompt and have to type
# type 'exit' to get back to your bash prompt
# e.g. AVOID THIS IN NON-INTERACTIVE SCRIPTS!!!
sftp -rpC -P <port> "<user>@<host_ip>:<SRC_PATH>"
Connected to <host_ip>.
Changing to: <SRC_PATH>
sftp> 
sftp> exit
  
# Download a folder and all of its contents,
# to name and path in "<DST_PATH>"
# note that any symlinks will copy only the symlink itself
# and NOT the data the link references
sftp -rpC -P <port> "<user>@<host_ip>:<SRC_PATH>" "<DST_PATH>"

2. Local push (e.g. from local terminal, upload files from local to remote)

Firstly, if you just try to reverse the parameters, like what my original answer had, then you will get a connection error. Why doesn't sftp make the upload syntax consistent with the download syntax? Your guess is as good as mine... I for one would have been a huge fan of that approach. But if you try it, here's what happens (note: this output is from 2 machines running Fedora 35 with correctly configured ssh where scp and rsync ... -e ssh work just fine and do not prompt for authentication).

# single file
sftp -pC -P <port> test.txt "<user>@<host_ip>:<DST_PATH>"
ssh: Could not resolve hostname test.txt: Name or service not known
Connection closed.  
Connection closed

# dir
sftp -rpC -P <port> "<SRC_PATH>" "<user>@<host_ip>:<DST_PATH>"
ssh: Could not resolve <SRC_PATH>: Name or service not known
Connection closed.  
Connection closed

Per the answer I linked to above, the correct way to do this is instead as follows

# push a single file from local to remote
sftp -pC -P <port> "<user>@<host_ip>:<DST_PATH>" <<< 'put test.txt'
# or
echo 'put test.txt' | sftp -pC -P <port> "<user>@<host_ip>:<DST_PATH>"

# push a entire dir (and its contents) from local to remote
# notice the inner single quotes to escape <SRC_PATH>
# for enclosing paths containing whitespace
sftp -rpC -P <port> "<user>@<host_ip>:<DST_PATH>" <<< "put '<SRC_PATH>'"
# or
echo "put '<SRC_PATH>'" | sftp -rpC -P <port> "<user>@<host_ip>:<DST_PATH>"
# or
printf 'put "%s" "%s"\n' "<SRC_PATH>" "<DST_PATH>" | \
sftp -rpC -P <port> "<user>@<host_ip>"

I found in my scripting that using a similar syntax for downloads, e.g.

# pull a entire dir (and its contents) from remote to local
# notice the inner single quotes to escape <SRC_PATH>
# for enclosing paths containing whitespace
printf 'get "%s" "%s"\n' "<SRC_PATH>" "<DST_PATH>" | \
sftp -rpC -P <port> "<user>@<host_ip>"

was less confusing but all of the above should work.

To copy multiple paths at once, an array works (or you could create a batch file listing each put command on a separate line and pass that to sftp using the -b <batchfile> option).

# copy multiple paths from array
arrPaths=("<SRC_PATH_1>" "<SRC_PATH_2>" "<SRC_PATH_3>");
printf 'put %s\n' "${arrPaths[@]}" | sftp -rpC -P <port> "<user>@<host_ip>:<DST_PATH>"

2022-Oct-10 Update: After running into some issues with rsync copies that contained spaces, I have updated the commands to use the -s option (explained in this answer)

However, one observation I have noticed where rsync differs from scp and sftp (in regards to handling spaces) is that scp was always fine with paths that contained inner quotes such as

scp -rp "remoteuser@remotehost:'/tmp/rsync-test/stuff from server/the data.txt'" "."

Whereas, if you do that in rsync, you'll get an error:

$ rsync -saXULz --progress -e ssh \
  "remoteuser@remotehost:'/tmp/rsync-test/stuff from server/the data.txt'" "."
 
receiving incremental file list
rsync: [sender] change_dir "/home/remoteuser/'/tmp/rsync-test/stuff from server" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1839) [Receiver=3.2.5]
rsync: [Receiver] write error: Broken pipe (32)

The fix for this is quite easy but maybe not intuitive if you are using to using the inner quoted path syntax from scp like I was: just use -s and get rid of the inner quotes. rsync is smart enough to handle it and it just works.

So the above command that had an error, would just be changed to:

$ rsync -saXULz --progress -e ssh \
  "remoteuser@remotehost:/tmp/rsync-test/stuff from server/the data.txt" "."

And all should be good (assuming scp works fine for the same path).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .