I have an Amazon Linux instance running SSH acting as an SFTP server. Clients log in, and are chrooted into an NFS-mounted directory. Users can read, write, and delete files, but renaming files fails with a non-specific "protocol error".

Here is a copy of my sshd_config file:

Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
UsePrivilegeSeparation yes

KeyRegenerationInterval 3600
ServerKeyBits 1024

SyslogFacility AUTH
LogLevel INFO

LoginGraceTime 120
PermitRootLogin prohibit-password
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes

IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no

PermitEmptyPasswords no

ChallengeResponseAuthentication no

PasswordAuthentication yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

AcceptEnv LANG LC_*

# Subsystem sftp /usr/lib/openssh/sftp-server -u 0002
Subsystem sftp internal-sftp -l DEBUG -u 002 -d %u

UsePAM yes
Match Group sftpusers
    ChrootDirectory /autohome
    AllowTCPForwarding no
    X11Forwarding no
    ForceCommand internal-sftp -l DEBUG -u 002 -d %u

I've seen reference to sftp rename not working when the source and destination are on separate filesystems, but that's not the case here. I've also seen reference to sftp rename not working on filesystems that do not support hard links, but I think our NFS server (AWS File Storage Gateway) should be fine. I'm at a loss, any help is appreciated.

  • If you strace the sshd instance handling the SFTP session while trying the rename operation, you should be able to see exactly what system call the sftp server logic is trying to do, and exactly what system error it's getting.
    – Kenster
    Feb 22 '18 at 20:40
  • Turns out my assumption was wrong. Our NFS mount isn't allowing hard links, so that's a separate problem. Thanks for pushing me in the right direction. If you add this as an answer I'll mark it as correct.
    – Jon Buys
    Feb 22 '18 at 22:52
  • I'm not sure a troubleshooting tip counts as an answer. You can answer your own question, and I think it'd be a good idea to describe what the problem was and how you figured it out.
    – Kenster
    Feb 22 '18 at 23:07

Thanks to @Kenster's tip I found the issue. I was mistaken in assuming that the AWS File Storage Gateway NFS mount supported hard links, as the documentation clearly states that it does not.

I was so sure that was the case that I wound up tracing system calls with strace. If you attach an sftp client to your server while ssh'd into it, get the pid of the current sftp process with ps -eaf | grep sftp. Then you can trace the system calls with strace and save the output to a file with this command: strace -ff -p 2116 -o sftp_rename.log where -ff is following the child processes, -p is the pid, and -o is the output file.

That'll give you some really terrible looking output, but what I found interesting was this bit:

write(7, "\0\0\0L\0\0\0\3\0\0\0Drename old \"/testuse"..., 80) = 80
lstat("/testuser/test/asdfasdf.txt", {st_mode=S_IFREG|0664, st_size=159, ...}) = 0
link("/testuser/test/asdfasdf.txt", "/testuser/test/as.txt") = -1 ENOTSUPP (Unknown error 524)

Which I then tested with a simple link command to create a hard link, which failed.

# ln asdfasdf.txt link.txt
ln: failed to create hard link ‘link.txt’ => ‘asdfasdf.txt’: Unknown error 524

Which led me back to AWS's documentation. That's not all though, apparently SFTP rename will work with certain clients (like Paramiko) that implement a vender specific CMD_EXTENDED protocol, as Paramiko does:

    oldpath = self._adjust_cwd(oldpath)
    newpath = self._adjust_cwd(newpath)
    self._log(DEBUG, 'posix_rename({!r}, {!r})'.format(oldpath, newpath))
        CMD_EXTENDED, "posix-rename@openssh.com", oldpath, newpath

There doesn't seem to be any way to force using the posix-rename option for all clients, but at least we know what happened and why.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.