SSHFS/SFTP : Vim hangs when trying to edit file over sshfs and can't upload file over MTU size over sftp

Question

My university provides an ssh access to the students university accounts, and I just set up an sshfs to access some folders on my remote account.

I am running pop_os on my local machine and using systemd automount with following options : noauto,x-systemd.automount,_netdev,user,idmap=user,follow_symlinks,workaround=rename,IdentityFile=/home/me/.ssh/identityfile,allow_other,default_permissions,uid=1000,gid=1000,ServerAliveInterval=15

The issue

Automount works fine, I can browse my remote folder, but I have some trouble with replacing files and recursively copying folders to the remote.

But every time I try to edit a file on the remote fs with vim, the editor hangs for 60 seconds (very precisely, which I guess has something to do with the connection timeout parameters) before opening the file. When the file finally opens I get and error stating E297: Write error in swap file and on close E72: Close error on swap file from vim.

But sometimes (particularly after I have reloaded the dameon and restarted the automount service after updating options in /etc/fstab, the file opens instantly with no errors.

What I have tried

I ran strace when opening a file on the sshfs, here is a sample of the log where the issue seems to be happening :

     0.000032 openat(AT_FDCWD, ".../myremotefile.c", O_RDONLY) = 3
     0.041890 readlink(".../myremotefile.c", 0x7ffd99b19540, 4095) = -1 EINVAL (Invalid argument)
     0.000302 openat(AT_FDCWD, ".../.myremotefile.c.swp", O_RDONLY) = -1 ENOENT (No such file or directory)
     0.029883 openat(AT_FDCWD, ".../.myremotefile.c.swp", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
     0.103207 openat(AT_FDCWD, ".../.myremotefile.c.swx", O_RDONLY) = -1 ENOENT (No such file or directory)
     0.074409 openat(AT_FDCWD, ".../.myremotefile.c.swx", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
     0.116541 fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
     0.000419 fstat(5, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
     0.000318 close(5)                  = 0
     0.000268 unlink(".../.myremotefile.c.swx") = 0
     0.075053 close(4)                  = 0
     0.000448 unlink(".../.myremotefile.c.swp") = 0
     0.061916 stat(".../.myremotefile.c.swp", 0x7ffd99b1a4d0) = -1 ENOENT (No such file or directory)
     0.077126 lstat(".../.myremotefile.c.swp", 0x7ffd99b1a660) = -1 ENOENT (No such file or directory)
     0.041249 openat(AT_FDCWD, ".", O_RDONLY) = 4
     0.000218 fchdir(4)                 = 0
     0.000197 chdir(".../myremotefolder") = 0
     0.000086 getcwd("/home/myhomefolder/.../myremotefolder", 4096) = 34
     0.000029 fchdir(4)                 = 0
     0.000025 close(4)                  = 0
     0.000027 lstat(".../.myremotefile.c.swp", 0x7ffd99b1a9e0) = -1 ENOENT (No such file or directory)
     0.043704 openat(AT_FDCWD, ".../.myremotefile.c.swp", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW, 0600) = 4
     0.127965 fcntl(4, F_GETFD)         = 0
     0.000331 fcntl(4, F_SETFD, FD_CLOEXEC) = 0
     0.000347 openat(AT_FDCWD, ".", O_RDONLY) = 5
     0.000221 fchdir(5)                 = 0
     0.000145 chdir(".../myremotefolder") = 0
     0.046751 getcwd("/home/myhomefolder/.../myremotefolder", 4096) = 34
     0.000375 fchdir(5)                 = 0
     0.000351 close(5)                  = 0
     0.000262 lseek(4, 0, SEEK_SET)     = 0
     0.000082 write(4, "b0VIM 8.1\0\0\0\0\20\0\0\217\266\235]>\0\0\0\335\16\0\0balo"..., 4096) = 4096
     0.000645 select(1, [0], [], [0], {tv_sec=0, tv_usec=0}) = 0 (Timeout)
     0.000176 chmod("..../.myremotefile.c.swp", 0644) = -1 EIO (Input/output error)
    60.064610 close(3)                  = -1 ENOTCONN (Transport endpoint is not connected)

You can see there that the process hangs for 60 seconds on close.

The issue seems to be happening when vim tries to create the .swp file, but I can't figure out why, because I have no trouble manually creating or deleting files or folders on the remote fs.

I don't understand sshfs and fuse enough to fix this on my own.

I can provide the strace logs for gcc and mv too. Ask me if you want precisions.

Thanks in advance for your answers.

Edit : I set the directory option in vimrc so that vim creates its swap file in my local .vim folder, which has solved the swap file errors I had.

But I still got some I/O errors while doing other operations, so I believe what was happening with vim is just the symptom of a more complex issue that I will try to investigate.

As stated by @roaima the issue seems to be caused by the connection unexpectedly closing, which happened when vim tried to create the swap file in the remote fs.

I recently noticed that I can't copy some files no matter what, I can't say for sure now but I think it may be files larger than a certain size, I need to do some tests.

Edit2 : After doing some tests today I noticed that when I try to upload a file exceeding the MTU size (1500) over ssh (I tried uploading a file over sshfs, scp and in a sftp session), when doing so with ssh ServerAliveInterval option set, connection drops after 60 seconds.

Answer 1

The sshfs FUSE filesystem is implemented by presenting a filesystem on top of sftp, the file transfer protocol. As a result, any file access such as editing with vi[m] requires the sshfs subsystem first to copy the file to a cache on the local filesystem. If the file is particularly large, or the network between your client and the server is particularly slow, it will take a measurable amount of time to transfer the file before it's accessible locally.

It's (very) broadly equivalent to the following (except it uses sftp instead of scp)

# Copy the remote file to a temporary local cache
scp -p remote:/path/to/file /tmp/file.tmp
checksum=$(cksum /tmp/file.tmp)

# Action on remote file is implemented by performing the action locally
vi /tmp/file.tmp

# Simplified; we would also need to handle local rm/mv -> remote rm/mv, etc.
[[ "$(cksum /tmp/file.tmp)" != "$checksum" ]] && scp -p /tmp/file.tmp remote:/path/to/file

As a consequence, you'll find that trying to run gcc locally will be measurably slower than just logging in to the remote server and running it there. To be honest I'm not overly surprised that "gcc crashes when trying to compile files on the remote fs". It shouldn't, of course, but then think about what's actually going on in the background...

Answer 2

Okay, so lowering the MTU size seems to have resolved all my I/O issues over SSHFS. As pointed by @roaima the issues were probably caused by the firewall of the university server blocking ICMP packets too eagerly.

Thanks to @roaima for guiding me toward the solution to that issue. And thanks to @derobert for suggesting to lower the MTU.