6

My university provides an ssh access to the students university accounts, and I just set up an sshfs to access some folders on my remote account.

I am running pop_os on my local machine and using systemd automount with following options : noauto,x-systemd.automount,_netdev,user,idmap=user,follow_symlinks,workaround=rename,IdentityFile=/home/me/.ssh/identityfile,allow_other,default_permissions,uid=1000,gid=1000,ServerAliveInterval=15

The issue

Automount works fine, I can browse my remote folder, but I have some trouble with replacing files and recursively copying folders to the remote.

But every time I try to edit a file on the remote fs with vim, the editor hangs for 60 seconds (very precisely, which I guess has something to do with the connection timeout parameters) before opening the file. When the file finally opens I get and error stating E297: Write error in swap file and on close E72: Close error on swap file from vim.

But sometimes (particularly after I have reloaded the dameon and restarted the automount service after updating options in /etc/fstab, the file opens instantly with no errors.

What I have tried

I ran strace when opening a file on the sshfs, here is a sample of the log where the issue seems to be happening :

     0.000032 openat(AT_FDCWD, ".../myremotefile.c", O_RDONLY) = 3
     0.041890 readlink(".../myremotefile.c", 0x7ffd99b19540, 4095) = -1 EINVAL (Invalid argument)
     0.000302 openat(AT_FDCWD, ".../.myremotefile.c.swp", O_RDONLY) = -1 ENOENT (No such file or directory)
     0.029883 openat(AT_FDCWD, ".../.myremotefile.c.swp", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
     0.103207 openat(AT_FDCWD, ".../.myremotefile.c.swx", O_RDONLY) = -1 ENOENT (No such file or directory)
     0.074409 openat(AT_FDCWD, ".../.myremotefile.c.swx", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
     0.116541 fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
     0.000419 fstat(5, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
     0.000318 close(5)                  = 0
     0.000268 unlink(".../.myremotefile.c.swx") = 0
     0.075053 close(4)                  = 0
     0.000448 unlink(".../.myremotefile.c.swp") = 0
     0.061916 stat(".../.myremotefile.c.swp", 0x7ffd99b1a4d0) = -1 ENOENT (No such file or directory)
     0.077126 lstat(".../.myremotefile.c.swp", 0x7ffd99b1a660) = -1 ENOENT (No such file or directory)
     0.041249 openat(AT_FDCWD, ".", O_RDONLY) = 4
     0.000218 fchdir(4)                 = 0
     0.000197 chdir(".../myremotefolder") = 0
     0.000086 getcwd("/home/myhomefolder/.../myremotefolder", 4096) = 34
     0.000029 fchdir(4)                 = 0
     0.000025 close(4)                  = 0
     0.000027 lstat(".../.myremotefile.c.swp", 0x7ffd99b1a9e0) = -1 ENOENT (No such file or directory)
     0.043704 openat(AT_FDCWD, ".../.myremotefile.c.swp", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW, 0600) = 4
     0.127965 fcntl(4, F_GETFD)         = 0
     0.000331 fcntl(4, F_SETFD, FD_CLOEXEC) = 0
     0.000347 openat(AT_FDCWD, ".", O_RDONLY) = 5
     0.000221 fchdir(5)                 = 0
     0.000145 chdir(".../myremotefolder") = 0
     0.046751 getcwd("/home/myhomefolder/.../myremotefolder", 4096) = 34
     0.000375 fchdir(5)                 = 0
     0.000351 close(5)                  = 0
     0.000262 lseek(4, 0, SEEK_SET)     = 0
     0.000082 write(4, "b0VIM 8.1\0\0\0\0\20\0\0\217\266\235]>\0\0\0\335\16\0\0balo"..., 4096) = 4096
     0.000645 select(1, [0], [], [0], {tv_sec=0, tv_usec=0}) = 0 (Timeout)
     0.000176 chmod("..../.myremotefile.c.swp", 0644) = -1 EIO (Input/output error)
    60.064610 close(3)                  = -1 ENOTCONN (Transport endpoint is not connected)

You can see there that the process hangs for 60 seconds on close.

The issue seems to be happening when vim tries to create the .swp file, but I can't figure out why, because I have no trouble manually creating or deleting files or folders on the remote fs.

I don't understand sshfs and fuse enough to fix this on my own.

I can provide the strace logs for gcc and mv too. Ask me if you want precisions.

Thanks in advance for your answers.

Edit : I set the directory option in vimrc so that vim creates its swap file in my local .vim folder, which has solved the swap file errors I had.

But I still got some I/O errors while doing other operations, so I believe what was happening with vim is just the symptom of a more complex issue that I will try to investigate.

As stated by @roaima the issue seems to be caused by the connection unexpectedly closing, which happened when vim tried to create the swap file in the remote fs.

I recently noticed that I can't copy some files no matter what, I can't say for sure now but I think it may be files larger than a certain size, I need to do some tests.

Edit2 : After doing some tests today I noticed that when I try to upload a file exceeding the MTU size (1500) over ssh (I tried uploading a file over sshfs, scp and in a sftp session), when doing so with ssh ServerAliveInterval option set, connection drops after 60 seconds.

  • You say that you can "copy files from remote to local with no issues" but that "mv hangs for 60 seconds". Was it a mv from or to the remote? From the strace log, I see an error on the chmod call; are you sure of your permission options: uid, gid, etc – AlexisBRENON Oct 11 '19 at 20:53
  • 1
    Does sftp to the server work? Can you upload and download files with sftp? Also, have you tried lowering the MTU to, e.g., 1300? If it's actually MTU-related, that will probably work... – derobert Oct 15 '19 at 23:53
  • The the mv was from the local to the remote, I think my problem might be due to a bad firewall configuration. I can pull files without troubles, but as soon as I want to upload a file over 1500 bytes (size of the mtu) the sftp hangs, and the ssh connection drops. – Balocre Oct 16 '19 at 5:55
  • I have tried lowering a bit the MTU, I did not go as low as 1300 though, but it did not seem to have any effect. I will try that. – Balocre Oct 16 '19 at 5:56
1

The sshfs FUSE filesystem is implemented by presenting a filesystem on top of sftp, the file transfer protocol. As a result, any file access such as editing with vi[m] requires the sshfs subsystem first to copy the file to a cache on the local filesystem. If the file is particularly large, or the network between your client and the server is particularly slow, it will take a measurable amount of time to transfer the file before it's accessible locally.

It's (very) broadly equivalent to the following (except it uses sftp instead of scp)

# Copy the remote file to a temporary local cache
scp -p remote:/path/to/file /tmp/file.tmp
checksum=$(cksum /tmp/file.tmp)

# Action on remote file is implemented by performing the action locally
vi /tmp/file.tmp

# Simplified; we would also need to handle local rm/mv -> remote rm/mv, etc.
[[ "$(cksum /tmp/file.tmp)" != "$checksum" ]] && scp -p /tmp/file.tmp remote:/path/to/file

As a consequence, you'll find that trying to run gcc locally will be measurably slower than just logging in to the remote server and running it there. To be honest I'm not overly surprised that "gcc crashes when trying to compile files on the remote fs". It shouldn't, of course, but then think about what's actually going on in the background...

  • Hi @roaima the file I edit are not particularly large (for example I still get the issue for a 10 lines long bash script) and the connection is pretty decent. Although I understand why gcc can fail using this set up, editing a small file over vim shouldn't be an issue. – Balocre Oct 14 '19 at 16:15
  • 1
    Hi again @roaima, creating the vim swap file locally (using the set direcory dir option in vimrc) instead of creating them in the folder containing the file edited completely fixed the vim delay for me, which narrows down my issue to why creating a vim swap file on the remote hangs sshfs for 60 seconds. – Balocre Oct 14 '19 at 16:44
  • 1
    @Balocre agreed. I've just noticed the "Transport endpoint is not connected" message. That suggests your network connection was interrupted, breaking the ssh connection. Possible? – roaima Oct 14 '19 at 16:45
  • @Roiama yes it is I believe what is in fact happening, now I I have to determine why, my guess is that it is either a permission problem, which would somehow cause the connection to crash, but that would be strange because as stated I can create and delete file how I please on the remote (so why would vim not be able to), or a server configuarion/security issue, but I don't understand ssh enough yet to determine that, maybe I should try to contact the IT department. – Balocre Oct 14 '19 at 16:54
  • I will keep you informed, because I am doing some tests right now, but it seems I can't create file over a certain size (which is not very big, I just tried to move a jpeg to the remote and go a cp: failed to close /../myfileonremote.jpeg : Input/output error) which corrupted the file, and might be linked to the vim issue – Balocre Oct 14 '19 at 16:54
0

Okay, so lowering the MTU size seems to have resolved all my I/O issues over SSHFS. As pointed by @roaima the issues were probably caused by the firewall of the university server blocking ICMP packets too eagerly.

Thanks to @roaima for guiding me toward the solution to that issue. And thanks to @derobert for suggesting to lower the MTU.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.