File goes missing for a short period of time after saving it using VIM

Question

I am having a strange problem that seams to be caused by VIM, the Linux VFS cache, ecryptfs and/or something file system related:

1. I open a file in VIM, modify it and save it.
2. I try to access the file.

Expected behaviour

The file should be accessible as soon as :w reports the file as written.

Actual behaviour

The file does not exist.

If I wait a bit (usually less then a second) the file turns up.

This is especially cumbersome when working with Python code and left over pyc files. I often ended up starting the old code as the new py file was not ready, yet. I recently added export PYTHONDONTWRITEBYTECODE=1 to my .bashrc so I get a meaningful error message rather than executing the old code. It is still very awkward as any auto reloader code (e.g. from Django) fails to reload every 5-10 times because the file I changed goes missing for a short period of time. With the pyc files in place the auto reloader sometimes ended up loading the old pyc file and after that never found out that the py file has been modified and triggered another reload.

System details and configuration

My machine has plenty of RAM (32 GiB), an SSD and is basicly idle. Therefore I do not think that slow I/O is causing this. The file is very small (<1 KiB) and it also happens for empty files. I am using a crypted $HOME using ecryptfs so this might be part of the problem. I was unable to reproduce this on my /tmp mount which uses a tmpfs file system.

VIM settings

The reason why the file is moved away and replaced by a new file is caused by my VIM settings:

set backup
set backupskip=
set backupdir=$HOME/.vimbackup
set writebackup

I would expect the new file to be accessible right after VIM reports that the file has been written. I checked the VIM documentation for any hints of a delayed write, but did not find anything. I was unable to reproduce this using the shell commands mv, cp and rm so I think VIM is doing something different.

---

What else could be causing this? How can I solve this.

Answer 1

This is a bug but it isn't related to the one that Aaron linked to. I'm unable to reproduce it at the moment, so can you please file a new bug here: https://bugs.launchpad.net/ecryptfs/+filebug

You can copy and paste from the description above, but I also need to know more about the Linux distribution and kernel version that you're using. Thanks!

Answer 2

It's probably a bug / by-design defect in ecryptfs. See this launchpad bug report.

In a nutshell, ecryptfs simply doesn't write the file to disk immediately. My guess this is because of the encryption overhead (the filesystem probably encrypts the data in a background thread and writes it only after that has completed).

The bug is from 2009 and the priority is "Wishlist" (which is below "Low"). I am a bit worried how "enterprise" and such a behavior mixes; as you found out, a lot of code expects files to be usable right away after saving them.

Try to use TrueCrypt instead.

Answer 3

I think the behaviour you see is just because the Vim backup process is slow. On my plain Ext4 system, this problem manifests as a "file is empty" error from the compiler.

To check the timings, I used this Bash sequence:

strace -tt -o /dev/stdout gvim --nofork main.cxx | grep 'main.cxx\|close'

With backups turned on, I see a 200 ms gap between the file disappearing and it finally getting saved:

09:06:49.587341 rename("main.cxx", "main.cxx~") = 0
09:06:49.668654 open("main.cxx", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 12
09:06:49.755454 close(12)               = 0

When I use set nowritebackup in my .vimrc, the file does not get renamed; there's only an open and a close:

09:19:45.731416 open("main.cxx", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 12
09:19:45.815763 close(12)               = 0

Is this a good explanation for the problem you're experiencing?

---

P.S. I couldn't find any other mention of this problem on the Vim issue tracker, or on the Vim developer/user mailing lists, so I can't say whether the Vim team would consider changing the backup system call sequence.