4

I am developing software that will utilize inotify to track changes on a large amount of files (tens to hundreds of thousands of files). I have come up with these ideas:

  • one watch per file
  • one watch per parent directory
  • avoid inotify and periodically scan the fs for changes (not preferred)

I will have a database of all of the files I am watching and some basic stat information (like mtime and size), however, I would have to stat each file in that directory until I found the one that changed.

Which would be faster, tons (100,000+) of inotify watches or tons of stat calls?

I'm thinking that reducing the number of stat calls would be better, but I don't know enough about inotify.

Note:

This will be running on a workstation, not a server. It's main purpose is to synchronize changes (potentially to an entire filesystem) between a client and a remote server.

5
  • There are filesystems designed for this kind of replication. I recommend investigating them rather than reinventing the locomotive (it's more complex than a wheel). I don't have a filesystem to offer but I remember one being mentioned on this site recently, hopefully someone can chime in with proposals. Commented Aug 10, 2011 at 0:55
  • I would be very interested. The only thing I can think of is an NFS filesystem, which isn't exactly what I'm looking for.
    – beatgammit
    Commented Aug 10, 2011 at 4:59
  • 1
    Ah, found it: ChironFS. Even if you don't end up using it, its design documentation and source code are likely to be interesting reads. Commented Aug 10, 2011 at 7:26
  • @Gilles- Thanks. I assumed it would be a FUSE-based fs. I'll probably end up needing more features than it has, and it looks a little too young, but definitely an interesting read.
    – beatgammit
    Commented Aug 10, 2011 at 15:37
  • Using a filesystem based on this is not the way to go. This would be for servers where you have admins. This mentality is what makes Linux still in the 1% marketshare. This functionality should be able to be used with a quickly installed flatpak/snap after a download. This comment is for left here for eternity as a reminder.
    – Lothar
    Commented Jul 16, 2023 at 15:42

1 Answer 1

2

When you read() an inotify fd, the name field of the returned struct tells you which file was modified relative to the directory being watched, so you shouldn't have to stat every file in a directory after the event.

See http://linux.die.net/man/7/inotify

Specifically:

struct inotify_event {
    int      wd;       /* Watch descriptor */
    uint32_t mask;     /* Mask of events */
    uint32_t cookie;   /* Unique cookie associating related
                          events (for rename(2)) */
    uint32_t len;      /* Size of 'name' field */
    char     name[];   /* Optional null-terminated name */
};

The name field is only present when an event is returned for a file inside a watched directory; it identifies the file pathname relative to the watched directory. This pathname is null-terminated, and may include further null bytes to align subsequent reads to a suitable address boundary.

3
  • So, is there any gain to adding a listener to individual files?
    – beatgammit
    Commented Aug 9, 2011 at 1:50
  • Only if there are going to be many events on files you don't care about in that directory. Commented Aug 9, 2011 at 1:55
  • WARNING. THIS WILL NOT WORK. Just imagine hard links. The kernel does not maintain a hardlink database. It might only report for modifications if the user is opening the file via the same file path. So a general purpose watcher still has to scan peridocially unless newer APIs like fannotify will be provided by the kernel.
    – Lothar
    Commented Jul 16, 2023 at 15:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .