4

I want to be able to monitor the number of open file in Linux. Currently I am counting the number of files in /proc/<pid>/fd and fdinfo.

My load balancer has about a million FDs. So this ends up taking about 20 seconds to perform.

However this results in high cpu usage: 47.18% [kernel] [k] __d_lookup

Is there a more efficient way to get this number?

  • 1
    Maybe look at SystemTap or such and sample instead of getting an exact count? Iterating million FDs is probably going to be slow inside the kernel, too. – thrig May 18 '17 at 18:41
  • @thrig Ya, systemtap might be able to pull it from the task struct? Since Linux enforces file limits, it must track this sanely somehow somewhere. – Kyle Brandt May 18 '17 at 19:49
  • I'd expect the kernel to track the number of open files per user (for RLIMIT_NOFILE) and altogether, but to list the number of open files for each process, there may be no better way than trawling /proc. You can get an approximate value from FDSize in /proc/$pid/status, if you have processes with many files open. – Gilles May 18 '17 at 21:38
  • How are you doing the count? Are you accidentally stating each file? I can count the number of files in a directory with ~200K files in less than a second: time ls | wc -l 210409 real 0m0.278s – Mark Wagner May 20 '17 at 0:20
1

For a dedicated load balancer I would track the total of files opened in the system, instead of wasting I/O and CPU resources counting them by process. The remaining open files by non-wanted processes should be a meaningless value for the intended result.

For knowing the global open files by the Linux system, there is no need to count them; the Linux kernel keeps track of how many files it has open.

To know that, either run:

cat /proc/sys/fs/file-nr | awk ' { print $1 } '

or

sysctl fs.file-nr | awk ' { print $1 } '

This is much more efficient than counting all the files open with the output of lsof, that will travel all /proc/$PID/fd directories, and will affect negatively your system I/O / CPU resources.

0

The simple:

lsof | wc -l

will tell you the number of files open in the system.

  • That seems to be even slower. – Kyle Brandt May 18 '17 at 18:24
  • lsof does what you're doing, reading <pid>/fd files, but is compiled in C. There's probably not a much faster way of doing it - you're probably limited by the speed of your system. 47% cpu means that the bottleneck is not your processor, but is probably waiting for IO. – einonm May 18 '17 at 18:26
  • @KyleBrandt you can always prepend the command with time to measure the execution time to check which is faster. – einonm May 18 '17 at 18:34
  • I did, I said seems just because I only repeated the test a couple times. – Kyle Brandt May 18 '17 at 18:35
  • @KyleBrandt From vague memory, reading /proc/pid/fd is the only way to get this info, which is an open() and read() syscall per file - which in turn reads a set of kernel structs and returns the answer. You may be able to recompile lsof to shortcut the pipe to 'wc' and print out the total, but I don't think you're going to save much by doing that. You're constrained by the hardware. Would reducing the load on each machine be possible? – einonm May 18 '17 at 18:47
0

How efficient is this SystemTap code for your use case? It's not a perfect view as it only tracks changes from when it begins (so anything opened prior to the start would be missed), and will need additional work to make the output more legible or suitable.

global procfdcount

probe begin {
    printf("begin trace...\n\n")
}

probe syscall.open {
    procfdcount[pid()]++
}

probe syscall.close {
    p = pid()
    procfdcount[p]--
    if (procfdcount[p] < 0) {
        procfdcount[p] = 0
    }
}

probe kprocess.exit {
    p = pid()
    if (p in procfdcount) {
        delete procfdcount[p]
    }
}

probe timer.s(60) {
    foreach (p in procfdcount- limit 20) {
        printf("%d %lu\n", p, procfdcount[p])
    }
    printf("\n")
}

Run via something like:

... (install systemtap here) ...
# stap-prep
... (fix any reported systemtap issues here) ...
# stap procfdcount.stp

The downside of this method is the need to identify all "open files" (sockets, etc) and then to adjust the count via appropriate system call hooks (if available); the above only tracks file files. Another option would be to call task_open_file_handles for tasks that get onto the CPU, and to display the most recent of those counts periodically.

global taskopenfh

probe begin {
    printf("begin trace...\n\n");
}

probe scheduler.cpu_on {
    p = pid();
    if (p == 0) next;
    taskopenfh[p] = task_open_file_handles(pid2task(p));
}

probe timer.s(60) {
    foreach (p in taskopenfh-) {
        printf("%d %lu\n", p, taskopenfh[p]);
    }
    delete taskopenfh;
    printf("\n");
}

This would though miss anything that was not on the CPU; a full walk of the processes and then tasks would be necessary for a complete list, though that might be too slow or too expensive if you've got millions of FDs.

Also these probes don't appear to be stable, so maybe eBPF or something in the future? E.g. the second one on Centos 7 blows up after some time on

ERROR: read fault [man error::fault] at 0x0000000000000008 (((&(fs->fdt))))
near identifier 'task_open_file_handles' at
/usr/share/systemtap/tapset/linux/task.stp:602:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.