5

I want to detect errors in application's execution logic. E.g.:

  • forgot to call free() on address returned by malloc()
  • did not close file handle returned by open()
  • invalid flags passed to open()
  • invalid file handle passed to poll()
  • write() called on fd that wasn't opened for writing
  • pass invalid flags to open() e.g. open("/etc/fstab", 4)
  • calling close() on an invalid fd
  • ...

I think there are hundreds more.

Maybe the tool can be run similar to ftrace or strace, but a kernel log containing the faulty calls would be sufficient too.

5
  • - forgot to call free() on address returned by malloc() - This is not about the kernel, but about memory management. did not close file handle returned by open() - This is not strictly necessary. You've lumped together very different classes of errors, the first two are programming errors, not kernel errors. Jul 6 at 9:41
  • Do you want to do this statically or dynamically? Jul 7 at 7:02
  • @JörgWMittag What do you mean?
    – somega
    Jul 7 at 10:07
  • Do you want to detect these conditions as they are happening at runtime or do you want to detect them without running the program? Jul 7 at 19:59
  • 1
    @JörgWMittag At runtime. I think a static tool cannot know for example which arguments will be passed to open() or whether free() will be called.
    – somega
    Jul 8 at 14:18

3 Answers 3

9

forgot to call free() on address returned by malloc()

Well, malloc and free aren't kernel calls! What malloc() (which is a libc library function, normal user process code!) does

  1. look up in the memory pool it keeps whether there's an available chunk of the requested size
  2. if so, mark it as used and return it to the program, if not, call sbrk (or equivalently, mallocing of anonymous memory was common) to ask the kernel for an amount of new virtual memory pages, add these to the pool, and then satisfy the program's request.

free just takes s piece of memory previously returned through malloc; if so, it marks it as unused in the memory pool. (If not, undefined behaviour happens, but most libc's will abort at that point.) Most implementations of free don't ever try to even return the memory to the OS!

Now, if you want memory sanitation, there's tools (valgrind, gcc -fsanitize and more) that watch these free and malloc calls, and even trace whether the address of a malloced piece of memory is still "saved" somewhere in the program, or whether, e.g. at the end of a function, the pointer holding that address just ceases to exist, so that nobody can possibly remember that the memory was allocated. That would be an actual fault; just not immediately freeing memory, or deferring the freeing to the end of the program is not a problem, at all. The whole point of malloc is that you get memory with a potentially infinite lifetime! (hint: if you worry about these kinds of things, and you'd be right to, don't write C. Write in a language that allows for object life times to be tracked properly. That would be languages like Rust, or C++, but the latter only if not taught by someone who thinks of C++ as extension to C. I have large programs where I never once used new or worse, malloc in my C++ code. Smart pointers can take a lot of the pitfalls of your shoulders, even in C++, which very much allows you to do manual memory control, but in modern variants also very much encourages you not to by offering zero-cost object lifetime tracking. )

did not close file handle returned by open()

That's not a problem! Even more than with memory, it's perfectly acceptable and even sensible to keep files open till the end of a program; for example, locks on files wouldn't work if you relinquish them right away. And a control interface would need to be kept open until the program shuts down.

Again, if you're worried that within your program's control flow, you might be opening thousands of files and forget to close them, don't write in C, but in a language where a file handle has a life time and can close the underlying file descriptor when not needed anymore.

Just: "there's a file opened, it has not yet been closed" simply isn't a problem, especially not on POSIX systems, where concurrent file access is normal and in many aspects even well-defined.

invalid flags passed to open()

How do you know that, other than things that return an error code anyways?

I mean, it's very normal for a library to check whether a file can be opened "write + append" mode, but it's not a problem if it can't be.

If you want to observe any time a system call is made, get its arguments and what it "returns" to user land, the ptrace syscall is your friend, as e.g. used by the popular strace program. Other options involve writing eBPF probes or uprobes, which can be used for very efficient and even "intelligently filtering" logging of such things.

invalid file handle passed to poll()

Same problem as before, this might just be your program checking whether a file handle can be polled; that's not the case for all (pseudo-) file systems.

Additionally, poll is actually also the name of a wrapper function (symbol) supplied at least by glibc if necessary, and "invalid" to that might be different than "invalid" to the poll syscall.

10
  • 1
    +1 for sanitizers as I believe its the only right way to go. Rust offers quite a lot. I can only remember times when I would never run my programs before passing them through lint.
    – MC68020
    Jul 6 at 11:13
  • “it's perfectly acceptable and even sensible to keep files open till the end of a program” — Won't that depend on the pattern of use? It should be fine to keep the same (limited number of) files open continually; but if if opens more files the longer it runs, then it had better close them at the same rate, else it'll cause trouble sooner or later! (And since the question asks about an ‘application’, it seems reasonable to assume it could be running for an indefinite period.)
    – gidds
    Jul 6 at 20:47
  • @gidds that's very true! Point is that you need some analysis, and domain knowledge of what the application does, to know whether a usage pattern is problematic. I'll go as far as saying the pure observation of files opened is at best a small puzzle piece :) Jul 7 at 6:26
  • @MarcusMüller Agreed! But I think small pieces of the puzzle are the best we can hope for — AIUI, the full question as posed would be on a par with solving the halting problem
    – gidds
    Jul 7 at 8:10
  • @MarcusMüller I have added some more examples. Maybe you can say something about them. For example when the program calls open("/etc/fstab", 4) it's 100% safe to say that there is something wrong in the program. (Because 4 is not a valid flag.) How to detect something like this? Use strace?
    – somega
    Jul 7 at 10:21
4

The first of these, failing to free a block of memory allocated by malloc(), doesn’t necessarily involve the kernel — memory allocations are handled by the C library. Valgrind’s Memcheck can detect these.

To trace errors returned by the kernel, you can run your program with strace -Z (available since strace 5.2): this will only trace system calls which return an error. You’ll still need to post-process the result to look for example for EBADFD, EINVAL etc.

Note that failing to call free() or close() isn’t necessarily an error; such resources are freed when the program exits anyway, so there are cases where it’s perfectly acceptable to not relinquish resources explicitly.

3
  • It's worth pointing out that once you go as low-level as strace -Z, you've kind of departed from the stated goal of "detect[ing] errors in application's execution logic", as a kernel-call error could come from anywhere, not just the application. (By which I mean, the kernel call that returns an error could be made by a function in a shared library loaded by a program module dynamically-loaded by another shared library that's linked with the application in question.) Some applications are directly responsible for very few of the kernel calls made on their behalf.
    – FeRD
    Jul 7 at 13:29
  • 1
    @FeRD yes, the question changed substantially (in intent) since I wrote this answer. It was initially titled “Tool to detect faulty kernel calls”. Jul 7 at 13:31
  • @StephenKitt Understood. Honestly I think the question has lost all (useful) focus, which is really what I was responding to. The answer is perfectly good for what it is. The question, in its current form, reminds me of the Babbage anecdote about people asking him whether his Analytical Engine would produce correct answers if they input wrong data.
    – FeRD
    Jul 7 at 13:34
0

There are various static code analysis tools available to find common coding errors like this. I can't say if it covers all of your scenarios but SonarQube is one such tool with support for the C language: https://www.sonarqube.org/features/multi-languages/c

There are hundreds of SonarQube rules for C: https://rules.sonarsource.com/c

SonarQube - C - Dynamically allocated memory should be released

If static code analysis isn't sufficient, you may need dynamic analysis to examine the running program: https://en.wikipedia.org/wiki/Dynamic_program_analysis

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.