Tool to detect errors in application's execution logic

Question

I want to detect errors in application's execution logic. E.g.:

• forgot to call free() on address returned by malloc()
• did not close file handle returned by open()
• invalid flags passed to open()
• invalid file handle passed to poll()
• write() called on fd that wasn't opened for writing
• pass invalid flags to open() e.g. open("/etc/fstab", 4)
• calling close() on an invalid fd
• ...

I think there are hundreds more.

Maybe the tool can be run similar to ftrace or strace, but a kernel log containing the faulty calls would be sufficient too.

Answer 1

forgot to call free() on address returned by malloc()

Well, malloc and free aren't kernel calls! What malloc() (which is a libc library function, normal user process code!) does

1. look up in the memory pool it keeps whether there's an available chunk of the requested size
2. if so, mark it as used and return it to the program, if not, call sbrk (or equivalently, mallocing of anonymous memory was common) to ask the kernel for an amount of new virtual memory pages, add these to the pool, and then satisfy the program's request.

free just takes s piece of memory previously returned through malloc; if so, it marks it as unused in the memory pool. (If not, undefined behaviour happens, but most libc's will abort at that point.) Most implementations of free don't ever try to even return the memory to the OS!

Now, if you want memory sanitation, there's tools (valgrind, gcc -fsanitize and more) that watch these free and malloc calls, and even trace whether the address of a malloced piece of memory is still "saved" somewhere in the program, or whether, e.g. at the end of a function, the pointer holding that address just ceases to exist, so that nobody can possibly remember that the memory was allocated. That would be an actual fault; just not immediately freeing memory, or deferring the freeing to the end of the program is not a problem, at all. The whole point of malloc is that you get memory with a potentially infinite lifetime! (hint: if you worry about these kinds of things, and you'd be right to, don't write C. Write in a language that allows for object life times to be tracked properly. That would be languages like Rust, or C++, but the latter only if not taught by someone who thinks of C++ as extension to C. I have large programs where I never once used new or worse, malloc in my C++ code. Smart pointers can take a lot of the pitfalls of your shoulders, even in C++, which very much allows you to do manual memory control, but in modern variants also very much encourages you not to by offering zero-cost object lifetime tracking. )

did not close file handle returned by open()

That's not a problem! Even more than with memory, it's perfectly acceptable and even sensible to keep files open till the end of a program; for example, locks on files wouldn't work if you relinquish them right away. And a control interface would need to be kept open until the program shuts down.

Again, if you're worried that within your program's control flow, you might be opening thousands of files and forget to close them, don't write in C, but in a language where a file handle has a life time and can close the underlying file descriptor when not needed anymore.

Just: "there's a file opened, it has not yet been closed" simply isn't a problem, especially not on POSIX systems, where concurrent file access is normal and in many aspects even well-defined.

invalid flags passed to open()

How do you know that, other than things that return an error code anyways?

I mean, it's very normal for a library to check whether a file can be opened "write + append" mode, but it's not a problem if it can't be.

If you want to observe any time a system call is made, get its arguments and what it "returns" to user land, the ptrace syscall is your friend, as e.g. used by the popular strace program. Other options involve writing eBPF probes or uprobes, which can be used for very efficient and even "intelligently filtering" logging of such things.

invalid file handle passed to poll()

Same problem as before, this might just be your program checking whether a file handle can be polled; that's not the case for all (pseudo-) file systems.

Additionally, poll is actually also the name of a wrapper function (symbol) supplied at least by glibc if necessary, and "invalid" to that might be different than "invalid" to the poll syscall.

Answer 2

The first of these, failing to free a block of memory allocated by malloc(), doesn’t necessarily involve the kernel — memory allocations are handled by the C library. Valgrind’s Memcheck can detect these.

To trace errors returned by the kernel, you can run your program with strace -Z (available since strace 5.2): this will only trace system calls which return an error. You’ll still need to post-process the result to look for example for EBADFD, EINVAL etc.

Note that failing to call free() or close() isn’t necessarily an error; such resources are freed when the program exits anyway, so there are cases where it’s perfectly acceptable to not relinquish resources explicitly.

Answer 3

There are various static code analysis tools available to find common coding errors like this. I can't say if it covers all of your scenarios but SonarQube is one such tool with support for the C language: https://www.sonarqube.org/features/multi-languages/c

There are hundreds of SonarQube rules for C: https://rules.sonarsource.com/c

If static code analysis isn't sufficient, you may need dynamic analysis to examine the running program: https://en.wikipedia.org/wiki/Dynamic_program_analysis