Best practices for attaching debug symbols to system libraries during development?

Question

I'm at a stage in a project where it would be really handy to have a debug version of a system package installed. On Ubuntu at least, adding the debug symbols to a library is a piece of cake. Practically every package has a -dbg variant that provides all the symbols you'd need for a useful backtrace.

However, I'm currently on Arch linux, where the general concensus is to edit the user makepkg.conf file, adding whatever debug flags to (C|CXX|CPP|LD)FLAGS. Then re-build the package yourself, and replace the current, optimised version with the debug build. Well, I suppose that's fair enough with a "source-based distribution", but it gets tedious pretty quickly.

So, what is / are the best practices for attaching debug symbols to a system package? How do other packagers do it?

I think I've seen that strip can extract debug symbols and save them in external files. Is it possible for gdb to pick up those symbol files during backtraces, with system applications not even bothering to look for them? How does that work, from a packagers perspective?

It's just an idea, but is it a good idea to create a chroot environment in which to develop? (I have a problem atm where a package has an ABI incompatibility between its debug and release builds, which is a bit of a pain. Everything linked to its shared lib also complains about missing symbols, so reverted to optimised build..)

Answer 1

If you a distributing a source package, the (autotools) norm is to compile the debugging symbols in by default.

I think that once upon a time mainstream linux distros left them in binaries too; I may be wrong about that. There is a misconception that removing debugging symbols "optimizes" software. It doesn't. The only difference including debugging symbols makes is the space a file takes up on disk. It does not affect the memory usage, since they are not loaded into memory during normal use (so, it also does not affect anything else). Try profiling a stripped and unstripped binary. They're the same.

The purpose of splitting them out of distro packages is just to reduce the size of each package so that the entire install is, eg., 2.5 GB instead of 3.8 GB or whatever. If your package gets picked up for inclusion in an official repository, the distribution will package it from source. They won't use a package you premade, so you doing this work now (creating a separate debugging package) will not make a difference in that respect.

If you are independently distributing library binary packages for various distros, nobody is going to care if debugging symbols are compiled in and most people who program with the library are going to want them. For the few people who are bothered for some strange reason, they are easy to strip anyway.

So, if you want my opinion as a programmer and linux user, just leave them in, at least for now. An obvious preoccupation with "premature optimization" -- especially, premature optimizations that aren't really optimizations -- doesn't look good. In other words, the literal answer to your question is, "The best practice for attaching debugging symbols to system libraries during development is to compile them in."

That said, I did notice this page WRT .deb packages when I was trying to confirm my belief that once upon a time they were always included anyway. Since you included dpkg in your tags, it might be useful to you.

Answer 2

Adding OPTIONS+=(debug !strip) adds this (from /etc/makepkg.conf) to your build options: DEBUG_CFLAGS="-g -fvar-tracking-assignments".

Neither of these disables any optimizations (https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html). You get an optimized build with debug symbols, not what many people mean when they talk about a "debug build". To get that, you'd also enable -O0 or -Og ("optimize for debugging".)

When you debug with gdb, often print some_local will give you (optimized out), because the debug format can't track variables that are live in registers (not spilled to memory). Of course, even a perfect debug format couldn't really fix cases where a variable really was optimized out, and no register or memory holds a value matching the C source. You can still (fairly) reliably get backtraces, and function args to functions that didn't inline.

---

https://wiki.archlinux.org/index.php/Debug_-_Getting_Traces currently says that you can now use (debug strip) to get a separate somepkg-debug package that puts symbol info in /usr/lib/debug. (Like what Debian/Ubuntu distribute as somepkg-dbg.) IDK if that was the case in 2013 when you asked this.

You can't of course use that debug package with an existing binary package, because any tiny difference in anything could lead to wrong debug info.

Unfortunately, there's no repo / package-system for distributing debug packages for pre-built binary packages. Thus, you still need to locally compile any package you want debug symbols for.

On the plus side, this is a good chance to enable -march=native to make binaries optimized specifically for your system. e.g. enable everything your CPU supports, like BMI2 for more efficient variable-count shift instructions, hardware popcnt, and AVX / AVX2 / FMA / AVX512 vector instructions. -march=native also sets -mtune=native, which is good.

The perf overhead of leaving debug info in the same file as libraries should be negligible. The whole /usr/lib/libc.so.6 doesn't get loaded into RAM, only the pages that are needed get mapped. The debug info is grouped together inside the binary so those pages probably stay cold on disk.