As Linux/Unix users, we run into problems frequently. And after long hours of problem solving, we develop the skill of debugging.
Now, what are good principles, methods or best practises when trying to debug general unix problems?
What tools should I have, as an average user, to make finding the cause of problems easier?
The methods will depend on the kind of the problem.
In general "How To Ask Questions The Smart Way" by Eric S. Raymond and Rick Moen is sometimes a helpful advice to focus on the problem and to check if you have thought about important parts of the problem.
Your first source of information during debugging are the logfiles your system/application writes. The common place for them is your terminal or a file in /var/log/. Many applications support different types of loglevels which you should increase if you can't find any usable messages. Often there is an -v verbose switch to get more messages.
Still nothing usable? Check your configuration files, permissions of the files needed by the application and maybe you have to change the config of your systemlogger, for example /etc/syslog-ng.conf.
If you have an error message, a google search will often lead to message board entries or usenet postings discussing the problems behind it. Its likely that you can find a solution there. A project users mailinglist, message boards and IRC channels can also be very helpful.
Sometimes applications crash without any messages. A great tool to discover the application flow , beside reading and modifying the code is strace.
This tool will trace system calls and signals. When errors are caught by the application, you can still discover problems in the systrace.
Another approach would be debugging the application with gdb. You should be an advanced user and know what to do, to use this.
If you want a single, general principle for debugging, it would be this: Understand how the system works, as much as you can. Understand each component of the system, and the failure modes of each component. Be aware of which components you have changed recently, and which components may have changed or failed on their own.
If you were looking for specifics, then echox's answer has lots of good information.
David Agans wrote a very nice book on debugging, in my opinion. It also contained a set of guidelines for debugging.
On top of that, general (domain) knowledge and experience are a great help in seeing patterns. Study how things are built, take them apart. Perform routine maintenance. Set up obscure experiments. Read, read, read. Make stuff. Write constantly. Help other people with their problems. Pick your battles. Keep calm. Smile. :)
