127

Start your bash script with bash -x ./script.sh or add in your script set -x to see debug output. Additional with bash 4.1 or later: If you want to write the debug output to a separate file, add this to your script: exec 5> debug_output.txt BASH_XTRACEFD="5" See: https://stackoverflow.com/a/25593226/3776858 If you want to see line numbers add this: ...


109

I think, general principles of network troubleshooting are: Find out at what level of TCP/IP stack(or some other stack) occurs the problem. Understand what is the correct system behavior, and what is deviation from normal system state Try to express the problem in one sentence or in several words Using obtained information from buggy system, your own ...


62

A segmentation fault is the result of a memory access violation. The program has referred to a memory address outside of what was allocated to it, and the OS kernel responds by killing the program with SIGSEGV. This is a mistake, since there is no point in trying to access inaccessible memory (it cannot be done). Mistakes of this sort are easy to make, ...


55

You can force the crontab to run with following command: run-parts /etc/cron.daily


50

Using set -x I always use set -x and set +x. You can wrap areas that you want to see what's happening with them to turn verbosity up/down. #!/bin/bash set -x ..code to debug... set +x log4bash Also if you've done development work and are familiar with the style of loggers that go by the names log4j, log4perl, etc., then you might want to make use of ...


43

My first step would be to run strace on the process, best strace -s 99 -ffp 12345 if your process ID is 12345. This will show you all syscalls the program is doing. How to strace a process tells you more. If you insist on getting a stacktrace, google tells me the equivalent is pstack. But as I do not have it installed I use gdb: tweedleburg:~ # sleep ...


40

You can simulate the cron user environment as explained in "Running a cron job manually and immediately". This will allow you to test the job works when it would be run as the cron user. Excerpt from link: Step 1: I put this line temporarily in the user's crontab: * * * * * /usr/bin/env > /home/username/tmp/cron-env then took it out once the file ...


40

I knew I was grasping at straws, but UNIX never fails! Here's how I managed it: bash$ gdb --pid 8909 ... Loaded symbols for /lib/i386-linux-gnu/i686/cmov/libnss_files.so.2 0xb76e7424 in __kernel_vsyscall () Then at the (gdb) prompt I ran the command, call write_history("/tmp/foo") which will write this history to the file /tmp/foo. (gdb) call ...


34

There's a bash debugger, bashdb, which is an installable package on many distros. It uses bash's built-in extended debugging mode (shopt -s extdebug). It looks a lot like gdb; here's a sample session to give some flavor: $ ls 1st.JPG 2ndJPG.JPG $ cat ../foo.sh for f in *.JPG do newf=${f/JPG/jpg} mv $f $newf done $ bashdb ../foo.sh (foo.sh:1): 1: ...


33

Two answers have been given for finding the stack trace of a program (remember to install debugging symbols first!). If you want to find out where a system call got stuck, examine /proc/PID/stack, which lists the kernel stack. Example: $ cat /proc/self/stack [<ffffffff81012b72>] save_stack_trace_tsk+0x22/0x40 [<ffffffff81213abe>] proc_pid_stack+...


32

Use ip route get. From Configuring Network Routing : The ip route get command is a useful feature that allows you to query the route on which the system will send packets to reach a specified IP address, for example: # ip route get 23.6.118.140 23.6.118.140 via 10.0.2.2 dev eth0 src 10.0.2.15 cache mtu 1500 advmss 1460 hoplimit 64 ...


30

help running provides some hints: There are step and next instuctions (and also nexti and stepi). (gdb) help next Step program, proceeding through subroutine calls. Usage: next [N] Unlike "step", if the current source line calls a subroutine, this command does not enter the subroutine, but instead steps over the call, in effect treating it as a single ...


29

Use set -x in the shell. $ alias hello='echo hello world!' $ hello hello world! $ set -x $ hello + echo hello world! hello world! Using set -x turns on the xtrace shell option (set +x turns it off) and should work in all Bourne-like shells, like bash, dash ksh93, pdksh and zsh. This prompts the shell to display the command that gets executed after alias ...


28

As far as i know there is no way to directly do that as cron has a special purpose - running schedules commands at a specific time. So the best thing is to either to manually create a crontab entry or write a script which removes and resets the environment.


28

On most unix systems, you can use GDB. gdb -batch -ex bt -p 1234 There's also pstack (not a standard utility, you'll probably have to install it manually). It looks like an equivalent of AIX's procstack. But on my Debian wheezy amd64, it seems to always error out. On i386, for a program compiled without debugging symbols, it doesn't print any symbol, not ...


27

How about: readelf -p .comment a.out


23

You can stop both processing by sending them SIGSTOP (replace pid1 and pid2 by the actual PIDs or use killall and the application name): kill -SIGSTOP pid1 pid2 The printing on the terminal (or wherever stdout is redirected to) should stop. Then continue one of them using kill -SIGCONT pid1 If the error messages appear immediately, you know its the ...


23

There's often confusion between process forking and execution. When you do at the prompt of a bash shell. $ sh -c 'exec env ps' The process P1 issuing that $ prompt is currently running bash code. That bash code forks a new process P2 that executes /bin/sh which then executes /usr/bin/env, which then executes /bin/ps. So P2 has in turn executed code of ...


22

You can only debug a setuid or setgid program if the debugger is running as root. The kernel won't let you call ptrace on a program running with extra privileges. If it did, you would be able to make the program execute anything, which would effectively mean you could e.g. run a root shell by calling a debugger on /bin/su. If you run Gdb as root, you'll be ...


20

The standard method to debug scripts in most Bourne-based shells, like bash is to write set -x at the top of your script. This will make bash more verbose about what's being done/executed, and how arguments are evaluated. -x Print commands and their arguments as they are executed. this is useful for either, the interpreter or inside scripts. For example: ...


19

xtrace output goes to stderr, so you could redirect stderr to /dev/null: i_know_what_this_does() { echo do stuff } 2> /dev/null If you still want to see the errors from the commands run inside the functions, you could do i_know_what_this_does() ( { set +x; } 2> /dev/null # silently disable xtrace echo do stuff ) Note the use of (...) instead ...


17

On Linux, assuming you want to know what is writing to the same resource as your shell's stdout is connected to, you could do: strace -fe write $(lsof -t "/proc/$$/fd/1" | sed 's/^/-p/') That would report the write() system calls (on any file descriptor) of every process that have at least one file descriptor open on the same file as fd 1 of your shell.


16

objdump + gdb minimal runnable example TLDR: objdump -s core to dump memory GDB to find failing line, previously mentioned at: How to view core files for debugging purposes in Linux? Now for a the full educational test setup: main.c #include <stddef.h> #include <stdio.h> #include <stdlib.h> #include <string.h> int myfunc(int i) {...


15

gdb is the GNU debugger which can be used to examine the core file. BTW bt (backtrace) is a useful gdb command to examine the program call stack. gdb binary-file core-file


14

A surprising number of "network problems" boil down to DNS problems of one kind or another. Initial troubleshooting should use ping -n w.x.y.z in order to leave out DNS resolution of a hostname, and just check IP connectivity. After that, use route -n to check the default IP route without DNS resolution. After verifying IP connectivity, and routing, ...


14

If you think about how strace works then it makes total sense that none of the builtins to Bash would be traceable. strace can only trace actual executables, whereas the builtins are not. For example, my cd command: $ type cd cd is a function cd () { builtin cd "$@"; local result=$?; __rvm_project_rvmrc; __rvm_after_cd; return $...


14

Many of the flags that can be passed to bash on the command line are set flags. set is the shell built-in which can toggle these flags at runtime. For example, calling a script as bash -x foo.sh is basically the same as doing set -x at the top of the script. Knowing that set is the shell built-in responsible for this lets us know where to look. Now we can ...


14

gdb will ask you to confirm certain commands, if the value of the confirm setting is on. From Optional Warnings and Messages: set confirm off Disables confirmation requests. Note that running GDB with the --batch option (see -batch) also automatically disables confirmation requests. set confirm on Enables confirmation requests (the default). ...


14

However, are there any other clever tools/methods to see if process listening on TCP port receives a message? You can use strace with -e trace=network. This is what it prints on accepting a TCP connection, receiving an HTTP request, sending an HTTP response and closing the connection: $ strace -v -f -e trace=network -p `cat logs/my_server.pid` Process ...


14

You could use shell-expand-line, which is bound to Control-Alt-e by default: $ bind -p | grep shell-expand-line "\e\C-e": shell-expand-line Among other things, it will replace aliases in the current line with their definition so you can see a command you're still going to run. Example: $ install dicelab # now press C-Alt-e $ sudo apt-get install dicelab #...


Only top voted, non community-wiki answers of a minimum length are eligible