174

Start your bash script with bash -x ./script.sh or add in your script set -x to see debug output. Additional with bash 4.1 or later: If you want to write the debug output to a separate file, add this to your script: exec 5> debug_output.txt BASH_XTRACEFD="5" See: https://stackoverflow.com/a/25593226/3776858 If you want to see line numbers add this: ...


107

You can force the crontab to run with following command: run-parts /etc/cron.daily


82

Using set -x I always use set -x and set +x. You can wrap areas that you want to see what's happening with them to turn verbosity up/down. #!/bin/bash set -x ..code to debug... set +x log4bash Also if you've done development work and are familiar with the style of loggers that go by the names log4j, log4perl, etc., then you might want to make use of ...


79

You can simulate the cron user environment as explained in "Running a cron job manually and immediately". This will allow you to test the job works when it would be run as the cron user. Excerpt from link: Step 1: I put this line temporarily in the user's crontab: * * * * * /usr/bin/env > /home/username/tmp/cron-env then took it out once the file ...


71

A segmentation fault is the result of a memory access violation. The program has referred to a memory address outside of what was allocated to it, and the OS kernel responds by killing the program with SIGSEGV. This is a mistake, since there is no point in trying to access inaccessible memory (it cannot be done). Mistakes of this sort are easy to make, ...


66

help running provides some hints: There are step and next instuctions (and also nexti and stepi). (gdb) help next Step program, proceeding through subroutine calls. Usage: next [N] Unlike "step", if the current source line calls a subroutine, this command does not enter the subroutine, but instead steps over the call, in effect treating it as a single ...


48

I knew I was grasping at straws, but UNIX never fails! Here's how I managed it: bash$ gdb --pid 8909 ... Loaded symbols for /lib/i386-linux-gnu/i686/cmov/libnss_files.so.2 0xb76e7424 in __kernel_vsyscall () Then at the (gdb) prompt I ran the command, call write_history("/tmp/foo") which will write this history to the file /tmp/foo. (gdb) call ...


48

My first step would be to run strace on the process, best strace -s 99 -ffp 12345 if your process ID is 12345. This will show you all syscalls the program is doing. How to strace a process tells you more. If you insist on getting a stacktrace, google tells me the equivalent is pstack. But as I do not have it installed I use gdb: tweedleburg:~ # sleep ...


47

Use ip route get. From Configuring Network Routing : The ip route get command is a useful feature that allows you to query the route on which the system will send packets to reach a specified IP address, for example: # ip route get 23.6.118.140 23.6.118.140 via 10.0.2.2 dev eth0 src 10.0.2.15 cache mtu 1500 advmss 1460 hoplimit 64 ...


41

Two answers have been given for finding the stack trace of a program (remember to install debugging symbols first!). If you want to find out where a system call got stuck, examine /proc/PID/stack, which lists the kernel stack. Example: $ cat /proc/self/stack [<ffffffff81012b72>] save_stack_trace_tsk+0x22/0x40 [<ffffffff81213abe>] proc_pid_stack+...


38

There's a bash debugger, bashdb, which is an installable package on many distros. It uses bash's built-in extended debugging mode (shopt -s extdebug). It looks a lot like gdb; here's a sample session to give some flavor: $ ls 1st.JPG 2ndJPG.JPG $ cat ../foo.sh for f in *.JPG do newf=${f/JPG/jpg} mv $f $newf done $ bashdb ../foo.sh (foo.sh:1): 1: ...


32

Use set -x in the shell. $ alias hello='echo hello world!' $ hello hello world! $ set -x $ hello + echo hello world! hello world! Using set -x turns on the xtrace shell option (set +x turns it off) and should work in all Bourne-like shells, like bash, dash ksh93, pdksh and zsh. This prompts the shell to display the command that gets executed after alias ...


28

On most unix systems, you can use GDB. gdb -batch -ex bt -p 1234 There's also pstack (not a standard utility, you'll probably have to install it manually). It looks like an equivalent of AIX's procstack. But on my Debian wheezy amd64, it seems to always error out. On i386, for a program compiled without debugging symbols, it doesn't print any symbol, not ...


25

You can stop both processing by sending them SIGSTOP (replace pid1 and pid2 by the actual PIDs or use killall and the application name): kill -SIGSTOP pid1 pid2 The printing on the terminal (or wherever stdout is redirected to) should stop. Then continue one of them using kill -SIGCONT pid1 If the error messages appear immediately, you know its the ...


24

objdump + gdb minimal runnable example TLDR: objdump -s core to dump memory GDB to find failing line, previously mentioned at: How to view core files for debugging purposes in Linux? Now for a the full educational test setup: main.c #include <stddef.h> #include <stdio.h> #include <stdlib.h> #include <string.h> int myfunc(int i) {...


23

There's often confusion between process forking and execution. When you do at the prompt of a bash shell. $ sh -c 'exec env ps' The process P1 issuing that $ prompt is currently running bash code. That bash code forks a new process P2 that executes /bin/sh which then executes /usr/bin/env, which then executes /bin/ps. So P2 has in turn executed code of ...


22

gdb will ask you to confirm certain commands, if the value of the confirm setting is on. From Optional Warnings and Messages: set confirm off Disables confirmation requests. Note that running GDB with the --batch option (see -batch) also automatically disables confirmation requests. set confirm on Enables confirmation requests (the default). ...


22

First, I'm afraid that explanation of -o option served by http://explainshell.com is not entirely correct. Given that set is a bulit-in command, we can see its documentation with help by executing help set: -o option-name Set the variable corresponding to option-name: allexport same as -a braceexpand same as -B ...


20

On Linux, assuming you want to know what is writing to the same resource as your shell's stdout is connected to, you could do: strace -fe write $(lsof -t "/proc/$$/fd/1" | sed 's/^/-p/') That would report the write() system calls (on any file descriptor) of every process that have at least one file descriptor open on the same file as fd 1 of your shell.


19

The standard method to debug scripts in most Bourne-based shells, like bash is to write set -x at the top of your script. This will make bash more verbose about what's being done/executed, and how arguments are evaluated. -x Print commands and their arguments as they are executed. this is useful for either, the interpreter or inside scripts. For example: ...


18

If you think about how strace works then it makes total sense that none of the builtins to Bash would be traceable. strace can only trace actual executables, whereas the builtins are not. For example, my cd command: $ type cd cd is a function cd () { builtin cd "$@"; local result=$?; __rvm_project_rvmrc; __rvm_after_cd; return $...


17

You probably don’t want to “solve” this problem; according to the Debian glibc manpage for ld.so, /etc/ld.so.nohwcap When this file is present the dynamic linker will load the non-optimized version of a library, even if the CPU supports the optimized version. It’s not installed by a package, it can be created by the system administrator to ...


16

Many of the flags that can be passed to bash on the command line are set flags. set is the shell built-in which can toggle these flags at runtime. For example, calling a script as bash -x foo.sh is basically the same as doing set -x at the top of the script. Knowing that set is the shell built-in responsible for this lets us know where to look. Now we can ...


16

if your libraries are not on standard path then either you need to add them to the path or add non-standard path to LD_LIBRARY_PATH export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<Your_non-Standard_path> Once you done any one of above things then you need to update the dynamic linker run-time binding by executing below command: sudo ldconfig UPDATE: You ...


16

This is a matter of opinion, but I think set -eu is a bad idea for interactive use. Here’s what each one does: set -e causes the shell to exit whenever a pipeline, list, or compound command exits with non-zero status; set -u causes the shell to exit whenever it attempts to expand a parameter which isn’t defined. In an interactive shell, this means that any ...


16

As fra-san mentioned, GNU sed introduced a --debug option which does pretty much what you’re looking for, in version 4.6; so e.g if you run: printf '%s\n' one two | sed --debug 'H;1h;$x;$s/\n/_/g' the output is SED PROGRAM: H 1 h $ x $ s/\n/_/g INPUT: 'STDIN' line 1 PATTERN: one COMMAND: H HOLD: \none COMMAND: 1 h HOLD: one COMMAND: $ x ...


15

gdb is the GNU debugger which can be used to examine the core file. BTW bt (backtrace) is a useful gdb command to examine the program call stack. gdb binary-file core-file


15

I don't think there's a builtin command equivalent, but some combination of these four variables from the zsh/Parameter module can be used: funcfiletrace This array contains the absolute line numbers and corresponding file names for the point where the current function, sourced file, or (if EVAL_LINENO is set) eval command was called. The array is ...


15

When I want to do this, I use the ftrace framework. Start by mounting the special file system: mount -t tracefs nodev /sys/kernel/tracing (as root; you should become root for all this, you’ll be doing everything as root anyway, and it’s easier to have a root shell than to use sudo). Then change to that directory: cd /sys/kernel/tracing It contains a ...


Only top voted, non community-wiki answers of a minimum length are eligible