BASH Programming

How to Debug a Bash Script like a Boss

A lot of things can happen in a bash script that unless you know how to debug like a boss, you might as well be up a creek without a paddle. Here a short list of debugging techniques followed by examples that can be applied to any issue in bash programming.

  • Use options (bash specific)
  • Use breakpoints
  • Use debug output
  • Use shopts (bash specific)

Use options to debug scripts like a boss (bash specific)

Bash comes with a set of options (let’s just say 25 just to keep things simple) to allow for modification of script behavior if the need presents itself. In case you are about to debug a bash script it does. Have no fear, there only 3 that you will need to get started. The other will come into play later.

Here is a simple bash script using options for debugging.

#!/bin/bash
## debug-options
## version 0.0.1 - initial
##################################################
debug-options() {
#################################################
echo ${FUNCNAME^^}
#################################################
## 1. set -x
## 1. set -v
## 1. set -e
#################################################
## 1. set -x, set -v          #
test 0 -eq 0      # were invisible #
echo -n ""        #           #
set -x -v          # wax on       #
test 0 -eq 0    # now are visible # + test 0 -eq 0
echo -n ""      #                           # + echo -n ''
set +x +v       # wax off              # + set +x +y
###########################################
## 1. set -e
set -e
test ! 0 -eq 0 || true # does not echo done
# i.e. exits with status 1
# without || true
test ! 0 -eq 0 || {
# may provide useful debug info here before
# exiting script with exit status as error
false || true # without || true
}
set +e
##################################################
echo done
##################################################
}
##################################################
if [ ${#} -eq 0 ]
then
true
else
exit 1 # wrong args
fi
##################################################
debug-options
##################################################
## generated by create-stub2.sh v0.1.2
## on Mon, 20 May 2019 21:59:34 +0900
## see <https://github.com/temptemp3/sh2>
############################################

Source: debug-options.sh

Now consider the following script that may not succeed. It is simple but perfect to help practice bash script debugging.

#!/bin/bash
## debug-random-error
## version 0.0.1 - initial
##################################################
debug-random-error() {
test $(( ${RANDOM} % 2 )) -eq 0
echo success
}
##################################################
if [ ${#} -eq 0 ]
then
true
else
exit 1 # wrong args
fi
##################################################
debug-random-error
##################################################
## generated by create-stub2.sh v0.1.2
## on Mon, 20 May 2019 21:40:46 +0900
## see <https://github.com/temptemp3/sh2>
##################################################

Source: debug-random-error.sh

When you are debugging a bash script don’t be afraid to ask yourself what-if.

Example i) What-if the program exits with an exit status of 0 but the outcome says otherwise

What-if there is an explosion in one of your functions but the script keeps on chugging. In fact, before anything big gets lit up, simple test will fail. The set -e option command will cause your script to halt on any failed test and exit with an exit status as error.

The question now is where to introduce the exit on error option command while debugging.

There are 3 options.

Option i) Modify the script, interpreter line

#!/bin/bash -e
## debug-random-error
## ...

Now you run the script 10 times.

for i in {1..10} ; do bash debug-random-error.sh ; done

And get the following results.

success
success
success
success
success
success
success
success
success
success

You are really lucky or something is wrong. It is more likely that something is wrong.

It turns out that this errexit depends on the script being run as an executable following the chmod +x command.

Now we run the script 10 times again. Feeling lucky?

for i in {1..10} ; do ./debug-random-error.sh ; done
success
success
success
success
success

Now that is more like it.

Although it may seem like a viable option for setting errexit, it doesn’t always work. Copy and paste with caution.

Now let’s try something that at least works.

Option ii) Modify the script, below the interpreter line

#!/bin/bash
set -e
## debug-random-error
## ...

As expected, this removes the interpreter line dependency. Now what if we don’t want to touch the script we are about to debug?

Option iii) Don’t modify the script, specify in command line

>  /bin/bash -e /path/to/debug-random-error.sh

Personally, I like this option. After scripts that require debugging are not pretty.

Now suppose that we want to expand on using errexit to debug failed test by showing all the code that ran before the script went boom.

Option iv) Add -v -x

It may seem like the only thing that you get out of adding -v and -x to Option i through iii is getting vexed, it allows you to see all the code that ran before exiting due to a failed test.

Try it.

>  bash -v -e -x ./debug-random-error.sh

If you are so lucky your outcome will be as follows.

...
else
exit 1 # wrong args
fi
+ '[' 0 -eq 0 ']'
+ true
##################################################
debug-random-error
+ debug-random-error
+ test 1 -eq 0

One is not equal to zero after all.

Now off to bigger and better things, debugging bash scripts in your crontab!

Example ii) What-if the script is running in crontab

I know, debugging a bash script set to run in your crontab is a little too much to get started out with but if you don’t understand it now, you will later.

Here’s the gist on the crontab for anyone who doesn’t want to skip until the next code snippet.

The crontab is a list of commands set to run at specific times. Unlike scripts run in interactive mode, crontab output may write to a file or standard output which would be sent somewhere else like your email.

Let’s suppose that we receive any output written to standard output through email and have debug-random-error.sh running every minute. The crontab line would be as follows.

* * * * * /bin/bash -v -e -x /path/to/debug-random-error.sh 2>&1

Note that I redirected standard error to standard output. Why?

The reason why is that extra output from -v or -x  is written to standard error by default. That is,

/bin/bash -v -x debug-random-error.sh 2>/dev/null
is the same as
/bin/bash debug-random-error.sh

in most cases.

We covered how to use options to help debug a bash script as a while. Now let’s dive into how to debug sections of code contained within a script.

Example iii) What-if you only want to debug a section of code in your bash script

When debugging only a section of code in your script, you can surround the code with opening and closing set command lines as follows.

# the code before
set -v -x -e  # turn on options
# some suspicious bash code lines
set +v +x +e # turn off options
# the code after.
Remember,
set -o turns on options and
set +o turns off options.
Surround the sections of code with an opening and closing set commands.
#########

We covered how to use options to debug a bash script. Now to another technique, setting breakpoints.

Example iv) What if a * in the script is causing it to spill out all the files in the working directory

If  there is a * unprotected by quotes, it may be the case that the script coule perform actions on all the files within the working directory including itself to produce unexpected behavior observable while debugging.

In this case we want to use the

set -f command which turns of  globbing.

Globbing is where the * may interpretted as all the files you have laying around.

So beware of the glob!

Use breakpoints to debug scripts like a boss

Another technique that comes in handy when debugging bash scripts, is using breakpoints to pause execution at an point in the script as follows.

# the code before
set -v -x -e  # turn on options
# maybe some lines of code
read # press enter to continue
set +v +x +e # maybe turn off options
# the code after.

In brief we covered how the read bash builtin may be used to help you debug sections of a bash script. Now to the next technique, debug output.

Use output to debug scripts like a boss

Maybe your script has a debug mode builtin. In that case extra information, debug output, may be written to another location besides standard output, let’s say standard error as follows.

echo debug output 1>&2
# debug output with the same appearance as regular output

To avoid confusion, we may use colored output such as a little bash script I cooked up called cecho.sh. Here is short snippet displaying usage.

. ${SH2}/cecho.sh
var = 1
cecho yellow "var: ${var}" # var: 1 in yellow
cecho green "doing something..." # in green
cecho green "done" # also in green

You can see how colored debug output may be less confusion than writing vanilla bash spells for every debug output line.

Disclaimer: There are other methods available for colored output for use in debugging. The method above is my personal choice.

Example iv) Debug output usage

In bash curl example, I mentioned a bash a script to download files I wrote called downldr.sh. Here are some lines to explain colored output for debugging in the development of a bash script.

Your bash script may contain a operation that may only happen once such as the creation of a directory. You may include this in debug output in yellow.

cecho yellow $( mkdir -v path-of-directory-to-create )

Source: downldr.sh Line 43

Your bash script may contain a operation that is skipped depending on conditions. You may include this in debug output in green.

cecho green skipping ...

Source: downldr.sh Line 22

How to use debug output for debugging depends on preference your preference or style guide of choice if any. However, I suggest to at least use a color different than that of standard output if displayed in the terminal.

Use shopts to debug scripts like a boss (bash specific)

In addition to the set command, bash also has shopt to modify shell behavior.

Here are some shell options that may prove usefull while debugging a script.

Set options using shopt -s optname.
Unset options using shopt -u optname.

Check options using shopt -p. In most cases you are likely to be dealing with defaults. But if you have to check at least you know how.

expand_aliases

Debuging scripts with aliases can be misleading at times. If you have to turn off aliase expansion using shopt -u expand_aliases which is enabled by default in interactive mode. In the case of a script, you are bound to find a shopt -s expand_aliases line somewhere. If you have to comment it out.

Example v: mystery aliases

#!/bin/bash
## debug-aliases
## version 0.0.1 - initial
##################################################
shopt -s expand_aliases    # expand_aliases (on)
alias mystery=test
alias big-mystery='
{
mystery $( test ! $(( RANDOM % 2 )) -eq 0 || echo "!" )
}
'

mystery
echo ${?} # 1
mystery !
echo ${?} # 0
big-mystery
echo ${?} # 0 or 1
shopt -u expand_aliases # expand_aliases (off)
mystery
echo ${?} # debug-aliases.sh: line 12: mystery: command not found
mystery !
echo ${?} # debug-aliases.sh: line 14: mystery: command not found
big-mystery
echo ${?} # 0 or 1
##################################################
## generated by create-stub2.sh v0.1.2
## on Wed, 22 May 2019 21:32:43 +0900
## see <https://github.com/temptemp3/sh2>
##################################################

Source: debug-aliases.sh

failglob

If you suspect globbing issus and have errexit set, you may want to experiment with shopt -s failglob. Just for fun, here is a quick example.

# assumming there are no files starting with mystery
echo mystery* # mystery*
shopt -s failglob
echo mystery* # bash: no match: mystery*

restricted_shell

If you suspect the issue is getting lost and not behaving you can try running in restricted mode. Consider the following example script:

#!/bin/bash
## debug-restricted
## version 0.0.1 - initial
##################################################
debug-restricted() {
cd ..
}
##################################################
if [ ${#} -eq 0 ]
then
true
else
exit 1 # wrong args
fi
##################################################
debug-restricted
##################################################
## generated by create-stub2.sh v0.1.2
## on Wed, 22 May 2019 21:54:27 +0900
## see <https://github.com/temptemp3/sh2>
##################################################

Source: debug-restricted.sh

In interactive mode are within the script we run the above script in the restricted shell as follows.

bash -r debug-restricted.sh # ./debug-restricted.sh: line 7: cd: restricted

shift_verbose

If the script uses shift you may enable shift verbose as follows.

shopt –s shift_verbose
shift  # bash: shift: shift count out of range

There are the options that I have decided to introduce to help debug a script. Find more at the usual manual location for the shopt builtin.

Conclusion

In order to debug a bash script like a boss, you may need to apply bash specific shell behavior modifications. However, other debugging techniques such as debug output and setting breakpoints apply.

About the author

Nicholas Shellabarger

Nicholas Shellabarger

A developer and advocate of shell scripting and vim. His works include automation tools, static site generators, and web crawlers written in bash. For work he tools with cloud computing, app development, and chatbots. He codes in bash, python, or php, but is open to offers.