How do you troubleshoot a shell script?

Shell scripts are powerful tools for automating tasks in Unix-like systems, but they can also become complex and error-prone as they grow. When a script fails or behaves unexpectedly, identifying the cause and correcting it requires a systematic approach. Troubleshooting shell scripts effectively can save hours of guesswork and ensure the reliability of automated processes.

In this article, we explore proven techniques to troubleshoot shell scripts that behave unpredictably or crash during execution.

1. Start with the Shebang Line

The first line of a shell script should usually be a shebang (e.g., #!/bin/bash or #!/bin/sh). It tells the system which interpreter to use. If a script behaves oddly, confirm that it’s being run with the expected shell.

For example, using /bin/sh instead of /bin/bash may lead to differences in behavior due to varying support for features between shells.

2. Enable Debugging Modes

Bash and other shells provide several options for increasing script visibility during execution:

• set -x: Prints each command and its expanded arguments before executing it. This is extremely helpful.
• set -e: Makes the script exit immediately when a command fails.
• set -u: Flags unset variables as errors, preventing the use of undefined variables.

To enable these modes, add the following line after your shebang:

set -euxo pipefail

This combination enables several safety checks: tracing, exiting on errors, unset variable detection, and checking for failures in any part of a pipeline.

3. Use Echo and Logging Statements

If you’re trying to trace data flow or identify a failed logic condition, inserting echo or logging statements throughout the script can help. They should output key variable values and milestones, such as:

echo "Processing file: $filename"

In scripts meant for production, it’s advisable to redirect such output to a dedicated log file for later analysis:

./myscript.sh >> /var/log/myscript.log 2>&1

This helps preserve the visibility of what happened once the script completes (or crashes).

4. Check File and Directory Paths

Many shell scripts fail due to incorrect or nonexistent file paths. Validate input paths before using them:

if [ ! -f "$config_file" ]; then
  echo "Error: Config file not found at $config_file"
  exit 1
fi

Relative paths, in particular, can vary depending on where the script is called from. Consider using absolute paths or cd $(dirname "$0") to base your logic on the script’s location.

5. Handle Input and Output Carefully

Ensure commands that deal with input/output (such as loops over cat or read) handle whitespace and special characters correctly. Using double quotes around variables can prevent unexpected word splitting:

while IFS= read -r line; do
  echo "Line: $line"
done < "$input_file"

6. Analyze Exit Codes

Each command in a shell script returns an exit code. You can check the special variable $? immediately after a command to inspect the result:

some_command
if [ $? -ne 0 ]; then
  echo "some_command failed"
  exit 1
fi

For readability and reliability, consider using logical checks directly:

if ! some_command; then
  echo "Error: failed to run some_command"
fi

7. Test in Isolation

If a script is large, break it down into smaller sections and test them independently. This micro-testing approach helps isolate the source of failure and verify logic step by step.

8. Use Linters and Validators

Tools like ShellCheck can detect issues with syntax, portability, and idioms. They provide recommendations and often catch subtle bugs before you run the script.

shellcheck myscript.sh

Incorporating these checks into your workflow during development ensures scripts are clean and reliable from the start.

9. Version Control and Backups

Using version control systems like Git allows you to roll back changes and understand when an issue was introduced. It’s also best practice to document script changes and bug fixes in commit messages, helping others (or future you) understand the script’s evolution.

10. Consult Logs and System Messages

If your script interfaces with external systems or services (e.g., databases, network APIs), consult the associated logs. Operating system logs (in /var/log) can also provide hints if something fails at the environment level.

Conclusion

Troubleshooting a shell script is a practical skill honed through experience and attention to detail. By observing best practices like enabling debug modes, validating inputs, monitoring outputs, and leveraging tools like ShellCheck, you can diagnose and resolve issues with confidence. Adopting a structured approach not only fixes current issues but also contributes to writing more robust and maintainable scripts in the future.