User Tools

Site Tools


Debugging and IDEs


When doing C++ programming (programming for ROS included), it is helpful to have an IDE (integrated development environment), which has features such as code completion, code browsing and graphical debugging. Here are some suggestions:

The first four of these use GNU gdb as the debugger. The instructions below for debugging with an IDE are written for CLion, but they should apply to other IDEs with gdb-based debugging too.

gdb basics

This section will get you acquainted with basic console gdb commands. These are not necessary when debugging in an IDE, but it can be helpful to know them, for example, if you wish to debug via ssh. For more details, see this, this and this tutorial.

Suppose that you have an application which has been compiled with debugging information. Invoke gdb:

gdb executable_name

You are now at the gdb prompt. To set a breakpoint at the entry point of the function main, call:

(gdb) break main

You can also set a breakpoint in a file using at a specific line by calling break file.cpp:XX (or just b instead of break), where XX is the line number. You can set a temporary breakpoint using tb.

To run the program, type

(gdb) run

When program stops at a breakpoint, you can to one of the following at the gdb prompt:

  • To continue execution, type continue (or just c)
  • To step to the next line, call next (or n).
  • To step into a function, call step (or s).
  • To run the function until it returns, call finish (or fin).
  • To print a source code listing, call list or l.
  • To inspect a variable, or evaluate a general C/C++ expression, call print your_expression (or just p instead of print). To print the type, use ptype (pt). To call a C++ function, use call function().
  • To view the stack frames, use where or backtrace (bt); change the frame with frame N, where N=0 is the innermost stack frame, or up N and down N
  • To enter the Text User Interface (TUI): Ctrl-x a; other useful shortcuts: Ctrl-X 2 (multi-windows), Ctrl-P and Ctrl-N: command history; Ctrl-L: repaint screen (sometimes gets garbled when debugged program prints to stdout); winheight name amount change size of window name (choices: src, cmd) by amount (example: +2, -3); Ctrl-X O or fs next or fs name: change focus to another window
  • Commands to be executed when a breakpoint is hit can be defined using the commands macro.
  • Disable “Type <return> to continue, or q <return> to quit”: set pagination off

Eigen & libstdc++ pretty printers

To ensure a pleasant gdb debugging experience, be sure to install the most recent pretty printers.

The gdb pretty printer for eigen library structures (e.g. vectors, matrices, quaternions) can be found in the eigen repository, under debug/gdb/ Save the file (e.g. in /home/user/gdb/) and add the following to your ~/.gdbinit file

import sys
sys.path.insert(1, '/home/user/gdb')
from printers import register_eigen_printers
register_eigen_printers (None)

When using a gdb-based IDE, the standard pretty printers included with libstdc++ don't allow expanding of smart pointers (the shared_ptr and unique_ptr from C++11), which can be cumbersome. To get around this:

  • the file is located at (on Ubuntu 16.04)

    or, if using CLion, to

  • apply this patch to – place the patch in the same directory as and run
    patch < pointer_printers.patch
  • update with the latest version from here if you want better support for shared_ptr (this will give you the ability to call .get() from gdb, as of Feb 2017)
  • add the following to .gdbinit to add support for calling e.g. the get method of unique/shared pointers:
    import sys
    sys.path.insert(2, '/path/to/clion/bin/gdb/renderers')
    from libstdcxx.v6.xmethods import register_libstdcxx_xmethods

Reversible debugging with rr

Unlike conventional debugging, rr enables you to rewind the execution of your application backwards in time. To begin, grab the source code from the official repo, install rr by calling

sudo apt install rr

…or build from source:

git clone
cd rr
mkdir build
cd build
cmake ../ -Ddisable32bit=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=False -DWILL_RUN_TESTS=False
make -j8
cpack -G DEB
sudo dpkg -i dist/*

After installing, call:

mkdir -p ~/gdb
rr gdbinit > ~/gdb/rr_gdbinit

Next, open (or create) the ~/.gdbinit file and add the following:

# hack for CLion
define target remote
source ~/gdb/rr_gdbinit
target extended-remote $arg0
set remotetimeout 100000

A guide for using rr can be found here. To summarize:

  • Record a run of the application:
    rr my_application

    It is recommended to add the -M switch, which prints the rr event counter on the beginning of each line of the console output of the debugged executable

  • To replay the most recent trace and debug it using a gdb shell, run
    rr replay

    The trace to replay can be selected by also specifying the trace directory name (traces are located in ~/.local/share/rr). The event number at which the execution will initially be paused for debugging can also be specified with the -g event_number argument when running rr replay.

When you know the event number where the bug happens (if you used -M during recording or replay), you can run the program up to this point by calling (in the gdb shell) run event_number. This command can be used at any time in order to go to the an event checkpoint, even if the execution has passed it - rr will inteligently rewind.

(Pro-tip: To avoid the “are you sure” messages when calling run, you can add set confirm off to your `~/.gdbinit`.)

For backwards execution, you can use the reverse variants of the basic gdb commands:

  • reverse-next or rn
  • reverse-continue or rc
  • reverse-step or rs
  • reverse-finish

Another useful rr-specific command is when, which prints out the most recent event number.

An alternative to using the gdb shell is to set up remote debugging in an IDE. To do this in CLion:

  • Open Run → Edit Configurations
  • Click the green + sign, and add a GDB Remote Debug configuration
  • Under target remote args, enter :50505
  • For the symbol file, set the executable hard link in the rr trace directory:
  • Invoke rr:
    rr replay -s 50505 -k -M

The -k parameter keeps the rr server running after you disconnect, and -M prints rr event numbers on the beginning of each line of the debugged program console output.

  • Start debugging in CLion by clicking the debug button. Make sure that the GDB remote configuration is selected.

The buttons for the reverse-step gdb commands can be added to CLion by installing UndoDB's plugin for CLion. UndoDB is a commercially available reversible debugger, similar to rr.

Some rr quirks/tips

  • CLion misbehaves when you invoke run <event number>. It somehow breaks the rr session and causes rr to quit. However, restart works fine (you just have to reconnect in CLion afterwards), although it's only for checkpoints. You can change the definition of restart in the rr_cmds file from run c$arg0 to run $arg0 to be able to use restart for both checkpoints and events (in that case, you'll have to prefix the checkpoints with c).
  • When you use call or set in gdb, rr creates a so-called diversion session, where the side effects of the functions you invoke persist (e.g. setting a variable, calling a procedure). When you call next, continue, or another gdb stepping command, rr terminates the diversion session and returns to the main timeline, where the changes you did in the diversion session are no longer visible. You can fool rr to create a diversion session and remain in it by calling print $_siginfo. From there, you can run the program forward (no backwards execution) by stepping or continuing, and you can examine the side effects of your changes. You can end the diversion session by calling when (which returns you to the point of diversion), or restarting to a checkpoint.
  • CLion issues a continue when connecting. Place a breakpoint on the next line (from your previous position) when reconnecting so you don't go too far forward.
  • By default, Linux kernel >=4.8.0 restricts some functionality needed by rr. To use rr, run the following command to change the relevant setting:
    sudo sh -c 'echo kernel.perf_event_paranoid=0 > /etc/sysctl.d/rr.conf'

ROS debugging - using Python 2 with gdb

ROS doesn't play well with Python 3 (you can't import rospy). However, by default, gdb in Ubuntu 16.04+ comes integrated with Python 3. To be able to publish messages from within the gdb shell, you can recompile gdb to use Python 2.

  • in the Ubuntu control panel (System Settings), open Software and Updates and make sure that Source code is checked
  • Execute the following commands:
    # make a packet which pulls all gdb build deps
    sudo apt install devscripts equivs
    mk-build-deps gdb
    sudo apt install ./gdb-build-deps*.deb
    # debs can be removed after installing
    rm gdb-build-deps*.deb
    # grab the gdb source
    apt source gdb
    cd gdb-7*
    # patch the configure invocation to use Python 2
    sed -i -E "s|python3|python2|" debian/rules
    # increment the debian package version
    # so that apt stops suggesting an update
    debchange -i "Use Python 2"
    debchange -r release
    # build and install the new Debian gdb packages
    DEB_BUILD_OPTIONS=nocheck fakeroot debian/rules binary -j8
    # if that fails, use DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage -us -uc -b -j8
    sudo dpkg -i ../gdb*.deb
  • (optional) If you wish to remove the installed gdb build dependencies, uninstall the generated gdb-build-deps package (sudo apt remove gdb-build-deps --auto-remove). To conserve space, you can also remove the gdb-source package.

Using ipython in gdb

When looking around the gdb's Python interface, it can be useful to have the ipython shell. (If using Python 3, be sure to use pip3).

  • To begin, be sure to uninstall the older Ubuntu versions:
    sudo apt remove ipython python-zmq
  • Install the latest versions of Jupyter and ipython:
    sudo -H pip install ipython jupyter
  • Add the following to your ~/.gdbinit file:
    define ipython
      shell jupyter qtconsole --existing --JupyterWidget.buffer_size=10000 --no-confirm-exit &
      python import IPython  
      python IPython.embed_kernel()
  • In gdb, call the ipython command. When finished, be sure to call quit in the Jupyter console. If you forget to do that, run the Jupyter console again manually (using jupyter qtconsole --existing).

Republishing ROS messages from an rr replay

software/debugging.txt · Last modified: 2018/07/04 11:51 by juraj