When doing C++ programming (programming for ROS included), it is helpful to have an IDE (integrated development environment), which has features such as code completion, code browsing and graphical debugging. Here are some suggestions:
The first four of these use GNU gdb as the debugger. The instructions below for debugging with an IDE are written for CLion, but they should apply to other IDEs with gdb-based debugging too.
This section will get you acquainted with basic console gdb commands. These are not necessary when debugging in an IDE, but it can be helpful to know them, for example, if you wish to debug via ssh. For more details, see this, this and this tutorial.
Suppose that you have an application which has been compiled with debugging information. Invoke gdb:
You are now at the gdb prompt. To set a breakpoint at the entry point of the function main, call:
(gdb) break main
You can also set a breakpoint in a file using at a specific line by calling
break file.cpp:XX (or just
b instead of
break), where XX is the line number. You can set a temporary breakpoint using
To run the program, type
When program stops at a breakpoint, you can to one of the following at the gdb prompt:
print your_expression(or just
pt). To call a C++ function, use
bt); change the frame with
frame N, where N=0 is the innermost stack frame, or
Ctrl-x a; other useful shortcuts:
Ctrl-N: command history;
Ctrl-L: repaint screen (sometimes gets garbled when debugged program prints to stdout);
winheight name amountchange size of window
fs name: change focus to another window
commandsmacro. For example, when running an rr replay, you could republish ROS messages this way using gdb's Python interface (TODO tutorial)
set pagination off
To ensure a pleasant gdb debugging experience, be sure to install the most recent pretty printers.
The gdb pretty printer for eigen library structures (e.g. vectors, matrices, quaternions) can be found in the eigen repository, under
debug/gdb/printers.py. Save the file (e.g. in
/home/user/gdb/) and add the following to your
python import sys sys.path.insert(1, '/home/user/gdb') from printers import register_eigen_printers register_eigen_printers (None) end
When using a gdb-based IDE, the standard pretty printers included with libstdc++ don't allow expanding of smart pointers (the shared_ptr and unique_ptr from C++11), which can be cumbersome. To get around this:
printers.pyfile is located at (on Ubuntu 16.04)
or, if using CLion, to
printers.py– place the patch in the same directory as
patch < pointer_printers.patch
xmethods.pywith the latest version from here if you want better support for shared_ptr (this will give you the ability to call
.get()from gdb, as of Feb 2017)
.gdbinitto add support for calling e.g. the
getmethod of unique/shared pointers:
python import sys sys.path.insert(2, '/path/to/clion/bin/gdb/renderers') from libstdcxx.v6.xmethods import register_libstdcxx_xmethods register_libstdcxx_xmethods(gdb) end
Unlike conventional debugging, rr enables you to rewind the execution of your application backwards in time. To begin, grab the source code from the official repo, build and install rr:
git clone https://github.com/mozilla/rr cd rr mkdir build cd build cmake ../ -Ddisable32bit=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=False -DWILL_RUN_TESTS=False make -j8 cpack -G DEB sudo dpkg -i dist/* mkdir -p ~/gdb rr gdbinit > ~/gdb/rr_gdbinit
Next, open (or create) the
~/.gdbinit file and add the following:
# hack for CLion define target remote target extended-remote $arg0 end define target hook-extended-remote source ~/gdb/rr_gdbinit end set remotetimeout 100000
A guide for using rr can be found here. To summarize:
It is recommended to add the
-M switch, which prints the rr event counter on the beginning of each line of the console output of the debugged executable
The trace to replay can be selected by also specifying the trace directory name (traces are located in
~/.local/share/rr). The event number at which the execution will initially be paused for debugging can also be specified with the
-g event_number argument when running
When you know the event number where the bug happens (if you used
-M during recording or replay), you can run the program up to this point by calling (in the gdb shell)
run event_number. This command can be used at any time in order to go to the an event checkpoint, even if the execution has passed it - rr will inteligently rewind.
(Pro-tip: To avoid the “are you sure” messages when calling
run, you can add
set confirm off to your `~/.gdbinit`.)
For backwards execution, you can use the
reverse variants of the basic gdb commands:
Another useful rr-specific command is
when, which prints out the most recent event number.
An alternative to using the gdb shell is to set up remote debugging in an IDE. To do this in CLion:
GDB Remote Debugconfiguration
target remote args, enter
rr replay -s 50505 -k -M
-k parameter keeps the rr server running after you disconnect, and
-M prints rr event numbers on the beginning of each line of the debugged program console output.
The buttons for the reverse-step gdb commands can be added to CLion by installing UndoDB's plugin for CLion. UndoDB is a commercially available reversible debugger, similar to rr.
run <event number>. It somehow breaks the rr session and causes rr to quit. However,
restartworks fine (you just have to reconnect in CLion afterwards), although it's only for checkpoints. You can change the definition of
run $arg0to be able to use
restartfor both checkpoints and events (in that case, you'll have to prefix the checkpoints with
setin gdb, rr creates a so-called diversion session, where the side effects of the functions you invoke persist (e.g. setting a variable, calling a procedure). When you call
continue, or another gdb stepping command, rr terminates the diversion session and returns to the main timeline, where the changes you did in the diversion session are no longer visible. You can fool rr to create a diversion session and remain in it by calling
print $_siginfo. From there, you can run the program forward (no backwards execution) by stepping or continuing, and you can examine the side effects of your changes. You can end the diversion session by calling
when(which returns you to the point of diversion), or restarting to a checkpoint.
sudo sh -c 'echo kernel.perf_event_paranoid=0 > /etc/sysctl.d/rr.conf'
ROS doesn't play well with Python 3 (you can't import
rospy). However, by default, gdb in Ubuntu 16.04 comes integrated with Python 3. To be able to publish messages from within the gdb shell, you can recompile gdb to use Python 2.
Source codeis checked
# make a packet which pulls all gdb build deps sudo apt install devscripts equivs mk-build-deps gdb sudo dpkg -i gdb-build-deps*.deb # dpkg will complain here about missing dependencies, # this is intended; running apt afterwards installs them sudo apt install -f rm gdb-build-deps*.deb # grab the gdb source apt source gdb cd gdb-7* # patch the configure invocation to use Python 2 sed -i -E "s|python3|python2|" debian/rules # increment the debian package version # so that apt stops suggesting an update debchange -i "Use Python 2" debchange -r release # build and install the new Debian gdb packages DEB_BUILD_OPTIONS=nocheck fakeroot debian/rules binary -j8 # if that fails, use DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage -us -uc -b -j8 sudo dpkg -i ../gdb*.deb
sudo apt remove gdb-build-deps --auto-remove). To conserve space, you can also remove the
When looking around the gdb's Python interface, it can be useful to have the ipython shell. (If using Python 3, be sure to use
sudo apt remove ipython python-zmq
sudo -H pip install ipython jupyter
ipythoncommand. When finished, be sure to call
quitin the Jupyter console. If you forget to do that, run the Jupyter console again manually (using
jupyter qtconsole --existing).