|
| 1 | +# Debugging with GDB |
| 2 | + |
| 3 | +In the most cases writing extensive logs and running profilers is sufficient for debugging. But in the most complex cases it may be useful to do interactive debugging with GDB. Additionally, GDB is the only option for debugging coredumps. Userver provides the capability to perform such debugging. |
| 4 | + |
| 5 | +First of all, you can use GDB on your service based on Userver just like any other binary (debug symbols are included by default in all build types). |
| 6 | + |
| 7 | +## Userver-specific debug features |
| 8 | + |
| 9 | +To use Userver-specific debug features, you need to allow execution of debug scripts, linked into your binary. |
| 10 | +It can be done by adding the following line to your ~/.gdbinit file |
| 11 | +``` |
| 12 | +add-auto-load-safe-path <path-to-your-binary> |
| 13 | +``` |
| 14 | +Alternatively, if you trust all the files you are debugging: |
| 15 | +``` |
| 16 | +add-auto-load-safe-path / |
| 17 | +``` |
| 18 | +See [GDB manual](https://www.sourceware.org/gdb/current/onlinedocs/gdb.html/Auto_002dloading-safe-path.html) for more details. |
| 19 | + |
| 20 | +### Custom pretty-printers |
| 21 | + |
| 22 | +The simplest extentions for GDB, that Userver provides, are pretty-printers for certain data stuctures. Below is an example comparing the output for a `formats::json::Value` with and without pretty-printers: |
| 23 | + |
| 24 | +``` |
| 25 | +(gdb) print value |
| 26 | +$1 = {["a"] = {1, 2, 3, 4}, ["c"] = 123, ["d"] = false, ["e"] = "", ["f"] = {["key1"] = "value1", ["key2"] = 987}} |
| 27 | +(gdb) disable pretty-printer |
| 28 | +181 printers disabled |
| 29 | +0 of 181 printers enabled |
| 30 | +(gdb) print value |
| 31 | +$2 = {holder_ = {static kInvalidVersion = 18446744073709551615, data_ = {__ptr_ = 0x10f27fe04118, __cntrl_ = 0x10f27fe04100}}, |
| 32 | + root_ptr_for_path_ = 0x10f27fe04118, value_ptr_ = 0x10f27fe04118, depth_ = 0, lazy_detached_path_ = {parent_value_ptr_ = 0x0, |
| 33 | + parent_depth_ = 0, virtual_path_ = {static __endian_factor = 2, __rep_ = {__s = {{__is_long_ = 0 '\000', __size_ = 0 '\000'}, |
| 34 | + __padding_ = {<No data fields>}, __data_ = '\000' <repeats 22 times>}, __l = {{__is_long_ = 0, __cap_ = 0}, __size_ = 0, |
| 35 | + __data_ = 0x0}}, __padding1_933_ = {__padding_ = 0x7fffffffd7c0 ""}, |
| 36 | + __alloc_ = {<std::__y1::__non_trivial_if<true, std::__y1::allocator<char> >> = {<No data fields>}, <No data fields>}, __padding2_933_ = { |
| 37 | + __padding_ = 0x7fffffffd7c0 ""}, static npos = 18446744073709551615}}} |
| 38 | +``` |
| 39 | + |
| 40 | +In addition, the output has a hierarchical structure that is displayed correctly when debugging from the IDE. |
| 41 | + |
| 42 | +### Coroutines exploration |
| 43 | + |
| 44 | +Userver provides GDB command `utask`, which mimics `thread` command and allows you to explore all coroutines (Userver Tasks), including running and suspended ones, in a manner similar to threads. |
| 45 | + |
| 46 | +#### Commands |
| 47 | + |
| 48 | +* `utask list`: Lists all tasks with their names (corresponding span names) and statuses. Example: |
| 49 | +``` |
| 50 | +(gdb) utask list |
| 51 | +Task State Span |
| 52 | +0x10f27fc40800 Suspended task_3 |
| 53 | +0x10f27fc3f000 Suspended task_2 |
| 54 | +0x10f27fc3d800 Suspended task_1 |
| 55 | +0x10f27fc3c000 Suspended task_0 |
| 56 | +0x10f27fc38000 Suspended span |
| 57 | +0x10f27fc42000 Running task_4 |
| 58 | +``` |
| 59 | + |
| 60 | +* `utask apply <task> <cmd...>`: Executes `<cmd...>` in the context of selected `<task>`. The `<task>` may be specified by its ID ("Task") or name ("Span") (as shown in `utask list`), or set to "all" to apply the command to all tasks. `<cmd...>` can be any GDB command, including Python scripts. |
| 61 | + |
| 62 | +#### Examples: |
| 63 | + |
| 64 | +1. Print "Hello world!" for all tasks |
| 65 | +``` |
| 66 | +(gdb) utask apply all print "Hello world!" |
| 67 | +Executing command `print "Hello world!"` for task 0x10f27fc40800 |
| 68 | +$1 = "Hello world!" |
| 69 | +Executing command `print "Hello world!"` for task 0x10f27fc3f000 |
| 70 | +$2 = "Hello world!" |
| 71 | +Executing command `print "Hello world!"` for task 0x10f27fc3d800 |
| 72 | +$3 = "Hello world!" |
| 73 | +Executing command `print "Hello world!"` for task 0x10f27fc3c000 |
| 74 | +$4 = "Hello world!" |
| 75 | +Executing command `print "Hello world!"` for task 0x10f27fc38000 |
| 76 | +$5 = "Hello world!" |
| 77 | +Executing command `print "Hello world!"` for task 0x10f27fc42000 |
| 78 | +$6 = "Hello world!" |
| 79 | +``` |
| 80 | + |
| 81 | +2. Get backtrace of the suspended `task_1` |
| 82 | +``` |
| 83 | +(gdb) utask apply task_1 backtrace |
| 84 | +Executing command `backtrace` for task 0x10f27fc3d800 |
| 85 | +#1 0x00000000006131d8 in boost::context::fiber::resume() && (this=0x7fffefae6e38) |
| 86 | + at /usr/include/boost/context/include/boost/context/fiber_fcontext.hpp:377 |
| 87 | +#2 boost::coroutines2::detail::pull_coroutine<engine::impl::TaskContext*>::control_block::resume (this=0x7fffefae6e38) |
| 88 | + at /usr/include/boost/coroutine2/include/boost/coroutine2/detail/pull_control_block_cc.ipp:147 |
| 89 | +#3 0x0000000000611b2c in boost::coroutines2::detail::pull_coroutine<engine::impl::TaskContext*>::operator() (this=<optimized out>) |
| 90 | + at /usr/include/boost/coroutine2/include/boost/coroutine2/detail/pull_coroutine.ipp:77 |
| 91 | +#4 engine::impl::TaskContext::Sleep (this=0x10f27fc3d800, wait_strategy=..., deadline=...) |
| 92 | + at /home/.../userver/core/src/engine/task/task_context.cpp:332 |
| 93 | +#5 0x00000000005fbaca in engine::impl::ConditionVariableAny<engine::Mutex>::WaitUntil (this=0x7fffefb68c50, lock=..., deadline=...) |
| 94 | + at /home/.../userver/core/src/engine/impl/condition_variable_any.cpp:72 |
| 95 | +#6 0x00000000003f98b6 in TestMultipleCoroutines(int, int)::$_0::operator()() const::'lambda'()::operator()() const (this=0x10f27fc3eba8) |
| 96 | + at /home/.../userver/scripts/gdb/tests/src/cmd/utask/gdb_test_utask.cpp:158 |
| 97 | +...... |
| 98 | +``` |
| 99 | + |
| 100 | +3. Run a Python script in the context of `task_1` (see [GDB Python API](https://www.sourceware.org/gdb/current/onlinedocs/gdb.html/Python-API.html)) |
| 101 | +``` |
| 102 | +(gdb) utask apply task_1 python print('Hello from python!', 'current frame:', gdb.selected_frame().function()) |
| 103 | +Executing command `python print('Hello from python!', 'current frame:', gdb.selected_frame().function())` for task 0x10f27fc3d800 |
| 104 | +Hello from python! current frame: boost::context::fiber::resume() && |
| 105 | +``` |
| 106 | + |
| 107 | +For now `utask` commands are implemented for only linux x86 platforms, but can be easily extended for other platforms. |
| 108 | + |
| 109 | +In addition, all of the above functionality works for debugging both a live process and coredumps. |
| 110 | + |
| 111 | +## Adding new pretty-printers and commands |
| 112 | + |
| 113 | +If you need a new pretty-printer or a GDB command, you can always implement it yourself in `userver/scripts/gdb` and bring us a PR! |
0 commit comments