nightly linux is failing due to debuginfo tests

[huonw]

E.g. http://buildbot.rust-lang.org/builders/nightly-linux/builds/86 fails with

---- [debuginfo-gdb] debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs stdout ----
    NOTE: compiletest thinks it is using GDB version 7.7

    error: line not found in debugger output: $1 = RegularStruct = {the_first_field = 101, the_second_field = 102.5, the_third_field = false}

    status: exit code: 0
    command: gdb -quiet -batch -nx -command=x86_64-unknown-linux-gnu/test/debuginfo-gdb/gdb-pretty-struct-and-enums-pre-gdb-7-7.debugger.script
    stdout:
    ------------------------------------------
    GNU gdb (GDB) 7.7
    Copyright (C) 2014 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-unknown-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word".
    Breakpoint 1 at 0x940: file /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/test/debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs, line 75.
    static void gdb-pretty-struct-and-enums-pre-gdb-7-7::zzz(void);
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib64/libthread_db.so.1".

    Breakpoint 1, gdb-pretty-struct-and-enums-pre-gdb-7-7::zzz () at /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/test/debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs:75
    75  fn zzz() { () }
    0x00007ffff7ffe8d7 in gdb-pretty-struct-and-enums-pre-gdb-7-7::main () at /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/test/debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs:72
    72      zzz();
    $1 = {the_first_field = 101, the_second_field = 102.5, the_third_field = false}
    $2 = {<No data fields>}
    $3 = CStyleEnumVar1
    $4 = CStyleEnumVar2
    $5 = CStyleEnumVar3
    A debugging session is active.

        Inferior 1 [process 17861] will be killed.

    Quit anyway? (y or n) [answered Y; input not from terminal]

    ------------------------------------------
    stderr:
    ------------------------------------------

    ------------------------------------------

    task '[debuginfo-gdb] debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs' failed at 'explicit failure', /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/compiletest/runtest.rs:1415

it seems the $1 line doesn't quite match.

cc @michaelwoerister

[alexcrichton]

recent log (with two failing tests)

---- [debuginfo-gdb] debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs stdout ----
    NOTE: compiletest thinks it is using GDB version 7.7

    error: line not found in debugger output: $1 = RegularStruct = {the_first_field = 101, the_second_field = 102.5, the_third_field = false}

    status: exit code: 0
    command: gdb -quiet -batch -nx -command=x86_64-unknown-linux-gnu/test/debuginfo-gdb/gdb-pretty-struct-and-enums-pre-gdb-7-7.debugger.script
    stdout:
    ------------------------------------------
    GNU gdb (GDB) 7.7
    Copyright (C) 2014 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-unknown-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word".
    Breakpoint 1 at 0x940: file /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/test/debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs, line 75.
    static void gdb-pretty-struct-and-enums-pre-gdb-7-7::zzz(void);
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib64/libthread_db.so.1".

    Breakpoint 1, gdb-pretty-struct-and-enums-pre-gdb-7-7::zzz () at /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/test/debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs:75
    75  fn zzz() { () }
    0x00007ffff7ffe8d7 in gdb-pretty-struct-and-enums-pre-gdb-7-7::main () at /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/test/debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs:72
    72      zzz();
    $1 = {the_first_field = 101, the_second_field = 102.5, the_third_field = false}
    $2 = {<No data fields>}
    $3 = CStyleEnumVar1
    $4 = CStyleEnumVar2
    $5 = CStyleEnumVar3
    A debugging session is active.

        Inferior 1 [process 30775] will be killed.

    Quit anyway? (y or n) [answered Y; input not from terminal]

    ------------------------------------------
    stderr:
    ------------------------------------------

    ------------------------------------------

    task '[debuginfo-gdb] debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs' failed at 'explicit failure', /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/compiletest/runtest.rs:1415


---- [debuginfo-gdb] debuginfo/gdb-pretty-struct-and-enums.rs stdout ----
    NOTE: compiletest thinks it is using GDB version 7.7

    error: line not found in debugger output: $1 = RegularStruct = {the_first_field = 101, the_second_field = 102.5, the_third_field = false, the_fourth_field = "I'm so pretty, oh so pretty..."}

    status: exit code: 0
    command: gdb -quiet -batch -nx -command=x86_64-unknown-linux-gnu/test/debuginfo-gdb/gdb-pretty-struct-and-enums.debugger.script
    stdout:
    ------------------------------------------
    GNU gdb (GDB) 7.7
    Copyright (C) 2014 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-unknown-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word".
    Breakpoint 1 at 0xb40: file /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/test/debuginfo/gdb-pretty-struct-and-enums.rs, line 172.
    static void gdb-pretty-struct-and-enums::zzz(void);
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib64/libthread_db.so.1".

    Breakpoint 1, gdb-pretty-struct-and-enums::zzz () at /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/test/debuginfo/gdb-pretty-struct-and-enums.rs:172
    172 fn zzz() { () }
    0x00007ffff7ffead6 in gdb-pretty-struct-and-enums::main () at /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/test/debuginfo/gdb-pretty-struct-and-enums.rs:169
    169     zzz();
    $1 = {the_first_field = 101, the_second_field = 102.5, the_third_field = false, the_fourth_field = {data_ptr = 0x7ffff7ffecb0 <str1128> "I'm so pretty, oh so pretty...blub", length = 30}}
    $2 = {true, 103, {data_ptr = 0x7ffff7ffecce <str1129> "blub", length = 4}}
    $3 = {-104.5, 105}
    $4 = {<No data fields>}
    $5 = CStyleEnumVar1
    $6 = CStyleEnumVar2
    $7 = CStyleEnumVar3
    $8 = {{MixedEnumCStyleVar}, {MixedEnumCStyleVar, 0, 58128, 255}, {MixedEnumCStyleVar, field1 = 6.9533558074690005e-310, field2 = 1}}
    $9 = {{MixedEnumTupleVar}, {MixedEnumTupleVar, 106, 107, false}, {MixedEnumTupleVar, field1 = 6.9533483476988382e-310, field2 = 0}}
    $10 = {{MixedEnumStructVar}, {MixedEnumStructVar, 32767, 0, false}, {MixedEnumStructVar, field1 = 108.5, field2 = 109}}
    $11 = {{Some}, {Some, 110}}
    $12 = {{None}, {None, 140733193388033}}
    $13 = {{NestedVariant1, {regular_struct = {the_first_field = 111, the_second_field = 112.5, the_third_field = true, the_fourth_field = {data_ptr = 0x7ffff7ffece0 <str1131> "NestedStructString1", length = 19}}, tuple_struct = {113.5, 114}, empty_struct = {<No data fields>}, c_style_enum = CStyleEnumVar2, mixed_enum = {{MixedEnumTupleVar}, {MixedEnumTupleVar, 115, 116, false}, {MixedEnumTupleVar, field1 = 6.9533483476992829e-310, field2 = -7496}}}}, {NestedVariant1, abc = {regular_struct = {the_first_field = 111, the_second_field = 112.5, the_third_field = true, the_fourth_field = {data_ptr = 0x7ffff7ffece0 <str1131> "NestedStructString1", length = 19}}, tuple_struct = {113.5, 114}, empty_struct = {<No data fields>}, c_style_enum = CStyleEnumVar2, mixed_enum = {{MixedEnumTupleVar}, {MixedEnumTupleVar, 115, 116, false}, {MixedEnumTupleVar, field1 = 6.9533483476992829e-310, field2 = -7496}}}}}
    $14 = {{NestedVariant2, {regular_struct = {the_first_field = 117, the_second_field = 118.5, the_third_field = false, the_fourth_field = {data_ptr = 0x7ffff7ffed00 <str1132> "NestedStructString10\001\033\003;8", length = 20}}, tuple_struct = {119.5, 120}, empty_struct = {<No data fields>}, c_style_enum = CStyleEnumVar3, mixed_enum = {{MixedEnumStructVar}, {MixedEnumStructVar, 0, 0, false}, {MixedEnumStructVar, field1 = 121.5, field2 = -122}}}}, {NestedVariant2, abc = {regular_struct = {the_first_field = 117, the_second_field = 118.5, the_third_field = false, the_fourth_field = {data_ptr = 0x7ffff7ffed00 <str1132> "NestedStructString10\001\033\003;8", length = 20}}, tuple_struct = {119.5, 120}, empty_struct = {<No data fields>}, c_style_enum = CStyleEnumVar3, mixed_enum = {{MixedEnumStructVar}, {MixedEnumStructVar, 0, 0, false}, {MixedEnumStructVar, field1 = 121.5, field2 = -122}}}}}
    A debugging session is active.

        Inferior 1 [process 30788] will be killed.

    Quit anyway? (y or n) [answered Y; input not from terminal]

    ------------------------------------------
    stderr:
    ------------------------------------------

    ------------------------------------------

    task '[debuginfo-gdb] debuginfo/gdb-pretty-struct-and-enums.rs' failed at 'explicit failure', /home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/obj/tmp/distcheck/rust-nightly/src/compiletest/runtest.rs:1415



failures:
    [debuginfo-gdb] debuginfo/gdb-pretty-struct-and-enums-pre-gdb-7-7.rs
    [debuginfo-gdb] debuginfo/gdb-pretty-struct-and-enums.rs

test result: FAILED. 91 passed; 2 failed; 5 ignored; 0 measured

[michaelwoerister]

It seems that GDB is running without the pretty printers enabled for some reason, but I don't see an error in the log. Maybe python scripting is not enabled for this specific GDB installation?

[alexcrichton]

Is there a way for us to check? It looks like it has a --python flag for executing a python script:

[rustbuild@ip-10-228-0-60 ~]$ gdb -v
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.

[root@ip-10-228-0-60 ~]# gdb -h
This is the GNU debugger.  Usage:

    gdb [options] [executable-file [core-file or process-id]]
    gdb [options] --args executable-file [inferior-arguments ...]
    gdb [options] [--python|-P] script-file [script-arguments ...]

Options:

  --args             Arguments after executable-file are passed to inferior
  -b BAUDRATE        Set serial port baud rate used for remote debugging.
  --batch            Exit after processing options.
  --batch-silent     As for --batch, but suppress all gdb stdout output.
  --return-child-result
                     GDB exit code will be the child's exit code.
  --cd=DIR           Change current directory to DIR.
  --command=FILE, -x Execute GDB commands from FILE.
  --eval-command=COMMAND, -ex
                     Execute a single GDB command.
                     May be used multiple times and in conjunction
                     with --command.
  --init-command=FILE, -ix Like -x but execute it before loading inferior.
  --init-eval-command=COMMAND, -iex Like -ex but before loading inferior.
  --core=COREFILE    Analyze the core dump COREFILE.
  --pid=PID          Attach to running process PID.
  --dbx              DBX compatibility mode.
  --directory=DIR    Search for source files in DIR.
  --epoch            Output information used by epoch emacs-GDB interface.
  --exec=EXECFILE    Use EXECFILE as the executable.
  --fullname         Output information used by emacs-GDB interface.
  --help             Print this message.
  --interpreter=INTERP
                     Select a specific interpreter / user interface
  -l TIMEOUT         Set timeout in seconds for remote debugging.
  --nw           Do not use a window interface.
  --nx               Do not read .gdbinit file.
  --python, -P       Following argument is Python script file; remaining
                     arguments are passed to script.
  --quiet            Do not print version number on startup.
  --readnow          Fully read symbol files on first access.
  --readnever        Do not read symbol files.
  --se=FILE          Use FILE as symbol file and executable file.
  --symbols=SYMFILE  Read symbols from SYMFILE.
  --tty=TTY          Use TTY for input/output by the program being debugged.
  --tui              Use a terminal user interface.
  --version          Print version information and then exit.
  -w                 Use a window interface.
  --write            Set writing into executable and core files.
  --xdb              XDB compatibility mode.

At startup, GDB reads the following init files and executes their commands:

For more information, type "help" from within GDB, or consult the
GDB manual (available as on-line info or a printed manual).
Report bugs to "<http://www.gnu.org/software/gdb/bugs/>".

[michaelwoerister]

Is there a way for us to check?

There's the py command which let's you evaluate a python code snippet from the GDB prompt. If you type py print("hi alex!") and it works, then Python support should be enabled.

It may also be that script autoloading is disabled for some reason. You can find that out with command show auto-load python-scripts in GDB.

[alexcrichton]

Hm, this looks suspicious in that case, because they both seem to be working!

[root@ip-10-228-0-60 ~]# gdb
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) py print("foo")
foo
(gdb) show auto-load python-scripts
Auto-loading of Python scripts is on.
(gdb) 

[michaelwoerister]

Are you sure this is the same machine? Because the first two logs show GDB v7.7 and yours shows v7.2 ...

[alexcrichton]

Aha, you are indeed correct. The gdb being used is not the system gdb because I purposely tried to install a newer version (7.7) when I last setup the nightly bots. I had to do so by building from source, however, and it looks like that does indeed have python scripting turned off by default.

Do you know if we're compatible with gdb 7.2? Should I stop using the 7.7 version and pray?

[michaelwoerister]

Yes, I think one has to run ./configure --with-python to get python support when building GDB from source.

Regarding GDB 7.2, we already deactivate some pretty printing tests based on the actual GDB version used for testing. The oldest version I have tested things with is 7.4 -- but chances are not too bad that it will just work with 7.2 as well.

BTW, compiletest now writes a line to standard out which indicates the GDB version assumed for the test, as in the log from above:

NOTE: compiletest thinks it is using GDB version 7.7

This can be helpful for debugging.