Segfault when debugging minimal project

[isbobbydigitalak]

Hello!

I can't get netcoredbg to work on even a simple project. Since I am using GUIX as my distro I decided to build netcoredbg from source so that we can rule out mis-matched libc version from the binary release and so on.

I have a very simple console test project that contains this source file

public class Program
{
    public static void Main(string[] args)
    {
        var msg = "Hey";
        Console.WriteLine("Hello, World!");
        Console.WriteLine(msg);
    }
}

I built it with EmbedAllSources set to true.

It crashes on the line src/debugger/manageddebugger.cpp:559 in the latest released version of netcoredbg. This appears to be because the pCordb argument to StartupCallback is null. I don't know how to figure out why that is the case so I didn't go further than this.

In case it helps I have also attached a gdb debugging session of what I tried:

<my-name>@mini<company-name> ~/<company-name>/src/test-netcore/bin/Debug/net7.0$ netcoredbg --interpreter=cli &
[1] 4593
<my-name>@mini<company-name> ~/<company-name>/src/test-netcore/bin/Debug/net7.0$ gdb --pid 4593
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 4593
Reading symbols from /home/<my-name>/.local/opt/netcoredbg/netcoredbg...
Reading symbols from /gnu/store/16zls6rxxma08js8f3rkgmrg99r7qrqr-gcc-11.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../libstdc++.so.6...
(No debugging symbols found in /gnu/store/16zls6rxxma08js8f3rkgmrg99r7qrqr-gcc-11.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../libstdc++.so.6)
Reading symbols from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libm.so.6...
(No debugging symbols found in /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libm.so.6)
Reading symbols from /gnu/store/16zls6rxxma08js8f3rkgmrg99r7qrqr-gcc-11.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../libgcc_s.so.1...
(No debugging symbols found in /gnu/store/16zls6rxxma08js8f3rkgmrg99r7qrqr-gcc-11.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/11.3.0/../../../libgcc_s.so.1)
Reading symbols from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6...
(No debugging symbols found in /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6)
Reading symbols from //gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/ld-linux-x86-64.so.2...
(No debugging symbols found in //gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/ld-linux-x86-64.so.2)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libthread_db.so.1".

Program received signal SIGTTOU, Stopped (tty output).
0x00007fbf5fc21d33 in tcsetattr () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/<my-name>/.local/opt/netcoredbg/netcoredbg 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libthread_db.so.1".
[New Thread 0x7ffff7a03640 (LWP 4645)]
ncdb> file dotnet
ncdb> set args test-netcore.dll
ncdb> r
[Detaching after fork from child process 4676]
[New Thread 0x7ffff7202640 (LWP 4677)]

Thread 3 "netcoredbg" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7202640 (LWP 4677)]
0x0000555555ac210f in netcoredbg::ManagedDebuggerHelpers::Startup (this=0x555555eaa5a0, punk=0x0) at /home/<my-name>/<company-name>/sftw/netcoredbg-src/netcoredbg-3.0.0-1018/src/debugger/manageddebugger.cpp:559
559	   IfFailRet(punk->QueryInterface(IID_ICorDebug, (void **)&iCorDebug));
(gdb) p punk
$1 = (IUnknown *) 0x0
(gdb) bt
#0  0x0000555555ac210f in netcoredbg::ManagedDebuggerHelpers::Startup (this=0x555555eaa5a0, punk=0x0)
    at /home/<my-name>/<company-name>/sftw/netcoredbg-src/netcoredbg-3.0.0-1018/src/debugger/manageddebugger.cpp:559
#1  0x0000555555ac2068 in netcoredbg::ManagedDebuggerHelpers::StartupCallback (pCordb=0x0, parameter=0x555555eaa5a0, hr=-2146231236)
    at /home/<my-name>/<company-name>/sftw/netcoredbg-src/netcoredbg-3.0.0-1018/src/debugger/manageddebugger.cpp:457
#2  0x00007ffff7e80026 in ?? () from /home/<my-name>/.local/opt/netcoredbg/libdbgshim.so
#3  0x00007ffff7eba66e in ?? () from /home/<my-name>/.local/opt/netcoredbg/libdbgshim.so
#4  0x00007ffff7eba58e in ?? () from /home/<my-name>/.local/opt/netcoredbg/libdbgshim.so
#5  0x00007ffff7eba40e in ?? () from /home/<my-name>/.local/opt/netcoredbg/libdbgshim.so
#6  0x00007ffff7ebbcb0 in ?? () from /home/<my-name>/.local/opt/netcoredbg/libdbgshim.so
#7  0x00007ffff7a893aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#8  0x00007ffff7b09f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
(gdb) b manageddebugger.cpp:457
Breakpoint 1 at 0x555555ac205b: file /home/<my-name>/<company-name>/sftw/netcoredbg-src/netcoredbg-3.0.0-1018/src/debugger/manageddebugger.cpp, line 457.
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /home/<my-name>/.local/opt/netcoredbg/netcoredbg 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libthread_db.so.1".
[New Thread 0x7ffff7a03640 (LWP 4771)]
ncdb> file dotnet
ncdb> set args test-netcore.dll
ncdb> r
[Detaching after fork from child process 4801]
[New Thread 0x7ffff7202640 (LWP 4802)]
[Switching to Thread 0x7ffff7202640 (LWP 4802)]

Thread 3 "netcoredbg" hit Breakpoint 1, netcoredbg::ManagedDebuggerHelpers::StartupCallback (pCordb=0x0, parameter=0x555555eaa5a0, hr=-2146231236) at /home/<my-name>/<company-name>/sftw/netcoredbg-src/netcoredbg-3.0.0-1018/src/debugger/manageddebugger.cpp:457
457	   self->Startup(pCordb);
(gdb) p self
$2 = (netcoredbg::ManagedDebugger *) 0x555555eaa5a0
(gdb) p pCordb
$3 = (IUnknown *) 0x0
(gdb) q

[viewizard]

(gdb) p pCordb
$3 = (IUnknown *) 0x0

Looks like some libdbgshim.so issue, ManagedDebuggerHelpers::StartupCallback callback called from libdbgshim.so back to netcoredbg, and looks like some error happens, since it return NULL for pCordb.
https://github.com/dotnet/diagnostics/blob/1f512d6f5568d3303053fd5279a06425aae53e1c/src/dbgshim/dbgshim.cpp#L417
You probably could check error code from HRESULT hr in

netcoredbg/src/debugger/manageddebugger.cpp

Line 453 in 27606c3

VOID ManagedDebuggerHelpers::StartupCallback(IUnknown *pCordb, PVOID parameter, HRESULT hr)

Note, libdbgshim.so is part of Diagnostics (https://github.com/dotnet/diagnostics), not netcoredbg, and now MS provide it as binary in NuGet package (that downloads during netcoredbg build from repo):
https://www.nuget.org/packages/Microsoft.Diagnostics.DbgShim

Also, you could test libdbgshim.so from latest netcoredbg release. This one works for sure (at least on ubuntu 20.04, that we use for CI), since all releases we test on CI for all tests pass.
And (just in case) you could build libdbgshim.so for your distro/envs from https://github.com/dotnet/diagnostics sources.

[isbobbydigitalak]

Thanks for the input! I will check the error code (though probably tomorrow or next week, already put a lot more time on this today than I should have!).

It is useful info that libdbgshim is downloaded as a binary. I ran ldd on it the one I have installed now and saw that it is not able to resolve some dependencies (libstdc++ and libgcc_s). That doesn't really explain to me how it manages to still call back though. Furthermore, in my guix packaging attempt I wrote with the binary release, it uses patchelf to fix up all those dependencies (ldd shows all things resolved on those binaries), yet, netcoredbg still had the same behaviour, so while it's a problem it might not be the full story.

Anyhow, since guix (which is like nix) uses crazy paths for everything I might have a higher chance to succeed by going and building libdbgshim from source like you suggest. If it ends up working then it should be able to provide some hints for how I can package netcoredbg from the binary releases with patchelf.

[gbalykov]

Since this is some non-standard environment, you can also try to run all netcoredbg tests, see https://github.com/Samsung/netcoredbg/blob/master/test-suite/README.md.
Also how did you install .net runtime, from binaries or built it yourself? It might be worth running .net runtime tests too to verify that it works.

[isbobbydigitalak]

I installed .NET runtime from binaries by modifying the package in the nonguix project (to update it to a newer version). It is probably worth running the runtime tests (if it's possible from binary build?), so far however, I have been using it for work without any issues, not only for extremely simple test programs.