[BUG]: dd-dotnet check process fails on the standard setup for a .NET 8 Azure App Installation

[petems]

Tracer Version(s)

1.90

Operating system and platform

Linux (x64)

Instrumentation Mode

Azure App Services Steps for .NET App (Sidecar) https://docs.datadoghq.com/serverless/azure_app_services/azure_app_services_linux/?tab=manual

TFM

net8.0

Bug Report

I'm having some issues with my reproduction setup of an Azure App Service running .NET 8.0 with a very simple app, mostly copied from the Azure example repo:

It's probably some env setting somewhere but I'm following the steps to the T.

Regardless, when I attempted to see what the issue actually was with the dd-dotnet check process command, it fails at the pathing step, but this seems to be a bug as the path used by the sidecar should be valid and taken into account in the logic to check.

Really I want to fix the core issue of why its not working in the first place, but if I could use the diagnostic step in the future it would probably be very helpful 😄

Reproduction Code

https://github.com/petems/Azure-App-Service-Terraform-Datadog-APM-Logs-Sandbox - What I'm using currently

1. Spin up minimal .NET 8.0 Azure App service

2. SSH Into running app with Azure CLI

3. Try to run /home/site/wwwroot/datadog/linux-x64/dd-dotnet check process <whatever-pid>

4. Fails on checking for /opt/datadog/ directory (as it's actually installed at /home/site/wwwroot/datadog)

Example in action:

Get PID:

root@webapp-255-80a243c8:~/site/wwwroot/datadog# ps -aux 
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   4348     0 ?        SNs  22:30   0:00 /bin/bash /opt/startup/startup.sh
root          40  0.0  0.0  15436   220 ?        SNs  22:31   0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
root        1063  0.0  0.0   4348    16 ?        SN   22:31   0:00 bash /run-diag.sh
root        1065  0.2  0.1 3234792 3420 ?        SNl  22:31   0:05 ./DiagServer
root        1069  0.4  3.9 273627296 74160 ?     SNl  22:31   0:08 /opt/dotnetcore-tools/dotnet-monitor collect --urls http://0.0.0.0:50051
root        1076  0.0  0.0   4024   416 ?        SNs  22:31   0:00 /usr/sbin/cron
root        1094  0.3  1.4 273618832 26836 ?     SNl  22:31   0:06 dotnet AzureAppServiceSample.dll
root        1911  0.0  0.0  15828  1672 ?        SNs  22:59   0:00 sshd: root@pts/0
root        1917  0.0  0.1   5708  2724 pts/0    SNs  22:59   0:00 -bash
root        1994  0.0  0.2   8488  4236 pts/0    RN+  23:04   0:00 ps -aux

Use PID with dd-trace

root@webapp-255-80a243c8:~/site/wwwroot/datadog# /home/site/wwwroot/datadog/linux-x64/dd-dotnet check process 1094
Running checks on process 1094
Process name: dotnet

---- STARTING TRACER SETUP CHECKS -----
Target process is running with .NET Core
1. Checking Modules Needed so the Tracer Loads:
 [WARNING]: The native loader library is not loaded into the process
 [WARNING]: The native tracer library is not loaded into the process
 [WARNING]: Tracer is not loaded into the process
2. Checking DD_DOTNET_TRACER_HOME and related configuration value:
 [SUCCESS]: DD_DOTNET_TRACER_HOME is set to '/home/site/wwwroot/datadog' and the directory was found correctly.
3. Checking CORECLR_PROFILER_PATH and related configuration value:
 [SUCCESS]: The environment variable CORECLR_PROFILER_PATH is set to the correct value of 
/home/site/wwwroot/datadog/linux-musl-x64/Datadog.Trace.ClrProfiler.Native.so.
4. Checking CORECLR_PROFILER and related configuration value:
 [SUCCESS]: The environment variable CORECLR_PROFILER is set to the correct value of {846F5F1C-F9AE-4B07-969E-05C26BC060D8}.
5. Checking CORECLR_ENABLE_PROFILING and related configuration value:
 [SUCCESS]: The environment variable CORECLR_ENABLE_PROFILING is set to the correct value of 1.
6. Checking if process tracing configuration matches Installer or Bundler:
Installer related documentation: https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/dotnet-core?tab=linux#install-the-tracer
 [FAILURE]: Error trying to check the Linux installer directory: Could not find a part of the path '/opt/datadog'.

Workarounds attempted: Do a mkdir -p /opt/datadog
Doesn't work, silently stops at that step (presumably permissions or logic around such)

root@webapp-255-80a243c8: mkdir -p /opt/datadog
root@webapp-255-80a243c8:~/site/wwwroot/datadog# /home/site/wwwroot/datadog/linux-x64/dd-dotnet check process 1094
Running checks on process 1094
Process name: dotnet

---- STARTING TRACER SETUP CHECKS -----
Target process is running with .NET Core
1. Checking Modules Needed so the Tracer Loads:
 [WARNING]: The native loader library is not loaded into the process
 [WARNING]: The native tracer library is not loaded into the process
 [WARNING]: Tracer is not loaded into the process
2. Checking DD_DOTNET_TRACER_HOME and related configuration value:
 [SUCCESS]: DD_DOTNET_TRACER_HOME is set to '/home/site/wwwroot/datadog' and the directory was found correctly.
3. Checking CORECLR_PROFILER_PATH and related configuration value:
 [SUCCESS]: The environment variable CORECLR_PROFILER_PATH is set to the correct value of 
/home/site/wwwroot/datadog/linux-musl-x64/Datadog.Trace.ClrProfiler.Native.so.
4. Checking CORECLR_PROFILER and related configuration value:
 [SUCCESS]: The environment variable CORECLR_PROFILER is set to the correct value of {846F5F1C-F9AE-4B07-969E-05C26BC060D8}.
5. Checking CORECLR_ENABLE_PROFILING and related configuration value:
 [SUCCESS]: The environment variable CORECLR_ENABLE_PROFILING is set to the correct value of 1.
6. Checking if process tracing configuration matches Installer or Bundler:
Installer related documentation: https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/dotnet-core?tab=linux#install-the-tracer

[petems]

Code and Terraform steps I'm using to setup the Azure App Service 8.0: https://github.com/petems/Azure-App-Service-Terraform-Datadog-APM-Logs-Sandbox

[lucaspimentel]

We discussed elsewhere, and it looks like the reason why the tracer is not loading might be that

CORECLR_PROFILER_PATH=/home/site/wwwroot/datadog/linux-musl-x64/Datadog.Trace.ClrProfiler.Native.so

should be this, without musl in the path

CORECLR_PROFILER_PATH=/home/site/wwwroot/datadog/linux-x64/Datadog.Trace.ClrProfiler.Native.so

While looking into this, we found a few issues we need to look into

• the .NET tracer diagnostic tool didn't detect this as a "bundle" (nuget pacakge) installation
• the datadog-ci CLI tool is always using the musl path here.
  • ✅ fixed in [SVLS-7166] [AAS .NET] use glibc by default for the profiler path, with the option for musl datadog-ci#1753
• these docs need some updates
  • ✅ docs PR here: Serverless > Azure App Services Linux > Update .NET instructions documentation#30470