How to generate heap dumps for .NET apps on TAS
search cancel

How to generate heap dumps for .NET apps on TAS

book

Article ID: 376123

calendar_today

Updated On:

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

This article shows two methods to collect heap dumps on .NET apps running on Tanzu Application Service

  1. Using Steeltoe actuators (recommended)
  2. Using dotnet tools manually (dotnet-dump, dotnet-gcdump, etc) - Linux Diego Cell
  3. Using dotnet tools manually (dotnet-dump, dotnet-gcdump, etc) - Windows Diego Cell

Resolution

Using Steeltoe actuators (recommended)

  1. Add Steeltoe to your app per the documentation: https://docs.steeltoe.io/api/v3/welcome/
    • If using Steeltoe bootstrapping, you will need at least the Steeltoe.Bootstrap.Autoconfig and Steeltoe.Management.EndpointCore dependencies for heap dumps. Steeltoe.Extensions.Configuration.CloudFoundryCore can also be added to enable heap dumps in the Apps Manager UI.
    • Several sample apps are located at https://github.com/SteeltoeOSS/Samples
  2. Configure your appsettings.json for heap dumps
    • Expose heapdump endpoint and/or cloudfoundry endpoint (if using Apps Manager integration)
    • On TAS, heap dumps are disabled by default and need to be enabled. More info: https://docs.steeltoe.io/api/v3/management/heapdump.html#status-on-cloud-foundry
    • Set HeapDumpType to Normal, With Heap, Triage, or Full for a dotnet-dump file: https://docs.steeltoe.io/api/v3/management/heapdump.html#configure-settings
    • Set HeapDumpType to gcdump for a dotnet-gcdump file (readable by Perfview or Visual Studio on Windows): https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-gcdump
    • Below is a sample appsettings.json configured for gcdump: 
        "Management": {
    "Endpoints": {
      "Actuator":{
        "Exposure": {
          "Include": [ "cloudfoundry", "heapdump" ]
        }
      },
      "HeapDump": {
        "Enabled": true,
        "HeapDumpType": "gcdump"
      }
    }
  }
  3. Push the app and you can download the heap dump from /actuator/heapdump or in Apps Manager UI depending on which endpoints you exposed
    • If using Apps Manager UI, follow the instructions to rename the .dmp file to a .dmp.gz and extract it: https://docs.steeltoe.io/api/v3/management/heapdump.html#status-on-cloud-foundry


Using dotnet tools manually (on Linux Diego Cells)
This process involves putting the dotnet tools on the app container and running it manually. If the app is self-contained (dotnet publish --self-contained true) you will also have to put the dotnet SDK on the app container to run the tools. You can either put the tools/sdk in the app root directory so it gets pushed with cf push or use scp to transfer the files to the container: https://docs.cloudfoundry.org/devguide/deploy-apps/ssh-apps.html#other-ssh-access

This below example is for a self contained app and uses cf push to get the files on the app container.

  1. Download and push the tools/sdk with the app
    •  
  2. SSH into the app container and extract the .NET SDK 
    cf ssh MyWebApp-steeltoe
mkdir dotnet-sdk && tar xvf dotnet-sdk-7.0.102-linux-x64.tar.gz -C dotnet-sdk
  3. Run the tool with the DOTNET_ROOT and TMPDIR environment variables
    • If it is a source based or non self-contained app, set DOTNET_ROOT=/home/vcap/deps/0/dotnet-sdk/
    • For self-contained apps, set DOTNET_ROOT to the directory you extracted the SDK to
    • TMPDIR should be /home/vcap/tmp/ 
      vcap@2ce44c60-99b4-4251-5012-e3e0:~/app$ DOTNET_ROOT=./dotnet-sdk TMPDIR=/home/vcap/tmp/ ./dotnet-gcdump ps
 22  MyWebApp-steeltoe  /home/vcap/app/MyWebApp-steeltoe  -steeltoe --urls http://0.0.0.0:8080

vcap@2ce44c60-99b4-4251-5012-e3e0:~/app$ DOTNET_ROOT=./dotnet-sdk TMPDIR=/home/vcap/tmp/ ./dotnet-gcdump collect -p 22
Writing gcdump to '/home/vcap/app/20230202_193432_22.gcdump'...
	Finished writing 974543 bytes.
vcap@2ce44c60-99b4-4251-5012-e3e0:~/app$
  4. Analyze the dump file in the container or transfer it out using https://docs.cloudfoundry.org/devguide/deploy-apps/ssh-apps.html#other-ssh-access
    • Example: 
      ~/Documents/MyWebApp-steeltoe % cf curl /v2/info
{
   "name": "VMware Tanzu Application Service",
   "build": "2.11.32-build.1",
   "support": "https://support.pivotal.io",
   "version": 0,
   "description": "https://docs.pivotal.io/pivotalcf/2-11/pcf-release-notes/runtime-rn.html",
   "authorization_endpoint": "https://login.run-27.slot-35.tanzu-gss-labs.vmware.com",
   "token_endpoint": "https://uaa.run-27.slot-35.tanzu-gss-labs.vmware.com",
   "min_cli_version": "6.23.0",
   "min_recommended_cli_version": "6.23.0",
   "app_ssh_endpoint": "ssh.run-27.slot-35.tanzu-gss-labs.vmware.com:2222",
   "app_ssh_host_key_fingerprint": "66:ba:b5:ec:db:46:e8:31:a8:ae:45:45:2c:2a:63:22",
   "app_ssh_oauth_client": "ssh-proxy",
   "doppler_logging_endpoint": "wss://doppler.run-27.slot-35.tanzu-gss-labs.vmware.com:443",
   "api_version": "2.164.0",
   "osbapi_version": "2.15",
   "routing_endpoint": "https://api.run-27.slot-35.tanzu-gss-labs.vmware.com/routing",
   "user": "7c9bbaa7-d537-4ec3-ad3f-6ecba8d03f6d"
}
~/Documents/MyWebApp-steeltoe % cf ssh-code
bNFaaXHpv1
~/Documents/MyWebApp-steeltoe % scp -P 2222 -o user="cf:$(cf curl /v3/apps/$(cf app MyWebApp-steeltoe --guid)/processes | jq -r '.resources[] | select(.type=="web") | .guid')/0" ssh.run-27.slot-35.tanzu-gss-labs.vmware.com:/home/vcap/app/20230202_193432_22.gcdump ~/Downloads/20230202_193432_22.gcdump
cf:adc061ce-53fe-4537-8353-d0adabed6d9d/0@ssh.run-27.slot-35.tanzu-gss-labs.vmware.com's password:
20230202_193432_22.gcdump                                                           100%  952KB 699.0KB/s   00:01
~/Documents/MyWebApp-steeltoe %

Using dotnet tools manually (On Windows Diego Cells)
This process involves transferring the dotnet tools onto a Windows Diego Cell and running dotnet-dump or dotnet-gcdump against the .NET application process to generate a heap dump. Make sure the Diego Cell has adequate space to store the dotnet tools and the heap dump. The zip archive itself is about ~280 MB and the contents of the extracted folder is presumably much more. The size of the heap dump will approximately match however much memory is being used by the application during the heap dump. 

NOTE: This method may ONLY work on Windows applications pushed with the binary buildpack. Specifically, on applications pushed with the binary_buildpack, we are able to identify the application process with the PowerShell command get-process | findstr "<App Name>" with "<App Name>" being the name of the application that we want to generate the heap dump from.

Here are the steps to generate a heap dump on a .NET application pushed with the binary_buildpack running on a Windows Diego Cell: 

1. Download the dotnet SDK tools onto our Ops Manager VM. In our case we will download dotnet SDK 7.0.404. Or, we can download the dotnet SDK onto our local machine, and transfer it to the Ops Manager VM. Here is the official Microsoft site for .NET 7: https://dotnet.microsoft.com/en-us/download/dotnet/thank-you/sdk-7.0.404-windows-x64-binaries

wget https://download.visualstudio.microsoft.com/download/pr/4f4fc500-65a8-42ca-a70c-c7f8827e7bd0/b6bbe369a0f4c16382811399bdb4a435/dotnet-sdk-7.0.404-win-x64.zip

2. In a separate terminal where we have access to the CF CLI, determine the Windows Diego Cell that the pimsSourceAPI app is running on. If there are multiple IP addresses for the Windows Diego Cell, we can just choose the first IP in the list. 

# Grab the app GUID
cf app $APP_NAME --guid 

# Grab the Windows Diego Cell that the app is running on
cf curl /v2/apps/$(cf app $APP_NAME --guid)/stats | grep "host" | jq '.[].stats.host'

# Log into the Ops Manager VM and find the Diego Cell ID corresponding to the IP found in the previous step
bosh -d <WINDOWS DEPLOYMENT NAME> vms | grep "$DIEGO_CELL_IP"
3. After downloading the SDK zip file, transfer the downloaded SDK to the Windows Diego Cell that the .NET application is running on using bosh SCP. Note that this might take some time to transfer:

bosh -d <WINDOWS DEPLOYMENT NAME> scp dotnet-sdk-7.0.404-win-x64.zip windows_diego_cell/$DIEGO_CELL_ID:C:/Users/dotnet-sdk-7.0.404-win-x64.zip
4. SSH into the Windows Diego Cell that we transferred our Dotnet SDK package to:

bosh -d <WINDOWS DEPLOYMENT NAME> ssh windows_diego_cell/$DIEGO_CELL_ID
5. Start a PowerShell session on the Windows Diego Cell by typing powershell in the CMD command line:
powershell

 

6. Move dotnet-sdk-7.0.404-win-x64.zip to the current Windows Diego Cell user’s home folder.

# Move to the user's directory where we SCP'ed this archive to
cd C:\Users

# Move this to the current user's folder
mv dotnet-sdk-7.0.404-win-x64.zip C:\Users\$env:USERNAME

# Change directories to current user's main folder
cd C:\Users\$env:USERNAME

7. Expand the dotnet SDK archive. Note that this can take up to several minutes to complete:

expand-archive .\dotnet-sdk-7.0.404-win-x64.zip

8. Navigate inside of the dotnet-sdk-7.0.404-win-x64 folder:

cd dotnet-sdk-7.0.404-win-x64
9. Confirm that the dotnet.exe executable is inside of the dotnet-sdk-7.0.404-win-x64 folder:

10. Using the dotnet.exe executable, we go ahead and download the dotnet-dump utility as well as the dotnet-gcdump utility with these following commands:

.\dotnet.exe tool install --global dotnet-dump
.\dotnet.exe tool install --global dotnet-gcdump

11. We navigate back to the current user’s home folder, and notice that a .dotnet folder has been created:

cd C:\Users\$env:USERNAME

12. We navigate to C:\Users\$env:USERNAME\.dotnet\tools directory and we notice that dotnet-dump and dotnet-gcdump have been installed inside here:

cd C:\Users\$env:USERNAME\.dotnet\tools

13. To use either of these heapdump binaries, we first need to set the Dotnet environment variables. We need to set the $env:DOTNET_ROOT environment variable as well as the $env:TMPDIR environment variable. For $env:DOTNET_ROOT we will set this to the location of our dotnet-sdk-7.0.404-win-x64 directory which in this case was C:\Users\$env:USERNAME. For the $env:TMDIR env variable, we can just set this to the user’s Downloads folder. We can set the environment variables as seen below:

$env:DOTNET_ROOT="C:\Users\$env:USERNAME\dotnet-sdk-7.0.404-win-x64\"
$env:TMPDIR="C:\Users\$env:USERNAME\Downloads\"

# We can confirm that the environment variables are properly set
echo $env:DOTNET_ROOT
echo $env:TMPDIR

14. We navigate back to C:\Users\$env:USERNAME\.dotnet\tools and confirm we can run the dotnet-gcdump and dotnet-dump executables

cd C:\Users\$env:USERNAME\.dotnet\tools

.\dotnet-dump.exe
.\dotnet-gcdump.exe

15. Next, we try to look for our application using the get-process command. We can grep for our app name. In our case our app name is mydotnetwebapp

get-process | findstr "mydotnetwebapp"

As a practical example in our internal labs, we had an app named mydotnetwebapp that we tried the heap dump on. This app had the PID of 1456.

16. Next, we can go ahead and generate a dump using dotnet-dump or dotnet-gcdump on the PID of our application. In our case, our PID is 1456 as seen in the screenshot for the previous step.

.\dotnet-dump.exe collect -p <PID>
.\dotnet-gcdump.exe collect -p <PID>

.\dotnet-dump.exe collect -p 1456
.\dotnet-gcdump.exe collect -p 1456

17. From here, we can analyze the dump in real time or transfer the dump back to the Ops Manager VM. For the option of transferring the heap dump file back to the Ops Manager VM, we can open a separate terminal, and execute the bosh scp command seen below:

# Transfer back to Ops Man VM
bosh -d <WINDOWS DEPLOYMENT NAME> scp windows_diego_cell/<Diego Cell ID>:"C:/Users/<Username>/.dotnet/tools/<Dump File Name>" .

# Analyze live
.\dotnet-dump.exe analyze <Heap Dump File>
.\dotnet-gcdump.exe analyze <Heap Dump File>

Transferring .dmp file back to Ops Man VM:

Analyze live with dotnet tools:



Powered by Wolken Software