How to capture thread dumps of a running task in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This article covers how to capture thread dumps of a running task in Tanzu Application Service for VMs.

Environment

Product Version: 2.9

Resolution

You need to capture a few thread dumps to give you the state of threads at different points of the task execution. You can start with three sets of thread dumps captured at intervals of 10-15 seconds using the procedure described below.

Note: Two sets should be captured while the task is running. And the last one towards the end of its execution.

1. Using cf CLI, run the following command, where taskname is the name of the task.

 cf tasks taskname 

Make sure that it returns an output similar to the following:

Getting tasks for app itask in org system / space test as admin... OK id name state start time command 9 itask RUNNING Thu, 15 Apr 2021 06:10:28 UTC JAVA_OPTS="-agentpath:$PWD/.java-buildpack/open_jdk_jre/bin/jvmkill-1.16.0_RELEASE=printH eapHistogram=1 -Djava.io.tmpdir=$TMPDIR -XX:ActiveProcessorCount=$(nproc) -Djava.ext.dirs=$PWD/.java-buildpack/container_security_provider:$PWD/.java-buildpack/ope n_jdk_jre/lib/ext -Djava.security.properties=$PWD/.java-buildpack/java_security/java.security $JAVA_OPTS" && CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculato r-3.13.0_RELEASE -totMemory=$MEMORY_LIMIT -loadedClasses=12947 -poolType=metaspace -stackThreads=250 -vmOptions="$JAVA_OPTS") && echo JVM Memory Configuration: $CALCULATED_MEMORY && JAVA_OPTS="$JAVA_OPTS $CALCULATED_MEMORY" && MALLOC_ARENA_MAX=2 SERVER_PORT=$PORT eval exec $PWD/.java-buildpack/open_jdk_jre/bin/java $JAVA_OPTS -cp $PWD/. org.springframework.boot.loader.JarLauncher --spring.cloud.task.executionid=19 8 itask RUNNING Thu, 15 Apr 2021 06:10:20 UTC JAVA_OPTS="-agentpath:$PWD/.java-buildpack/open_jdk_jre/bin/jvmkill-1.16.0_RELEASE=printH eapHistogram=1 -Djava.io.tmpdir=$TMPDIR -XX:ActiveProcessorCount=$(nproc) -Djava.ext.dirs=$PWD/.java-buildpack/container_security_provider:$PWD/.java-buildpack/ope n_jdk_jre/lib/ext -Djava.security.properties=$PWD/.java-buildpack/java_security/java.security $JAVA_OPTS" && CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculato r-3.13.0_RELEASE -totMemory=$MEMORY_LIMIT -loadedClasses=12947 -poolType=metaspace -stackThreads=250 -vmOptions="$JAVA_OPTS") && echo JVM Memory Configuration: $CALCULATED_MEMORY && JAVA_OPTS="$JAVA_OPTS $CALCULATED_MEMORY" && MALLOC_ARENA_MAX=2 SERVER_PORT=$PORT eval exec $PWD/.java-buildpack/open_jdk_jre/bin/java $JAVA_OPTS -cp $PWD/. org.springframework.boot.loader.JarLauncher --spring.cloud.task.executionid=18

2. Get cell_id of the cell where the app is running. Using bosh cli, bosh ssh to one of the diego-cells or diego brain and execute

cfdot tasks | jq -r '{command: .action.serial.actions[1].run.args[1], app_id:
.network.properties.app_id, task_guid: .task_guid, state: .state,  cell_id: .cell_id}'

From the output, identify the cell-id.

{

 "command":
"JAVA_OPTS=\"-agentpath:$PWD/.java-buildpack/open_jdk_jre/bin/jvmkill-1.16.0_RELEASE=prin
tHeapHistogram=1 -Djava.io.tmpdir=$TMPDIR -XX:ActiveProcessorCount=$(nproc)
-Djava.ext.dirs=$PWD/.java-buildpack/container_security_provider:$PWD/.java-buildpack/ope
n_jdk_jre/lib/ext
-Djava.security.properties=$PWD/.java-buildpack/java_security/java.security $JAVA_OPTS\"
&&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculato
r-3.13.0_RELEASE -totMemory=$MEMORY_LIMIT -loadedClasses=12947 -poolType=metaspace
-stackThreads=250 -vmOptions=\"$JAVA_OPTS\") && echo JVM Memory Configuration:
$CALCULATED_MEMORY && JAVA_OPTS=\"$JAVA_OPTS $CALCULATED_MEMORY\" && MALLOC_ARENA_MAX=2
SERVER_PORT=$PORT eval exec $PWD/.java-buildpack/open_jdk_jre/bin/java $JAVA_OPTS -cp
$PWD/. org.springframework.boot.loader.JarLauncher --spring.cloud.task.executionid=19",
 "app_id": "8ac8d492-1d4c-48d2-8093-3d1d0c087c51",
 "task_guid": "7ff51ede-272f-4cf9-bf2d-5dbffd78b395",
 "state": "Running",
 "cell_id": "f0639a0a-5b74-4191-bd66-7bb343ec45f8"
} {

 "command":
"JAVA_OPTS=\"-agentpath:$PWD/.java-buildpack/open_jdk_jre/bin/jvmkill-1.16.0_RELEASE=prin
tHeapHistogram=1 -Djava.io.tmpdir=$TMPDIR -XX:ActiveProcessorCount=$(nproc)
-Djava.ext.dirs=$PWD/.java-buildpack/container_security_provider:$PWD/.java-buildpack/ope
n_jdk_jre/lib/ext
-Djava.security.properties=$PWD/.java-buildpack/java_security/java.security $JAVA_OPTS\"
&&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculato
r-3.13.0_RELEASE -totMemory=$MEMORY_LIMIT -loadedClasses=12947 -poolType=metaspace
-stackThreads=250 -vmOptions=\"$JAVA_OPTS\") && echo JVM Memory Configuration:
$CALCULATED_MEMORY && JAVA_OPTS=\"$JAVA_OPTS $CALCULATED_MEMORY\" && MALLOC_ARENA_MAX=2
SERVER_PORT=$PORT eval exec $PWD/.java-buildpack/open_jdk_jre/bin/java $JAVA_OPTS -cp
$PWD/. org.springframework.boot.loader.JarLauncher --spring.cloud.task.executionid=18",
 "app_id": "8ac8d492-1d4c-48d2-8093-3d1d0c087c51",
 "task_guid": "94cf228f-4124-4d44-8b14-cd9845ec1b0e",
 "state": "Running",
 "cell_id": "a681004a-885c-4bcb-beb5-38157ef9dd5e"
}

3. Using the cell_id from step 2, run the command this commnand:

​​​​​​bosh -d  cf-xxxxx ssh diego_cell/<cell-id>

4. Root to container with this command:

/var/vcap/packages/runc/bin/runc --root /run/containerd/runc/garden exec -t TASK_GUID
  /bin/bash

 

5. Execute this command to get PID:

ps -auxww | grep java

 

6. Run this command:.

 kill -3 PID

7. The thread dump is located and shown in the cf logs.