How to collect pprof data for otel-collector in EAR (Elastic Application Runtime)

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

pprof is Go builtin profiling tool that allows users to collect and analyze runtime data from applications, such as CPU usage, memory allocations, goroutines, and blocking operations.

Collecting pprof data is often necessary when investigating performance issues related to the OpenTelemetry Collector.

This article provides guidance on how to collect pprof data for the OpenTelemetry Collector in EAR (Elastic Application Runtime).

Resolution

Currently, pprof is disabled by default and must be explicitly enabled either for a specific VM or EAR system-wide.

Method one (for a specific VM):

1 - SSH into the affected VM and switch to the root privilege. Command examples:

• bosh -d DEPLOYMENT-NAME ssh diego_cell/0
• sudo -i

2 - Edit /var/vcap/jobs/otel-collector/config/config.yml and add the following configuration.

---
extensions:
  pprof:
service:
  extensions:
  - pprof 

3. Run monit restart otel-collector to make the change take effect.

Method two (EAR system-wide):

Using this method will require all VMs to be restarted.

1 - Adding the following configuration in Ops Man UI > TPCF > System Logging/OTEL > OpenTelemetry Collector Configuration. 

---
extensions:
  pprof:
service:
  extensions:
  - pprof 

2 - Check the Enable additional Open Telemetry Collector features box. 

3 - Perform apply change against EAR tile

 

Once pprod is enabled, you can go ahead SSH into the affected VM and generate pprof file.

$ curl http://localhost:1777/debug/pprof/heap -o /tmp/heap.pprof

Then use bosh scp command to copy the output file from the VM and upload it to the support ticket for further review.

$ bosh -d DEPLOYMENT-NAME scp diego_cell/0:/tmp/heap.pprof ~/Downloads

You can optionally remove the configuration change if pprof is no longer needed.