High CPU and Memory Usage on Gorouter and Diego Database VMs After Upgrade to EAR 10.4

Products

• VMware Tanzu Platform Core
• VMware Tanzu Application Service

Issue/Introduction

• High CPU and memory usage is observed on Gorouter and Diego Database VMs following foundation upgrades to 10.4.x versions. This resource exhaustion leads to container creation failures and external network errors. The root cause is the otel-collector process being managed by opamp-supervisor, which lacks a default memory cap. This causes memory to grow significantly when platform logging is configured.
• CPU and Memory is consumed by the otel-collector process:
  
  

Environment

• Elastic Application Runtime (EAR) 10.4.x
• Components: Gorouter, Diego Database

Cause

In EAR 10.4.x, the otel-collector process is managed by the opamp-supervisor BPM file. Unlike previous versions where the otel-collector was standalone and capped at 768MB, the opamp-supervisor configuration lacks this memory limit, causing unrestricted growth.

Resolution

This is fixed in EAR 10.4.4 patch release.

 

Workaround:

Apply memory limits to the opamp-supervisor process and the otel-collector pipeline.

1. SSH into the affected VM.
2. Edit /var/vcap/jobs/opamp-supervisor/config/bpm.yml.
3. Add the following memory limit configuration after line 11:
   
   limits:
    processes: 1024
    memory: 832Mib <-- new line
   
   
4. Update /var/vcap/jobs/otel-collector/config/config.yml to add the following processors if not already configured:
   
   processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 614
    spike_limit_mib: 153
  batch: 
   
   
5. Add the memory limiter and batch processors to the platform log pipeline within the same configuration file:
   
   logs/cf-internal-platform-logs:
   exporters:
      - ...
   processors:
      - memory_limiter
      - batch
   receivers:
      - filelog/cf-internal-component-logs
      - filelog/cf-internal-syslog
   
   
6. Restart: Restart the opamp-supervisor monit process:
   
   monit restart opamp-supervisor

 

 

 

Additional Information

If you require further assistance or the issue persists after applying the workaround, reach out to Broadcom Support. To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.