cc_puma_worker_backlog metric observed spiking in Tanzu Platform for Cloud Foundry

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

When using Tanzu Platform for Cloud Foundry running Cloud Controller Multi-Process Mode (Puma), the metric cc_puma_worker_backlog is observed to be spiking. This indicates a backlog of unprocessed CF API requests. During these spikes, you many experience increased latency and occasional unresponsiveness of CF API operations.

Environment

TPCF

Cause

This indicates an under-scaling of the Cloud Controller PUMA settings. More details on scaling can be found here:

https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/elastic-application-runtime/10-4/eart/managing-cf-multi-process-cloud-controller.html

https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/elastic-application-runtime/10-2/eart/managing-cf-scaling-cloud-controller.html

Resolution

To resolve this, validate your Cloud Controller PUMA configuration, located in Ops Manager -> TAS -> Cloud controller.

From the documentation:

By default, each worker is configured to use 10 threads, meaning that each worker can process up to 10 concurrent requests. We don’t recommend configuring more than 20 threads per worker.

If your configuration is under scaled, please review the above documentation and scale up to your environments needs. 

If you need assistance with the configuration, please raise a case with support:

https://support.broadcom.com/