BOSH tasks fail intermittently with CredhubClient::FailedRequest request could not be fulfilled

Products

• VMware Tanzu Application Service

Issue/Introduction

BOSH tasks are failing intermittently with errors such as - 

Error inserting ... into Credhub:

CredhubClient::FailedRequest -  

Credhub request returned 400 with error The request could not be fulfilled because the request path or body did not meet expectation. Please check the documentation for required formatting and retry your request.
Exited with 1.

BOSH director has high CPU and memory utilization.

Resolution

This problem is often associated with a BOSH director VM which is undersized for load placed onto it. This may happen in large environments with many deployments and frequent tasks. 

Scale the VM type of BOSH director to double the CPU and memory resources allocated under OpsManager -> Director Tile -> Resource Config. 

Additionally, it may be necessary to double the number of Workers under Ops Manager -> Director Tile -> Director Config Screen.

Run apply changes to scale the director and monitor the resource utilization (CPU & memory) on director after applied.