VMware Tanzu GemFire for VMs mem-check errand fails

Products

• VMware Tanzu Gemfire

Issue/Introduction

The BOSH mem-check errand for VMware Tanzu GemFire for VMs fails with the following error:

Analyzing deployment xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx...\n","stderr":"error getting cluster credentials: error getting deployment manifest: Fetching manifest: Finding deployment ''service-instance_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'': Director responded with non-successful status code ''404'' response ''{\"code\":70000,\"description\":\"Deployment ''service-instance_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'' doesn''t exist\"}

Resolution

The BOSH mem-check errand calculates and outputs the total amount of memory used across all PCC service instances. This errand helps VMware Tanzu GemFire for VMs (Tanzu GemFire) operators monitor resource costs, which are based on memory usage.

The following are scenarios when the BOSH mem-check errand can fail:

1. The underlying infrastructure issues (such as a faulty VM, etc.) can potentially trigger PCC smoke test failures, which in turn can leave an orphaned PCC deployment.
2. A delete service instance operation is in progress during or before the mem-check errand was run.
3. A service instance is deleted through bosh and not cf, the mem-check errand fails on the service instance that was deleted through bosh

To recover from this issue, follow the steps below:

1. Run the BOSH mem-check errand again and check the status. If it still fails, then check the output from the 'bosh deployments' command and verify that this deployment isn't listed.

2. Using the GUID from the error log, recursively search all the Orgs and spaces to locate the orphaned service instance using the 'bosh deployments' command. After locating the orphaned service instance, you can delete the service instances using the CF CLI. 
   
   Note: Before doing so, you must remove any existing service keys and app bindings.
   
   Please refer to the commands below.

Run cf delete-service-key SERVICE-INSTANCE-NAME KEY-NAME to delete the service key.
Run cf unbind-service APP-NAME SERVICE-INSTANCE-NAME to unbind your app from the service instance.
Run cf delete-service SERVICE-INSTANCE-NAME to delete the service instance.

References

• Monitoring VMware Tanzu GemFire Service Instances - https://docs.pivotal.io/p-cloud-cache/1-13/monitoring.html
• Create or Delete a Service Instance - https://docs.pivotal.io/p-cloud-cache/1-13/create-instance.html
• BOSH Deployment Command - https://bosh.io/docs/cli-v2/#deployment-mgmt