Cannot run Bosh commands with error "Director responded with non-successful status code '500' response"

Products

• VMware Tanzu Application Service

Issue/Introduction

Symptoms:

Bosh commands aren’t working, and a similar error is returned:

Error: Status: 500; ErrorMessage: <nil>; Description: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: ######, broker-request-id: ######, operation: bind - error-message: gathering binding info Could not fetch VMs info for deployment "service-instance_######": Listing deployment 'service-instance_######' vms infos: Director responded with non-successful status code '500' response ''; ResponseError: <nil>

 

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

OpsManager/Bosh director vm

Cause

Not enough space is left in the bosh director persistent disk

Resolution

1. To validate that the issue is indeed related to not having enough space left in the director persistent disk , ssh to the bosh director following the instructions mentioned in the doc:  Advanced Tanzu Operations Manager troubleshooting with BOSH CLI

2. Check the bosh director logs stored under var/vcap/sys/logs and you will find similar errors are prompted in the credhub and the postgres logs:

"base/16386/pg_internal.init.4894": No space left on device

When running df -h you should find /dev/sdc1 fully consumed similar to the below example

bosh/0:~$ df -h
Filesystem     Size Used Avail Use% Mounted on
devtmpfs       3.9G    0 3.9G  0% /dev
tmpfs          3.9G    0 3.9G  0% /dev/shm
tmpfs          3.9G  25M 3.9G  1% /run
tmpfs          5.0M    0 5.0M  0% /run/lock
tmpfs          3.9G    0 3.9G  0% /sys/fs/cgroup
/dev/sda1      2.9G 1.5G 1.3G 55% /
/dev/sdb2       56G 7.7G  45G 15% /var/vcap/data
tmpfs           16M 372K  16M  3% /var/vcap/data/sys/run
/dev/sdc1       50G  47G    0 100% /var/vcap/store

Also you can navigate to the operations manager GUI and from the bosh director tile , you can check the bosh director VM persistent disk consumption percentage.

3. Increase the persistent disk on the Bosh director tile from the resource config tab and Apply Changes to resolve the issue.