Cannot run Bosh commands with error " Director responded with non-successful status code '500' response "

Products

• VMware Tanzu Application Service

Issue/Introduction

Symptoms:

Bosh commands aren’t working, and a similar error is thrown:

Error: Status: 500; ErrorMessage: <nil>; Description: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: ######, broker-request-id: ######, operation: bind - error-message: gathering binding info Could not fetch VMs info for deployment "service-instance_######": Listing deployment 'service-instance_######' vms infos: Director responded with non-successful status code '500' response ''; ResponseError: <nil>

 

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Cause

Not enough space is left in the bosh director persistent disk

Resolution

1- To validate that the issue is indeed related to not having enough  space left issue in the director persistent disk , please ssh to the bosh director following the instructions mentioned in the following doc: https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-operations-manager/3-0/tanzu-ops-manager/install-trouble-advanced.html

2- Check the bosh director logs stored under var/vcap/sys/logs and you will find similar errors are prompted in the credhub and the postgres logs

"base/16386/pg_internal.init.4894": No space left on device

When running df -h you should find /dev/sdc1 fully consumed simlar to the below example

bosh/0:~$ df -h

Filesystem     Size Used Avail Use% Mounted on

devtmpfs       3.9G    0 3.9G  0% /dev

tmpfs          3.9G    0 3.9G  0% /dev/shm

tmpfs          3.9G  25M 3.9G  1% /run

tmpfs          5.0M    0 5.0M  0% /run/lock

tmpfs          3.9G    0 3.9G  0% /sys/fs/cgroup

/dev/sda1      2.9G 1.5G 1.3G 55% /

/dev/sdb2       56G 7.7G  45G 15% /var/vcap/data

tmpfs           16M 372K  16M  3% /var/vcap/data/sys/run

/dev/sdc1       50G  47G    0 100% /var/vcap/store

 

Also you can navigate to the operations manager GUI and from the bosh director tile , you can check the bosh director VM persistent disk consumption percentage as well .

3- Increase the persistent disk on the Bosh director tile from the resource config tab and Apply Changes which  should solve the issue.