Cannot run Bosh commands with error " Director responded with non-successful status code '500' response "

Products

• VMware

Issue/Introduction

Symptoms:

Bosh commands are not working and a similar error is thrown

 

 

Error: Status: 500; ErrorMessage: <nil>; Description: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: 9e5db8ec-d9da-4f77-b4d6-61345f3569e0, broker-request-id: 15ede0c7-a870-4ed1-8cb5-cc1dd762f710, operation: bind - error-message: gathering binding info Could not fetch VMs info for deployment "service-instance_9e5db8ec-d9da-4f77-b4d6-61345f3569e0": Listing deployment 'service-instance_9e5db8ec-d9da-4f77-b4d6-61345f3569e0' vms infos: Director responded with non-successful status code '500' response ''; ResponseError: <nil>

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Cause

No enough  space is left in the bosh director persistent disk

Resolution

1- To validate that the issue is indeed related to not having enough  space left issue in the director persistent disk , please ssh to the bosh director following the instructions mentioned in the following doc https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/install-trouble-advanced.html

2- Check the bosh director logs stored under var/vcap/sys/logs and you will find similar errors are prompted in the credhub and the postgres logs

 

"base/16386/pg_internal.init.4894": No space left on device

When running df -h you should find /dev/sdc1 fully consumed simlar to the below example

 

bosh/0:~$ df -h

Filesystem     Size Used Avail Use% Mounted on

devtmpfs       3.9G    0 3.9G  0% /dev

tmpfs          3.9G    0 3.9G  0% /dev/shm

tmpfs          3.9G  25M 3.9G  1% /run

tmpfs          5.0M    0 5.0M  0% /run/lock

tmpfs          3.9G    0 3.9G  0% /sys/fs/cgroup

/dev/sda1      2.9G 1.5G 1.3G 55% /

/dev/sdb2       56G 7.7G  45G 15% /var/vcap/data

tmpfs           16M 372K  16M  3% /var/vcap/data/sys/run

/dev/sdc1       50G  47G    0 100% /var/vcap/store

 

Also you can navigate to the operations manager GUI and from the bosh director tile , you can check the bosh director VM persistent disk consumption percentage as well .

3- Increase the persistent disk on the Bosh director tile from the resource config tab and Apply Changes which  should solve the issue.