Diego database is failing and unable to connect to NCP

Products

• VMware Tanzu Application Service
• VMware NSX for vSphere
• VMware NSX

Issue/Introduction

1. A user faces their diego database vm's failing.

# Example output from 'bosh vms' command:

diego_database/###-###-###-1234                  failing AZ2     10.xx.xxx.xx    vm-<NAME-GUID> medium          true   bosh-vsphere-esxi-ubuntu-jammy-go_agent/1.824

 

2. NSX logs will show errors such as:

500 Internal Server Error

The HTTP request returned error code 500, whereas 200 response codes were expected.

The server has either erred or is incapable of performing the requested operation

Cause

One reason for the issue/error is due to the NSX Manager's Appliance '/temp' directory is full. This can be confirmed by running 'df-h' command (as root) on nsx disk and checking '/temp' directory resources. 

Resolution

Reboot the NSX manager to reset the /temp directory resource.