ERROR: "UAA responded with non-successful status code '500' response '{}'" returned when running bosh commands

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)
• VMware Tanzu Application Service

Issue/Introduction

• When running bosh commands such as bosh vms or bosh stemcells, users may see errors like:
  
  Requesting token via client credentials grant: UAA responded with non-successful status code '500' response '{}'

EXAMPLE:
  
  
  
  
• This is more likely to occur on environments that have a large number of tiles or clusters configured and managed by Bosh Director
• When running bosh tasks commands, users might see many tasks in queued state.
• From Opsman GUI, selecting the Bosh Director tile, then the Status tab, the PERS. DISK column will show high consumption:
  
  
  
  
• From an SSH to Bosh Director VM, running df -h commands will show the /var/vcap/store directory 100% full:
  
  
  
  
• Running monit summary within Bosh Director VM SSH might show all processes running, or fluctuating between running and not monitored state.
• While SSH'ed into Bosh Director VM, running a du -sh * from the /var/vcap/store directory will show the blobstore as the primary consumer of disk space:
  
  

 

Environment

Bosh Director VM

Bosh Director manages VM's in TKGI and Tanzu Application Services, so either product may be impacted by this condition.

Cause

This error occurs when the persistent disk used by Bosh Director reaches critical consumption over 95%, leading to service failures and preventing the director from operating normally.

Resolution

This KB seeks specifically to provide details on resolving the issue by increasing the size of the persistent disk associated with Bosh Director if it hits 100% consumption, particularly if the /var/vcap/store/blobstore directory is the offending storage location. Some other resolutions to this issue are documented in the below KBs:

• KB 293686 Cleanup process for bosh Database
• KB 298121 Cleanup process for log bundles

 

Steps to increase Bosh Director Persistent disk to allow cleanup operations

 

The following reference documentation can be used to for BOSH Director Resource Config changes

1. 1. Connect to the Operations Manager GUI. 
   2. Navigate to the Bosh Director tile.
   3. Under Settings, select Resource Config
   4. On the Job named BOSH Director, under PERSISTENT DISK TYPE, change the dropdown to increase the disk size to the next larger size than current config
   5. Click the SAVE button
   6. Return to the INSTALLATION DASHBOARD and select REVIEW PENDING CHANGES.
   7. Deselect all products except BOSH Director, then click the APPLY CHANGES button.

 

This will redeploy the Bosh Director VM with a new disk. The data will be migrated from the old, small disk, to the new, larger disk. 

 

Once the Bosh Director VM has been updated with the larger disk, cleanup can be performed on the blobstore using the bosh clean-up commands referenced in the BOSH CLI documentation