Upgrade TKGI stuck Step 5/8 in "Uploading runtime config releases to the director"
search cancel

Upgrade TKGI stuck Step 5/8 in "Uploading runtime config releases to the director"

book

Article ID: 340500

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:

  • Upgrade failure on TKGi Management Console with error "Failed step 5/8: Deploying TKGI, Installation was unsuccessful. Error: Uploading runtime config releases to the director "
  • TKGI MC Upgrade Failure on Deploying pivotal-container-service with "Error: Failed to upload blob, code 1, output: 'Error running app - Putting dav blob xxx-xxx-xxx-xxx : Wrong response code: 500'
  • The upgrade failed on the operation of "Deploy pivotal-container-service.
  • Error code reported looks like:

    Exit code 1 ===== 2025-08-06 23:00:36 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=<BOSH_IP> --deployment=pivotal-container-service-########a2db7d3a7829 deploy --no-redact /var/tempest/workspaces/default/deployments/pivotal-container-service-########a2db7d3a7829.yml"; Duration: 207s; Exit Status: 1 Exited with 1.

  • The Opsmanager change log shows errors like:

    Task 9629721 | 23:00:28 | Creating new compiled packages: antrea-images/3041a794-####-####-####-c29da0da010d for ubuntu-jammy/1.506 (00:00:15)
                           L Error: Failed to upload blob, code 1, output: 'Error running app - Putting dav blob 3041a794-####-####-####-c29da0da010d: Wrong response code: 500; body: &lt;html&gt;&lt;head&gt;&lt;title&gt;500 Internal Server Error&lt;/title&gt;&lt;/head&gt;


  • Reviewing logs collected from the Opsman Support Bundle, you see errors in the bosh_director_logs/logs.zip_extracted/create-env-vm-logs-<DATE>-<TIME>-.tgz_extracted/blobstore/blobstore_error.log similar to:

    [alert] 12#0: *149 write() "/var/vcap/store/blobstore/store/a7/3041a794-####-####-####-c29da0da010d.0000000284" failed (28: No space left on device), client: <BOSH_IP>, server: , request: "PUT /a7/3041a794-####-####-####-c29da0da010d HTTP/1.1", host: "<BOSH_IP>:25250"

 

 

Environment

This was observed on a TKGI environment managed by Management Console during upgrade from 1.19.2 to 1.20, however, versioning is not critical for this failure condition.

Cause

This failure occurs when the Blobstore is configured to reside on Bosh director VM and the Bosh director runs out of space on the Persistent Disk under /var/vcap/store.

 

Resolution

Clean up the Bosh director persistent disk:

 

OR,

 

Expand the Bosh director persistent disk:

  1. Opsmanager UI > Bosh Director tile > Resource Config > increase the disk capacity
  2. Apply Changes in Opsmanager UI to ONLY THE BOSH TILE, Bosh director will be recreated with the new disk and data will be automatically copied from old disk to new disk
  3. Re-apply changes again on TKGi tile from TKGi Management Console to continue the upgrade 

 

 

Powered by Wolken Software