TKGi update cluster fails with error "Unable to render instance groups for deployment...Blobstore object 'GUID' not found"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Apply Changes in OpsMan shows the following error for a TKGi cluster update:

Task 12345 | 10:11:33 | Preparing deployment: Rendering templates (00:00:11)
 L Error: Unable to render instance groups for deployment. Errors are:
 Blobstore object '<Object-ID>' not found

Cause

Missing or corrupted releases in the Blobstore can cause this error.

It's been observed that in certain situations "bosh clean-up" command can cause this.

Resolution

Generally, the steps outlined in KB Get "Error: Unable to render instance groups for deployment...Blobstore object 'GUID' not found" during BOSH deployment can be followed.

However, not all the releases may be used just by the TKGi clusters. Some of the releases may be coming from Bosh configs or from other Bosh deployments different to TKGi. This means that once the release and job names have been identified with "cat  task.debug.log | grep <BLOB_GUID>", they may not show up in the manifest.yml file.

Releases coming from Bosh configs:

The job identified in the task.debug.log file may be coming from a Bosh config.
Bosh runtime configs are applied to all deployments. More information can be found in https://bosh.io/docs/runtime-config/ 

• To list Bosh configs:
  # bosh configs
  
  E.g.:
  

  $ bosh configs
ID   Type     Name                                                                    Team                                            Created At
4*   cloud    default                                                                 -                                               2025-03-20 16:56:50 UTC
34*  cloud    pivotal-container-service-<id>                                          pivotal-container-service-<id>                  2025-03-20 17:33:35 UTC
43*  cloud    service-instance_<id>                                                   pivotal-container-service-<id>                  2025-03-22 00:20:44 UTC
41*  cloud    service-instance_<id>                                                   pivotal-container-service-<id>                  2025-03-21 20:53:20 UTC
42*  cpi      default                                                                 -                                               2025-03-21 22:52:14 UTC
3*   runtime  director_runtime                                                        -                                               2025-03-20 16:56:50 UTC
35*  runtime  harbor-container-registry-<id>-harbor-bosh-dns-aliases                  -                                               2025-03-20 17:41:19 UTC
1*   runtime  ops_manager_dns_runtime                                                 -                                               2025-03-20 16:56:49 UTC
2*   runtime  ops_manager_system_metrics_runtime                                      -                                               2025-03-20 16:56:49 UTC

• Then, try to find the job/release names in the Bosh configs listed with the previous command.
  For example, if the job's name is "bosh-dns", we'll find it in "ops_manager_dns_runtime" Bosh config:
  
  $ bosh config <bosh-config-id>
  jobs:
  - name: bosh-dns
    release: bosh-dns
 releases:
 - name: bosh-dns
   version: 1.36.12
  
  
• The release associated to "bosh-dns" job is called "bosh-dns". This is an Open Source Bosh Release that can be found in https://bosh.io/releases/github.com/cloudfoundry/bosh-dns-release?all=1 
  There, we'll look for the specific version, 1.36.12 in the above example, and click on See Usage Details:
  
  
  It will show a command to upload the release directly from Bosh Github.
  
  
  In our case, since we want to fix corrupted releases, we'll need to append "--fix" flag at the end:
  
  # bosh upload-release --sha1 0502f4dea0dce66b390c8f5f721285d91eb68a7e "https://bosh.io/d/github.com/cloudfoundry/bosh-dns-release?v=1.36.12" --fix
  
  
• For any other Bosh config release, we can search the correct Bosh reference to upload the release by searching in a browser "bosh release <release-name>".
  For example, for "system-metrics" release:
  
  

 

Releases coming from Bosh deployments different to TKGi:

If during OpsMan Apply Changes we're updating several deployments, including non-TKGi ones, the problematic Blobstore object may be coming from one of these other deployments.
If the job/release name can't be found neither in the TKGi manifest.yml file nor in the Bosh configs, most likely this is the case.

• In OpsMan GUI identify all the tiles in use.
• Go to Support Portal Downloads and search the tile and version.
  
  For example, for Harbor Container Registry:
  
  
  
  
  
  
  
  
• Click on "I agree to the Terms and Conditions" and download the tile.
  
  
  
  
• The downloaded file will have a name "<tile-name>.pivotal", for example, harbor-container-registry-2.11.0-build.2.pivotal.
  Unzip the file:
  # unzip harbor-container-registry-2.11.0-build.2.pivotal
  
  A directory called "releases" is extracted. Inside the directory there'll be a ".tgz" file containing the necessary releases for the tile.
  
  # ls -lrt
-rwx------+ 1 <user> UsersGrp 1140248413 Mar 22 14:08 harbor-container-registry-2.11.0-build.2.pivotal
drwxrwxrwx+ 1 <user> UsersGrp          0 Mar 24 13:02 metadata
drwxr-xr-x+ 1 <user> UsersGrp          0 Mar 24 13:02 migrations
drwxrwxrwx+ 1 <user> UsersGrp          0 Mar 24 13:02 releases

# ls -lrt releases/
-rw-r--r--+ 1 <user> UsersGrp 1140190623 Mar 24 13:02 harbor-container-registry-2.11.0.tgz
  
  
• Upload the ".tgz" releases file to the machine where you're running the bosh commands from.
  Once there, re-upload the release:
  # bosh upload-release <release-name>.tgz --fix
  
  E.g.:
  # bosh upload-release harbor-container-registry-2.11.0.tgz --fix