TKGi MC (Management Console) upgrade from TKGi 1.17.6 to 1.18.5 - BOSH Director reporting 'Missing stemcell' and error advising required_stemcell_version 1.486 and required_stemcell_os ubuntu-jammy.

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

In the TKGi MC (Management Console) after upgrading from TKGI 1.17.6 to TKGi 1.18.5 the below errors are observed - where BOSH director advises that there is a 'Missing Stemcell' and the upgrade fails with below error advising required_stemcell_version 1.486 and required_stemcell_os ubuntu-jammy.

Perform network configuration SuccessCompleted
 
Deploy ops-manager 3.0.31+LTS-T SuccessCompleted
 
Generate and register certificates SuccessCompleted
 
Deploy bosh SuccessCompleted
 
Deploy pivotal-container-service 1.18.5-build.10 FailedFailed step 5/8: Deploying TKGI, installation failed to trigger: request failed: unexpected response: HTTP/1.1 422 Unprocessable Content Content-Length: 408 Cache-Control: private, no-store Connection: keep-alive Content-Security-Policy: script-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; object-src 'none'; Content-Type: application/json; charset=utf-8 Date: Thu, 19 Sep 2024 09:59:17 GMT Expires: Fri, 01 Jan 1990 00:00:00 GMT Pragma: no-cache Referrer-Policy: strict-origin-when-cross-origin Server: Ops Manager Strict-Transport-Security: max-age=63072000; includeSubDomains X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: SAMEORIGIN X-Permitted-Cross-Domain-Policies: none X-Request-Id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx X-Runtime: 7.503894 {"errors":["Installation is incomplete. One or more tiles require configuration."],"deployment_errors":{"products":[{"identifier":"p-bosh-xxxxxxxxxxxxxxxxxxxx","complete":false,"network":{"assigned":true},"availability_zone":{"assigned":true},"stemcells":[{"assigned":true,"required_stemcell_version":"1.486","required_stemcell_os":"ubuntu-jammy"}],"properties":[],"resources":{"jobs":[]},"verifiers":[]}]}}: installation failed to trigger: request failed: unexpected response: HTTP/1.1 422 Unprocessable Content Content-Length: 408 Cache-Control: private, no-store Connection: keep-alive Content-Security-Policy: script-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; object-src 'none'; Content-Type: application/json; charset=utf-8 Date: Thu, 19 May 2024 10:00:05 GMT Expires: Fri, 01 Jan 1990 00:00:00 GMT Pragma: no-cache Referrer-Policy: strict-origin-when-cross-origin Server: Ops Manager Strict-Transport-Security: max-age=63072000; includeSubDomains X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: SAMEORIGIN X-Permitted-Cross-Domain-Policies: none X-Request-Id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx X-Runtime: 7.503894 {"errors":["Installation is incomplete. One or more tiles require configuration."],"deployment_errors":{"products":[{"identifier":"p-bosh-xxxxxxxxxxxxxxxxxxxx","complete":false,"network":{"assigned":true},"availability_zone":{"assigned":true},"stemcells":[{"assigned":true,"required_stemcell_version":"1.486","required_stemcell_os":"ubuntu-jammy"}],"properties":[],"resources":{"jobs":[]},"verifiers":[]}]}}

Environment

TKGi 1.17.6 to 1.18.5
BOSH 3.0.31+LTS-T

Cause

The TKGi MC for 1.18.5 comes preload with base stemcell 1.486ubuntu-jammy

Resolution

1. SSH to 1.18.5 TKGI MC VM server( on either upgrade failed TKGi MC VM or on the newly deploy TKGi MC VM 1.18.5)
2. Stop TKGi API server by the following command:
   
   

   systemctl stop pks-mgmt-server.service

   
   
   
3. Backup DB and artifacts.yaml
   
   

   mkdir /root/backup
   
   cp /storage/data/artifacts/artifacts.yaml /root/backup/
   
   cp /storage/data/pks.db /root/backup/

4. Download 1.506 stemcell file to the folder /storage/data/artifacts/stemcells-ubuntu-jammy/1.506
   
   

   mkdir /storage/data/artifacts/stemcells-ubuntu-jammy/1.506
   
   cd /storage/data/artifacts/stemcells-ubuntu-jammy/1.506
   
   wget <download_url>

   
   
   
5. Update the version and path of pks-stemcell and harbor-stemcell to 1.506 in the /storage/data/artifacts/artifacts.yaml
   
   

   vi /storage/data/artifacts/artifacts.yaml

6. Replace the pks-stemcell and harbor-stemcell section with the following section:
   
   

   pks-stemcell:   name: pks-stemcell  version: "1.506"
   path: /storage/data/artifacts/stemcells-ubuntu-jammy/1.506/bosh-stemcell-1.506-vsphere-esxi-ubuntu-jammy-go_agent.tgz
   harbor-stemcell:   name: harbor-stemcell  version: "1.506"
   path: /storage/data/artifacts/stemcells-ubuntu-jammy/1.506/bosh-stemcell-1.506-vsphere-esxi-ubuntu-jammy-go_agent.tgz

7. Start TKGI api server by following command
   
   

   systemctl start pks-mgmt-server.service

8. Open 1.18.5 TKGI MC UI.
   
   8.1 continue to your upgrade as usual(If you deployed a new TKGI MC VM)
   
   or
   
   8.2 continue to your upgrade by applying the upgrade configuration again from UI(from TKGi MC VM that has failed)