"Missing stemcell from disk" error after OpsMan upgrade

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

After upgrading Ops Manager (e.g., from version 3.1.x to 3.2.x), BOSH deployments may fail with errors indicating that a required stemcell is missing or cannot be found on the OpsMan disk, even if a newer stemcell was previously uploaded to the environment. Specifically this issue may arise if the stemcell version that is used in environment is higher than the stemcell version builtin in new OpsManager version. 

You may see an error like shown below in the api output. In the ui you will see "Missing stemcell" message on the tile. Checking the Stemcell Library, a message saying "Stemcell missing from disk" will be shown. 

{"errors":["Installation is incomplete.  One or more tiles require configuration."],"deployment_errors":{"products":[{"identifier":"p-bosh-xyz","complete":false,"network":{"assigned":true},"availability_zone":{"assigned":true},"stemcells":[{"assigned":true,"required_stemcell_version":"1.1183","required_stemcell_os":"ubuntu-jammy"}],"properties":[],"resources":{"jobs":[]},"verifiers":[]}]}}

Environment

OpsManager/Bosh Director/ Stemcells

Cause

The BOSH Director requires the stemcell to be present on the Ops Manager VM disk for certain operations. During an Ops Manager upgrade, the existing Ops Manager disk is typically replaced.

Because stemcells are not included in the Ops Manager export ZIP file, the new Ops Manager instance will not have the stemcells that were manually uploaded to the previous version, leading to a discrepancy where the Director expects a stemcell that no longer exists on the local disk.

Resolution

To resolve this issue, you must re-upload the required stemcell to the new Ops Manager instance.

1. Identify the specific stemcell version required by the BOSH Director (this is usually the version that was uploaded to the previously).
2. Download the corresponding stemcell from the Broadcom Support Portal.
3. Log in to the Ops Manager VM or use the Ops Manager UI/API to upload the stemcell again. Use command bosh upload-stemcell <path-to-stemcell>
4. Verify the upload by running bosh stemcells to ensure the Director sees the stemcell as available.