How to fix the stemcell validation error with "Cannot open: No such file or directory"

Products

• Operations Manager

Issue/Introduction

After upgrading Ops Manager, it may fail with Apply Changes because of the following stemcell validation error with "Cannot open: No such file or directory."

Started validating
  Validating release 'bosh'... Finished (00:00:01)
    :
  Validating stemcell... Failed (00:00:00)
Failed validating (00:00:18)
Extracting stemcell from '/var/tempest/stemcells/bosh-stemcell-621.XYZ-**********-ubuntu-xenial-go_agent-raw.tgz':
  reading extracted stemcell manifest in '/home/tempest-web/.bosh/installations/********-****-****-****-************/tmp/stemcell-manager************':
    Extracting stemcell from '/var/tempest/stemcells/bosh-stemcell-621.XYZ-**********-ubuntu-xenial-go_agent-raw.tgz' to '/home/tempest-web/.bosh/installations/********-****-****-****-************/tmp/stemcell-manager************':
      Shelling out to tar:
        Running command: 'tar --no-same-owner -xzf /var/tempest/stemcells/bosh-stemcell-621.XYZ-**********-ubuntu-xenial-go_agent-raw.tgz -C /home/tempest-web/.bosh/installations/********-****-****-****-************/tmp/stemcell-manager************', stdout: '', stderr: 'tar (child): /var/tempest/stemcells/bosh-stemcell-621.XYZ-openstack-kvm-ubuntu-xenial-go_agent-raw.tgz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
':
          exit status 2

Exit code 1

===== 2022-06-14 09:10:11 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty create-env /var/tempest/workspaces/default/deployments/bosh.yml"; Duration: 18s; Exit Status: 1
{"type":"step_finished","id":"bosh_product.deploying","description":"Installing BOSH"}
Exited with 1.could not execute "apply-changes": installation was unsuccessful

This article introduces how to fix this error.

Environment

Product Version: 2.10

Resolution

In this particular case, you will list the product stemcell assignments for BOSH Director with the following commnad execution with using om CLI and check actual stemcell assignment:

$ om curl -p /api/v0/stemcell_associations | jq '.products[] | select(.identifier == "p-bosh")'

From the result of the above command, you will find that stemcell versions specified at "staged_stemecells" and "available_stemcells" are different as a below example:

{
  "guid": "p-bosh-b*bb2***a372***6a*6*",
  "identifier": "p-bosh",
  "label": "BOSH Director",
  "staged_product_version": "2.10.39-build.450",
  "deployed_product_version": "2.10.39-build.450",
  "is_staged_for_deletion": false,
  "staged_stemcells": [
    {
      "os": "ubuntu-xenial",
      "version": "621.123"
    }
  ],
  "deployed_stemcells": [
    {
      "os": "ubuntu-xenial",
      "version": "621.123"
    }
  ],
  "available_stemcells": [
    {
      "os": "ubuntu-xenial",
      "version": "621.236"
    }
  ],
  "required_stemcells": [
    {
      "os": "ubuntu-xenial",
      "version": "621.236",
      "enable_patch_security_updates": false
    }
  ]
}

In the above particular example, Ops Manager tries to extract the stemcell v621.123 as specifying at "staged_stemcells" section while the actual available version is 621.236 as specifying at "available_stemcells" section. As a result, Ops Manager can't find the stemcell v621.123 and Apply Changes fails with stemcell validation error with "Cannot open: No such file or directory."

You can fix the error by assign available stemcell with om assign-atemcell command. the above particular example, you will perform the following command line to assign the stemcell v621.236 as a staged stemcell verion.

om assign-stemcell -p p-bosh -s 621.236

After that, perform Apply Changes and it succeds witout the stemcell validation error.