Bosh health check VM creation fails

Products

• VMware Tanzu Application Service

Issue/Introduction

bosh-health-check VM  creation fails due to the below error .

request ID: 'cpi-12345')
Task 8 | 08:21:36 | Error: CPI error 'Bosh::Clouds::CloudError' with message 'could not find AMI 'ami-xxxxxxxxxxxx'' in 'create_vm' CPI method (CPI request ID: 'cpi-12345')
Updating deployment:
  Expected task '8' to succeed but state is 'error'ç

This can actually happen with any deployment but since Healthwatch deploys bosh-health-check every 5 minutes, the problem will probable surface with it if Healthwatch is installed.

Environment

TAS/Healthwatch on AWS

Cause

Why the AMI disappears in AWS is unknown. Contact AWS for to get the root case.

Resolution

Run "bosh stemcells" to find the stemcell containing the missing AMI. Let's say that the one missing is ami-12345678910

$ bosh stemcells
Using environment '10.xxx.xxx.xxx' as client 'ops_manager'

Name                                     Version  OS             CPI                               CID
bosh-aws-xen-hvm-ubuntu-jammy-go_agent   1.465*   ubuntu-jammy   xxxxxxxxxxxxxxxxxx  ami-12345678910

The failing stemcell in that case would be bosh-aws-xen-hvm-ubuntu-jammy-go_agent   1.465* 

Download the stemcell from support.broadcom.com portal and upload it again with "bosh upload-stemcell" with "--fix" flag. E.g 

 

bosh upload-stemcell /path/to/bosh-aws-xen-hvm-ubuntu-jammy-go_agent_1.465 --fix.

After this, the stemcell should appear with a new AMI and bosh-health-check VM  creation should succeed.