Stemcell upload issue with CPI error "execution expired"

Products

• Operations Manager

Issue/Introduction

This article explains how to troubleshoot a stemcell upload problem when the Apply changes display the following error:
 

Uploading stemcell 'bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.330'... Failed (00:01:18)
Cleaning up rendered CPI jobs... Finished (00:00:00)


creating stemcell (bosh-vsphere-esxi-ubuntu-xenial-go_agent 621.330):
  CPI 'create_stemcell' method responded with error: CmdError{"type":"Unknown","message":"execution expired","ok_to_retry":false}
Exit code 1

Test 1) Navigate to the Operations Manager UI  to Bosh director tile > Settings > vCenter Config to grab the vCenter host URL. Using this URL you can perform a network connectivity test:
 

#SSH into OPS man vm and test connectivity from this location 
nc -zv <vcenter url> 443

#example:

nc -vz  exsi.domain.com 443

Test 2) If the test above succeeds we may need to find out the ESXi host experiencing the problem, this can be done by manually attempting to upload the stemcell and collect the bosh task data.
 

#SSH into OPS man vm.
#Switch to root 
sudo -i

#Navigate to stemcell directory
cd /var/tempest/stemcells

#Elect the stemcell you need to upload and use bosh  upload-stemcell <stemcell> --fix
bosh upload-stemcell bosh-stemcell-1.18-vsphere-esxi-ubuntu-jammy-go_agent.tgz --fix

#The above should fail and give us the bosh task number we can get the data from.
bosh task <task number> --cpi

#example:
bosh task 1234 --cpi

#In this log snippet it will display the host URL the stemcell is being uploaded to:
INFO -- : Uploading file to https://<EXSi FQDN>/nfc/526e50c9-429f-4348-0959-2b01435f9b36/disk-0.vmdk


#Test network connectivity to this URL:
nc -zv <EXSi host fqdn> 443 

Environment

Product Version: 2.10

Resolution

The Stemcells are uploaded directly from the file's location in the OPS manager vm directly to the ESXi host, therefore if the connection to the Host fails the stemcell uploading process will not succeed. Once you have identified the ESXi as having the connectivity issue you need to engage your infrastructure team and investigate the connection issues.