Create stemcell fails with "execution expired"

Products

• Operations Manager

Issue/Introduction

When uploading stemcell and it fails with error message 

Uploading stemcell 'bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.265'... Failed (00:01:30)
Cleaning up rendered CPI jobs... Finished (00:00:00)

creating stemcell (bosh-vsphere-esxi-ubuntu-xenial-go_agent 621.265):
  CPI 'create_stemcell' method responded with error: CmdError{"type":"Unknown","message":"execution expired","ok_to_retry":false}

The stemcell upload will try to connect to the vcenter/esxi host and then start the deploy command but it will not progress above 0%

 Deploy OVF template sc-06668fb0-e028-44f3-b9a2-2dd0ab3b9b2d 

To understand where the failure occurs the first step would be to check if the connection between opsman vm and the vcenter works, using the following command:

nc -vz esxi-host 443

ubuntu@opsmgr-34-slot-34-tanzu-###-###:~$ nc -vz ###-10-#####.oc.vmware.com 443
Connection to ###-10-#####.oc.vmware.com 443 port [tcp/https] succeeded!

if the above succeeds next would be to try to see if openssl and wget commands will work. The openssl command is a simple command that will try to pull down the certificate information, same as wget command will try to pull down basic server information. See working examples below from OpsMan vm:

wget esxi-host 443 --no-check-certificate

ubuntu@opsmgr-34-slot-34-tanzu-###-####:~$ wget wdc-10-#####.oc.vmware.com 443 --no-check-certificate
--2022-10-05 14:15:50--  http://wdc-10-####.oc.vmware.com/
Resolving wdc-10-###.oc.vmware.com (wdc-10-####.oc.vmware.com)... 10.151.183.72
Connecting to wdc-10-###.oc.vmware.com (wdc-10-##.oc.vmware.com)|10.151.183.72|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://wdc-10-###.oc.vmware.com/ [following]
--2022-10-05 14:15:50--  https://wdc-10-#####.oc.vmware.com/
Connecting to wdc-10-r12esx08.oc.vmware.com (wdc-10-####.oc.vmware.com)|10.151.183.72|:443... connected.
WARNING: cannot verify wdc-10-#####.oc.vmware.com's certificate, issued by ‘OU=VMware Engineering,O=wdc-10-#####.oc.vmware.com,ST=California,C=US,DC=local,DC=vsphere,CN=CA’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 258 [text/html]
Saving to: ‘index.html’

index.html                                        100%[=============================================================================================================>]     258  --.-KB/s    in 0s      

2022-10-05 14:15:50 (33.7 MB/s) - ‘index.html’ saved [258/258]

--2022-10-05 14:15:50--  http://443/
Resolving 443 (443)... 0.0.1.187
Connecting to 443 (443)|0.0.1.187|:80... failed: Invalid argument.
FINISHED --2022-10-05 14:15:50--
Total wall clock time: 0.04s
Downloaded: 1 files, 258 in 0s (33.7 MB/s)

 

openssl s_client -showcerts -connect esxi-host:443

ubuntu@opsmgr-34-slot-34-####-#####-###:~$ openssl s_client -showcerts -connect wdc-10-######.oc.vmware.com:443
CONNECTED(00000003)
depth=0 C = US, ST = California, L = Palo Alto, O = VMware, OU = VMware Engineering, CN = wdc-10-######.oc.vmware.com, emailAddress = ####@vmware.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = California, L = Palo Alto, O = VMware, OU = VMware Engineering, CN = wdc-10-r12esx08.oc.vmware.com, emailAddress = ####@vmware.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Palo Alto/O=VMware/OU=VMware Engineering/CN=wdc-10-r12esx08.oc.vmware.com/emailAddress=#####@vmware.com
   i:/CN=CA/DC=vsphere/DC=local/C=US/ST=California/O=wdc-10-vc34.##.####.com/OU=VMware Engineering
-----BEGIN CERTIFICATE-----
MIIEazCCA1OgAwIBAgIJAMLDc5REvl1NMA0GCSqGSIb3DQEBCwUAMIGgMQswCQYD
....
PsCMDghlaNjxr2LyWaf7GsAjwz0c42LpeCl9IHA1aQ==
-----END CERTIFICATE-----
---

Environment

Product Version: 2.10

Resolution

If there is a problem with exchanging the information, the command wont receive any certificate info for the openssl command, and wget wont be able to download the page. Test the same commands from the jumpbox.
If the same commands work from the jumpbox but not from the operations manager vm, this would indicate that there is a network/firewall issue preventing the data exchange between the opsman and vcenter.