Virtual machine creation using bosh failing with error message "Host esxioomz is not a member of VDS dvSwitchTest"

Products

• Operations Manager
• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Application Service

Issue/Introduction

Virtual machine creation by bosh-health-check fails intermittently with the error below

{"time":1747733943,"stage":"Creating missing vms","tags":[],"total":1,"task":"bosh-health-check/c51f841c-1d3d-1f55-acbe-ddd43cfc287a (0)","index":1,"state":"started","progress":0}
{"time":1747734619,"stage":"Creating missing vms","tags":[],"total":1,"task":"bosh-health-check/c51f841c-1d3d-1f55-acbe-ddd43cfc287a (0)","index":1,"state":"failed","progress":100,"data":{"error":"Timed out pinging VM ''bosh-health-check_bosh-health_cz111w8ww4ww'' with agent ''4e2bb9d0-5e25-492b-8961-124018c6bb2a'' after 600 seconds"}}
{"time":1747734619,"error":{"code":450002,"message":"Timed out pinging VM ''bosh-health-check_bosh-health_cz111w8ww4ww'' with agent ''4e2bb9d0-5e25-492b-8961-124018c6bb2a'' after 600 seconds"}}
', "result_output" = '' WHERE ("id" = 61167685)

Environment

Tanzu Operations Manager 3.x

Cause

To debug this issue review of the bosh task debug and cpi logs is useful. On reviewing the debug and cpi logs it was observed that the cloning operation for the VMs was failing intermittently as the VDS was not realized on all the ESXi hosts. Below is the log snippet from the cpi and debug logs that shows the clone failure

61167685.txt:W, [2025-05-20T09:39:57.954595 #793986]  WARN -- [req_id cpi-280290]: Error running task 'VirtualMachine.clone'. Failed with message 'A specified parameter was not correct: spec.deviceChange.device.port.switchUuid' and fault message 'Host esxioomz is not a member of VDS dvSwitchTest'.

cpi-61167685.log:W, [2025-05-20T09:39:57.954595 #793986]  WARN -- [req_id cpi-280290]: Error running task 'VirtualMachine.clone'. Failed with message 'A specified parameter was not correct: spec.deviceChange.device.port.switchUuid' and fault message 'Host esxioomz is not a member of VDS dvSwitchTest'.

Resolution

The issue is with the underlying vSphere infrastructure and primarily caused due to missing switches on the ESXi host. To resolve this issue the VDS switch reported in the bosh CPI or debug logs must be present on all ESXi hosts. Once the switch is accessible and realized on all ESXi hosts VM creation will succeed.