vCLS VMs fail to deploy

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

vCLS VMs fail to deploy.

The vCLS VM deployment task reaches 18%, then the VMs are deleted.

In the vCenter UI, the failure error shows "The task was cancelled by a user" and "The object 'vim.VirtualMachine:vm-xxxx' has already been deleted or has not been completely created."

In /var/log/vmware/eam/eam.log on vCenter:

YYYY-MM-DDTHH:mm:ss |  INFO | cluster-agent-4 | DeployVmJob.java | 554 | Zero-config deployemnt attempt failed(EamIOException->EamAppException[Upload request failed. Response:HTTP/1.1 503 Service Unavailable [content-length: 95, content-type: text/plain, date: <date and time>, server: envoy, connection: close] [Content-Length: 95,Chunked: false]]), vSAN=false . Trying next

 

In /var/log/vmware/envoy/envoy-access.log on vCenter:
YYYY-MM-DDTHH:mm:ss info envoy[xxxxxxxxxxxxxxxx] [Originator@6876 sub=Default] YYYY-MM-DDTHH:mm:ss POST /external-tp/http1/ESXi_FQDN/443/xxxxxxxxxxxxxxxxxxxxxxxx/nfc/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx88d6/disk-0.vmdk HTTP/1.1 503 upstream_reset_before_response_started{connection_termination} UC 6340608 95 49598 - - 127.0.0.1:45418 127.0.0.1:1080 IP_ADDRESS:59410 IP_ADDRESS:443

 

In /var/run/log/rhttpproxy.log on ESXi:

YYYY-MM-DDTHH:mm:ss warning rhttpproxy[2100938] [Originator@6876 sub=Proxy Req 06293] Server localhost:12000 (/nfc) disconnected with a response pending

 

In /var/run/log/vpxa.log on ESXi:

YYYY-MM-DDTHH:mm:ss warning vpxa[2100468] [Originator@6876 sub=Default] Timed out reading HTTP request. : Read timeout after approximately 50000ms. Closing stream <io_obj p:0x000000xxxxxxxxxx, h:25, <TCP '127.0.0.1 : 12000'>, <TCP '127.0.0.1 : 24430'>>
YYYY-MM-DDTHH:mm:ss warning vpxa[2099684] [Originator@6876 sub=IO.Connection opID=SWI-xxxxxxx] Failed to read buffer from stream; <io_obj p:0x000000xxxxxxxxxx, h:-1, <TCP '127.0.0.1 : 12000'>, <TCP '127.0.0.1 : 24430'>> e: 125(Operation canceled), async: false, duration: 49439msec
YYYY-MM-DDTHH:mm:ss error vpxa[2099684] [Originator@6876 sub=HttpNfcServer opID=SWI-xxxxxxx] [DiskUploadWorker] Error reading input: N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000xxxxxxxxxx, h:-1, <TCP '127.0.0.1 : 12000'>, <TCP '127.0.0.1 : 24430'>>, duration: 00:00:49.439239 (hh:mm:ss.us))

Environment

vCenter 7.0

vCenter 8.0

ESXi 7.0

ESXi 8.0

Cause

This can happen when a firewall, an Intrusion Detection System (IDS), an Intrusion Prevention System (IPS), or a network configuration/performance issue causes ESXi network traffic to be slowed down or blocked. This results in the VM deployment task being cancelled, as the vpxa timeout is reached.

Resolution

The resolution to this would be to address the cause of the slow traffic. This would need to be addressed within the firewall/IDS/IPS system which is slowing network traffic on the ESXi hosts.

As a workaround, it is possible to increase the timeout on vpxa. See Cloud Director OVF upload fails with error Broken pipe for the steps.