vCLS VMs are rebooted every day when ESXi host has mutliple management vmkernel adapter

Products

• VMware vCenter Server

Issue/Introduction

- vCLS VMs are rebooted every day

- ESXi host where the vCLS vm is running has multiple vmkernel adapters tagged with 'Management'

- Sample log output:

/var/log/vmware/wcp/wcpsvc.log
---------------------------------
<timestamp> error wcp [vclib/datastore_upload.go:54] [opID=vCLS] Failed to upload file from (/tmp/hdcs2446645437/hdcs.cab) to dest path (vCLS-af181bb4-676c-4133-bedc-8a5c42936be1/hdcs.cab) : ServerFaultCode: A specified parameter was not correct: spec.url
<timestamp> error wcp [eamagent/password.go:79] [opID=vCLS] Copying CAB file to datastore failed
<timestamp> error wcp [eamagency/rotatepassword.go:135] [opID=vCLS] VM Customization for password failed to execute: ServerFaultCode: A specified parameter was not correct: spec.url
---------------------------------

/var/log/vmware/vpxd/vpxd.log
---------------------------------
<timestamp> error vpxd[18553] [Originator@6876 sub=MoSessionMgr opID=wcp-vCLS-1b] [SessionManagerMo::GetHostBySpecInt] url host not found by IP: http://<ip address>/folder/vCLS-af181bb4-676c-4133-bedc-8a5c42936be1/hdcs.cab?dsName=<datastore name>, <ip address>
<timestamp> warning vpxd[18553] [Originator@6876 sub=Vmomi opID=wcp-vCLS-1b] VMOMI activation LRO failed; <<52adaec5-95b9-28c0-dda5-e249fc7c505d, <TCP '<ip address> : 8085'>, <TCP '<ip address> : 45994'>>, SessionManager, vim.SessionManager.acquireGenericServiceTicket, <vim.version.unstable, official, uACFB792E>, {stm: {<io_obj t:N7Vmacore6System19TCPSocketObjectAsioE, h:74, <TCP '<ip address> : 8085'>, <TCP '<ip address> : 45994'>>, id: 177323, state(in/out): 3/1}, session: <52adaec5-95b9-28c0-dda5-e249fc7c505d, <TCP '<ip address> : 8085'>, <TCP '<ip address> : 45994'>>, req: {POST, /sdk}}>, N5Vmomi5Fault15InvalidArgument9ExceptionE(Fault cause: vmodl.fault.InvalidArgument
--> )
--> [context]zKq7AVECAQAAALnfcgEXdnB4ZAAAayZTbGlidm1hY29yZS5zbwAAtB9CABNEQ4Gp7icBdnB4ZACBJA0pAYFzDykBgQXMoQKBShXtAYGBSe0BgcJR7QGCzVw6AWxpYnZpbS10eXBlcy5zbwCBK6ugAgOTvhxsaWJ2bW9taS5zbwCB5Oh8AoGqLZ8CgfU9nwKBO1CeAoHYJ58CAPjrNwADRTgAjxlRBMSQCGxpYmMuc28uNgAEbJEQ[/context]
<timestamp> info vpxd[18553] [Originator@6876 sub=vpxLro opID=wcp-vCLS-1b] [VpxLRO] -- FINISH lro-211742
<timestamp> error vpxd[18553] [Originator@6876 sub=Default opID=wcp-vCLS-1b] [VpxLRO] -- ERROR lro-211742 -- 52adaec5-95b9-28c0-dda5-e249fc7c505d(52051f50-8bc8-a61d-077b-46076becc2ec) -- SessionManager -- vim.SessionManager.acquireGenericServiceTicket: :vmodl.fault.InvalidArgument
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    invalidProperty = "spec.url"
-->    msg = ""
--> }
--> Args:
-->
--> Arg spec:
--> (vim.SessionManager.HttpServiceRequestSpec) {
-->    method = "httpPut",
-->    url = "http://<ip address>/folder/vCLS-af181bb4-676c-4133-bedc-8a5c42936be1/hdcs.cab?dsName=<datastore name>"
--> }
---------------------------------

Environment

vCenter Server 7.x

vCenter Server 8.x

Cause

When ESXi host has multiple vnics, wcp tries to upload guest customization file via first found management vmkernel adapter.

Vpxd refers to an IP address that can be resolved from vCenter, so if that IP address is used by a second or later vmkernel adapter, file upload fails with a 'url host not found by IP' error.

Resolution

Currently, there is no resolution for this issue.

 

Note: There is no functional impact of this issue to any of the vCLS dependent services like DRS.

 

Workaround 1:

Use a single management vmkernel adapter.

Workaround 2:

Assign an IP address that can be resolved from vCenter to a first vmkernel adapter tagged with 'Management'.

e.g.) vmk0