vCenter HA Basic Mode fails during Passive/Witness cloning when using NFSv3 Datastore - vCenter Server
search cancel

vCenter HA Basic Mode fails during Passive/Witness cloning when using NFSv3 Datastore - vCenter Server

book

Article ID: 343081

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

When configuring vCenter High Availability (VCHA) in Basic mode, the process fails or stalls at 96% or 33% if the vCenter Server Appliance is stored on an NFSv3 datastore.

  • The ESXi host where the cloning occurs shows a "Not Responding" or "Disconnected" state in vCenter.

  • During repeated attempts to configure VCHA using the automatic cloning method, the process may fail at different stages with errors such as:
    • "An error occurred while communicating with the remote host"
    • "The session is not authenticated. You do not hold privileges "PropertyCollector.te#t session[########-#####-####-####-############]#######-####-####-#####-########"

  • From vCenter /var/log/vmware/vpxd/vpxd.log
    • YYYY-MM-DDTHH:MM:SS warning vpxd[############] [Originator@6876 sub=pbm opID=ProvisioningWizard-addMulti-24848-ngc:########-##-##] PBMCallback: ShouldSkipPostCloneCommonCallback: post clone callback is skipped - VM clone failed
      YYYY-MM-DDTHH:MM:SS error vpxd[############] [Originator@6876 sub=VmProv opID=FlowBasedWizard-apply-###-ngc-##-f-##] Get exception while executing action vpx.vmprov.RemoveSnapshot: N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)

  • From the ESXi /var/run/log/vmkernel.log
    • YYYY-MM-DDTHH:MM:SS Vmxnet3: 17293: Disable Rx queuing; queue size 256 is larger than Vmxnet3RxQueueLimit limit of 64.
      YYYY-MM-DDTHH:MM:SS cpu20:442393)Vmxnet3: 17651: Using default queue delivery for vmxnet3 for port 0x600002c
      YYYY-MM-DDTHH:MM:SS cpu14:68341 opID=########)VMKAPIMOD: 86: Failed to  check if port is Uplink : Failure

  • Error in vCenter /var/log/vmware/vpxd/vpxd.log

    • YYYY-MM-DDTHH:MM:SS warning vpxd[############] [Originator@6876 sub=VpxProfiler opID=HB-host-#####@#####-########] DoHostSync:host-##### [GetChangesTime] took 1629260 ms

      YYYY-MM-DDTHH:MM:SS warning vpxd[############ [Originator@6876 sub=pbm opID=ProvisioningWizard-addMulti-24848-ngc:########-##-##] PBMCallback: ShouldSkipPostCloneCommonCallback: post clone callback is skipped - VM clone failed

      YYYY-MM-DDTHH:MM:SS error vpxd[############] [Originator@6876 sub=VmProv opID=ProvisioningWizard-addMulti-24848-ngc:########-##-##] Unable to do delete snapshot clone-temp-######## because the host host.example.com is disconnected
      info vpxd[############] [Originator@6876 sub=VmProv opID=ProvisioningWizard-addMulti-24848-ngc:########-##-##] Done undo action vpx.vmprov.CreateSnapshot with output:

      YYYY-MM-DDTHH:MM:SS warning vpxd[############] [Originator@6876 sub=VpxProfiler opID=ProvisioningWizard-addMulti-24848-ngc:########-##-##] VpxLro::LroMain [TotalTime] took 9647077 ms
      error vpxd[############] [Originator@6876 sub=vpxLro opID=ProvisioningWizard-addMulti-24848-ngc:########-##-##] [VpxLRO] Unexpected Exception: N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)

      YYYY-MM-DDTHH:MM:SS info vpxd[############] [Originator@6876 sub=Default opID=ProvisioningWizard-addMulti-24848-ngc:########-##] [VpxLRO] -- ERROR t
      ask-1416563 -- vm-##### -- vim.VirtualMachine.clone: vmodl.fault.HostCommunication:
      --> Result:
      --> (vmodl.fault.HostCommunication) {
      -->    faultCause = (vmodl.MethodFault) null,
      -->    faultMessage =
      -->    msg = ""
      --> }
      --> Args:
      -->
      --> Arg folder:
      --> 'vim.Folder:6CA8####-###-###-####-C01D####:group-####'
      --> Arg name:
      --> "TestVC001"
      --> Arg spec:
      --> (vim.vm.CloneSpec) {
      -->    location = (vim.vm.RelocateSpec) {
      -->       service = (vim.ServiceLocator) null,
      -->       folder = 'vim.Folder:6CA8####-###-###-####-C01D####:group-####',
      -->       datastore = 'vim.Datastore:6CA8####-###-###-####-C01D####:group-####':datastore-#####',
      -->       diskMoveType = 

  • Error in ESXi /var/run/log/vmkernel.log

    • cpu20:442393)VSCSI: 273: handle 46587(vscsi0:12):Input values: res=0 limit=-2 bw=-1 Shares=1000
      cpu20:442393)Vmxnet3: 17293: Disable Rx queuing; queue size 256 is larger than Vmxnet3RxQueueLimit limit of 64.
      cpu20:442393)Vmxnet3: 17651: Using default queue delivery for vmxnet3 for port 0x600002c
      cpu20:442393)NetPort: 3208: resuming traffic on DV port 251
      cpu20:442393)Team.etherswitch: TeamESPolicySet:5942: Port 0x600002c frp numUplinks 2 active 2(max 2) standby 0
      cpu20:442393)Team.etherswitch: TeamESPolicySet:5950: Update: Port 0x600002c frp numUplinks 2 active 2(max 2) standby 0
      cpu20:442393)NetPort: 1662: enabled port 0x600002c with mac ##:##:##:##:##
      cpu20:442393)Vmxnet3: 17293: Disable Rx queuing; queue size 256 is larger than Vmxnet3RxQueueLimit limit of 64.
      cpu20:442393)Vmxnet3: 17651: Using default queue delivery for vmxnet3 for port 0x2000007
      cpu20:442393)NetPort: 1662: enabled port 0x2000007 with mac ##:##:##:##:##
      cpu14:68341 opID=#####)World: 12235: VC opID HB-host-#####@#####-########-DvsHandleHostReconnect-1fc1dfe0-3-1dd4 maps to vmkernel opID #####
      cpu14:68341 opID=#####)VMKAPIMOD: 86: Failed to  check if port is Uplink : Failure
      cpu14:68341 opID=#####)Team.etherswitch: TeamESLACPLAGEventCB:6277: Received a LAG DESTROY event version :0, lagId :0, lagLinkStatus :NOT USED,lagName :, uplinkName :, portLinkStatus :NOT USED, portID :0x0
      cpu14:68341 opID=#####)netioc: NetIOCSetRespoolVersion:245: Set netioc version for portset: DvsPortset-0 to 2,old threshold: 2
      cpu14:68341 opID=#####)netioc: NetIOCPortsetNetSchedStatusSet:1207: Set sched status for portset: DvsPortset-0 to Inactive, old:Inactive

Environment

  • vSphere ESXi 7.x
  • vSphere ESXi 8.x
  • vSphere ESXi 9.x
  • VMware vCenter Server 7.x
  • VMware vCenter Server 8.x
  • VMware vCenter Server 9.x

Cause

This issue occurs due to a limitation in the NFSv3 locking mechanism, which includes a 40-second lock timeout. The automatic clone operations used by the VCHA configuration process may exceed this timeout, resulting in the failure.

Resolution

Option 1: Use the Same ESXi Host for Clone Operations

  1. Ensure the source (Active) and target ESXi hosts for the Passive and Witness nodes are the same host during VCHA deployment.

  2. To avoid DRS anti-affinity rule errors when deploying all VCHA nodes on the same host:
    • Navigate to vCenter Advanced Settings.

    • Set the following key: config.vpxd.vcha.drsAntiAffinity = False
  3. After the configuration is complete, you may move the nodes to different ESXi hosts and revert the advanced setting to True.

Option 2: Use Manual Clone Method

  1. Configure VCHA using the Manual Clone method. This allows for more granular control over node placement and avoids datastore locking issues associated with NFSv3.

  2. Ensure that you are using NFS 4.1 configurations, which feature improved session and file locking semantics.

Additional Information

Cloning a Passive or Witness VM for a VCHA deployment to the same NFS 3.1 datastore as the source Active node VM fails. You must use NFS4 or clone the Passive and Witness VMs to a datastore different from the Active VM

Refer:
vCenter HA Clone Operation Fails During Deployment

NFS file locking mechanism

Powered by Wolken Software