SVMotion progress gets stuck at 23% and then times out with error: Cannot connect Host
search cancel

SVMotion progress gets stuck at 23% and then times out with error: Cannot connect Host

book

Article ID: 316409

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Perform Storage vMotion from one datastore in a cluster to another datastore in a different cluster within the same vCenter



Symptoms:

  • SVMotion Migration task times out with Error: Cannot Connect Host.
  • Observation : The VMs that are failing to SVMotion have snapshots associated to them. the VMs with no snapshots were able to migrate.



Environment

VMware vSphere 6.x

Cause

vpxa.log
 "Failed to connect to server" i.e. destination server

YYYY-MM-DDT0HH:MM:SS warning vpxa[######] [Originator@6876 sub=Libs opID=k1kc4bzp-ac2q-h5:#######] [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server ##.##.##.##:902
YYYY-MM-DDT0HH:MM:SS error vpxa[######] [Originator@6876 sub=NfcManager opID=k1kc4bzp-ac2q-h5:#######] [NfcClient] Unable to connect to NFC server: Failed to connect to server ##.##.##.##:902
YYYY-MM-DDT0HH:MM:SS error vpxa[######] [Originator@6876 sub=NfcManager opID=k1kc4bzp-ac2q-h5:#######] [NfcWorker] Error encountered while opening clients for copy spec:
--> N3Vim5Fault16HostConnectFault9ExceptionE(vim.fault.HostConnectFault)
YYYY-MM-DDT0HH:MM:SS error vpxa[######] [Originator@6876 sub=NfcManager opID=k1kc4bzp-ac2q-h5:#######] [NfcManagerImpl] Copy operation failed with error: vim.fault.HostConnectFault
YYYY-MM-DDT0HH:MM:SS verbose vpxa[######] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:#######] RecordOp ASSIGN: info.state, task-###. Applied change to temp map.
YYYY-MM-DDT0HH:MM:SS verbose vpxa[######] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:#######] RecordOp ASSIGN: info.cancelable, task-###. Applied change to temp map.
YYYY-MM-DDT0HH:MM:SS verbose vpxa[######] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:#######] RecordOp ASSIGN: info.error, task-###. Applied change to temp map.
YYYY-MM-DDT0HH:MM:SS verbose vpxa[######] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:#######] [CommitChangesAndNotify] Updating cached values
YYYY-MM-DDT0HH:MM:SS info vpxa[######] [Originator@6876 sub=vpxLro opID=k1kc4bzp-ac2q-h5:#######] [VpxLRO] -- FINISH task-###
YYYY-MM-DDT0HH:MM:SS info vpxa[######] [Originator@6876 sub=Default opID=k1kc4bzp-ac2q-h5:#######] [VpxLRO] -- ERROR task-###-- nfcManager -- nfc.NfcManager.copy: vim.fault.HostConnectFault:

nfcd.log:
Noted the following events in nfcd.log in continuous manner

2019-10-16T08:18:57Z nfcd[34831]: Failed to switch netstack to the vSphereProvisioning stack. Using default management network stack.

Resolution

  • Enable vSphere provisioning on vmkernel ports used for vMotion on both source and destination server.
  • Go to Host > Configure > VMkernel Adapters under Networking > Select the vmkernel port used for vMotion > Edit



Additional Information

  • Please refer to Vmware Document page for vSphere Networking section: 

         https://techdocs.broadcom.com/content/dam/broadcom/techdocs/us/en/pdf/vmware/vsphere/vsphere/legacy/vmware-vsphere-6-7.pdf


Supports the traffic for virtual machine cold migration, cloning, and snapshot migration. You can use the provisioning TCP/IP to handle Network File Copy (NFC) traffic during long-distance vMotion. NFC provides a file-specific FTP service for vSphere. ESXi uses NFC for copying and moving data between datastores. VMkernel adapters configured with the provisioning TCP/IP stack handle the traffic from cloning the virtual disks of the migrated virtual machines in long-distance vMotion. By using the provisioning TCP/IP stack, you can isolate the traffic from the cloning operations on a separate gateway. After you configure a VMkernel adapter with the provisioning TCP/IP stack, all adapters on the default TCP/IP stack are disabled for the Provisioning traffic.

https://techdocs.broadcom.com/content/dam/broadcom/techdocs/us/en/pdf/vmware/vsphere/vsphere/legacy/vmware-vsphere-6-7.pdf


Use the provisioning TCP/IP stack to isolate traffic for cold migration, VM clones, and snapshots, and to assign a dedicated default gateway, routing table, and DNS configuration for this traffic. To enable the Provisioning TCP/IP stack, assign it a new VMkernel adapter.
By using a separate TCP/IP stack, you can handle vMotion and cold migration traffic according to the topology of the network and as required for your organization:
  • Route the traffic for migration of powered on or powered off virtual machines by using a default gateway that is different from the gateway assigned to the default stack on the host.
  • By using a separate default gateway, you can use DHCP for IP address assignment to the VMkernel adapters for migration in a flexible way.
  • Assign a separate set of buffers and sockets.
  • Avoid routing table conflicts that might otherwise appear when many features are using a common TCP/IP stack.
  • Isolate traffic to improve security.



Impact/Risks:
NA

Powered by Wolken Software