HCX RAV/vMotion migrations are completed without establishing connectivity to a correct port group.

Products

• VMware HCX

Issue/Introduction

• After an HCX RAV/vMotion migration via a PowerCLI script, VMs lose network connectivity following switchover.
• Within the same migration wave, only certain VMs experienced network connectivity issues.
• The network settings of the disconnected VMs were listed as 'none'.
• For a failed VM, /vmfs/volumes/<datastore location>/<VM Name>/vmware.log it shows that the VM was booted with the network name set to None. 
  

  <timestamps> In(05) vmx - DICT ethernet0.allowGuestConnectionControl = "FALSE"
  <timestamps> In(05) vmx - DICT      ethernet0.virtualDev = "vmxnet3"
  <timestamps> In(05) vmx - DICT     ethernet0.networkName = "none"
  <timestamps> In(05) vmx - DICT     ethernet0.addressType = "vpx"
  <timestamps> In(05) vmx - DICT ethernet0.generatedAddress = "00:50:56:##:##:##"
  <timestamps> In(05) vmx - DICT ethernet0.uptCompatibility = "TRUE"
  <timestamps> In(05) vmx - DICT         ethernet0.present = "TRUE"
  ....
  <timestamps> In(05) vcpu-0 - VMXNET3 user: failed to connect 'Ethernet0' to vSwitch portgroup 'none'.
  <timestamps> In(05) vcpu-0 - VMXNET3 user: Ethernet0 Driver Info: version = 19932667 gosBits = 2 gosType = 2, gosVer = 25344, gosMisc = 212
  <timestamps> In(05) vcpu-0 - [msg.device.startdisconnected] Virtual device 'Ethernet0' will start disconnected.

• ESXi /var/run/log/hostd.log shows that failure is due to being unable to get the network none.

  <timestamps> Db(167) Hostd[12796883] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/<datastore location>/<VM Name>.vmx opID=#####-TxId: ########-####-####-####-############# sid=######## user=vpxuser:VSPHERE.LOCAL\Administrator] NIC: is present message: Device 'ethernet0' does not exist.
  <timestamps> Wa(164) Hostd[12796883] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/<datastore location>/<VM Name>.vmx opID=#####-TxId: ########-####-####-####-############# sid=######## user=vpxuser:VSPHERE.LOCAL\Administrator] Unable to get network object for network none

• ESXi  /var/run/log/vmkernel.log shows failed to connect to a nonexistent switch. 

  <timestamps> In(182) vmkernel: cpu2:########)World: vm ########: 6916: Starting world vmm0:<VM Name> of type 8
  <timestamps> In(182) vmkernel: cpu2:########)Migrate: 312: vmotion: Dest vmmLeaderID = ########, ts = ############, srcIP = <##.##.##.##> dstIP = <##.##.##.##> Dest wid = 0 using UNSHARED swap, unencrypted
  <timestamps> Wa(180) vmkwarning: cpu41:########)WARNING: Net: 2559: connect to vswitch none from world ######## failed: Not found

  The following compares the working and nonworking VMs - ESXi logs - /var/run/log/hostd.log. 
  • None working - no network information. 

    -->             backing = (vim.vm.device.VirtualEthernetCard.NetworkBackingInfo) {
    -->                deviceName = "none",
    -->                useAutoDetect = <unset>,
    -->                network = <unset>,
    -->                inPassthroughMode = <unset>

  • Working - network information.

    -->             backing = (vim.vm.device.VirtualEthernetCard.OpaqueNetworkBackingInfo) {
    -->                opaqueNetworkId = "########-####-####-####-#############",
    -->                opaqueNetworkType = "nsx.LogicalSwitch"

 

Environment

VMware HCX

Cause

• This occurred when using a custom power CLI script.
• VM was booted with the VM network set to none due to the configuration being passed incorrectly.  
• The same was not reproducible from the UI. 

Resolution

Follow the HCX - PowerCli operations and best practices, and validate the PowerCLI script.