vpxd service crashes when cloning a VM using API with Guest Customization

Products

• VMware vCenter Server

Issue/Introduction

• vpxd service crashes when cloning a VM using API with Guest Customization
• /var/log/vmware/vpxd/vpxd.log reports the below VM clone task

  info vpxd[25###07] [Originator@6##6 sub=OMM opID=HB-host-2##6@4###0-71###e-WorkQueue-59####8] Downloading VMX config for VM [vim.VirtualMachine:vm-####,WindowsServer-2022] on host [vim.HostSystem:host-2##6,esx01.example.com]
  
  warning vpxd[25###07] [Originator@6##6 sub=OMM opID=HB-host-2##6@4###0-71###e-WorkQueue-59####8] Cannot download vm config of [vim.VirtualMachine:vm-####,WindowsServer-2022]: Received SOAP response fault from [<<io_obj p:0x#######18, h:117, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-2##6/vpxa>]: downloadVMXConfig
  
  info vpxd[2566179] [Originator@6##6 sub=vpxLro opID=5#####7] [VpxLRO] -- BEGIN task-1212007 -- vm-#### -- vim.VirtualMachine.clone -- 520c####-####-####-####-########c683(529f####-####-####-####-########ba6c)
  
  info vpxd[2566179] [Originator@6##6 sub=vpxLro opID=5#####7-01] [VpxLRO] -- BEGIN lro-14578 -- vm-#### -- vim.VirtualMachine.clone -- 520c####-####-####-####-########c683(529f####-####-####-####-########ba6c)
  
  info vpxd[2566179] [Originator@6##6 sub=pbm opID=5#####7-01] Setting if empty profile passed for vm vm-#### to true
  
  info vpxd[2566179] [Originator@6##6 sub=pbm opID=5#####7-01] Setting if empty profile passed for disk vm-####:2000 to true
  
  warning vpxd[2566179] [Originator@6##6 sub=pbm opID=5#####7-01] Error querying associated capabilities for entity vm-####:2000
  -->       vm = 'vim.VirtualMachine:c4b7####-####-####-####-########e0de:vm-####',
  -->                value = "c4b7####-####-####-####-########e0de:vm-####:fgS####-####-####-####-########Ks="
  -->                      value = "c4b7####-####-####-####-########e0de:vm-####:fgS####-####-####-####-########Ks="
  
  error vpxd[2566179] [Originator@6##6 sub=VmProv opID=5#####7-01] Local-VC Full Clone failed at vpx.vmprov.SelectDestination for poweredOff VM 'WindowsServer-2022' (vm-####, ds:///vmfs/volumes/6685####-####-####-####-########3eaa/WindowsServer-2022_1/WindowsServer-2022.vmtx) to new name sample-test on host-2##6 (10.##.##.##) in pool resgroup-2##3 with ds ds:///vmfs/volumes/6685####-####-####-####-########3eaa/ to (DstLocation null) with fault vmodl.fault.InvalidArgument:
  

   

• API call which is used to clone a VM

  POST https://samplevc.example.com/sdk/vim25/8.0.3.0/VirtualMachine/vm-####/CloneVM_Task
  
  Content-Type: application/json
  vmware-api-session-id: 12#######################4e
  {
  "_typeName": "CloneVM_RequestType",
  "name": "sample-test",
  "folder": {
    "_typeName": "ManagedObjectReference",
    "type": "Folder",
    "value": "group-v1####"
  },
  "spec": {
    "_typeName": "VirtualMachineCloneSpec",
    "location": {
     "_typeName": "VirtualMachineRelocateSpec",
     "datastore": {
      "_typeName": "ManagedObjectReference",
      "type": "Datastore",
      "value": "datastore-3###"
     },
     "pool": {
      "_typeName": "ManagedObjectReference",
      "type": "ResourcePool",
      "value": "resgroup-3###"
     }
    },
    "template": false,
    "customization": {
     "_typeName": "CustomizationSpec",
     "identity": {
      "_typeName": "CustomizationSysprep",
      "guiUnattended": {
       "_typeName": "CustomizationGuiUnattended",
       "autoLogon": true,
       "autoLogonCount": 1,
       "password": {
        "_typeName": "CustomizationPassword",
        "value": "#######",
        "plainText": true
       },
       "timeZone": 35
      },
      "userData": {
       "_typeName": "CustomizationUserData",
       "fullName": "Administrator",
       "orgName": "Example",
       "computerName": {
        "_typeName": "CustomizationVirtualMachineName"
       },
       "productId": "ignored"
      },
      "guiRunOnce": {
       "_typeName": "CustomizationGuiRunOnce",
       "commandList": [
        "echo \"Hello there\" > C:\\test.txt"
       ]
      }
     },
     "globalIPSettings": {
      "_typeName": "CustomizationGlobalIPSettings",
      "dnsSuffixList": [
       "sampledns.example.com",
       "example.com"
      ]
     },
     "nicSettingMap": [
      {
       "_typeName": "CustomizationAdapterMapping",
       "adapter": {
        "_typeName": "CustomizationIPSettings",
        "ip": {
         "_typeName": "CustomizationDhcpIpGenerator"
        }
       }
      }
     ]
    },
    "config": {
     "_typeName": "VirtualMachineConfigSpec",
     "numCPUs": 4,
     "memoryMB": 16384,
     "annotation": "Example VM for sample-test"
    },
    "powerOn": true
  }
  }
  
  
  Guest customization specified is as below
  
  "customization": {
     "_typeName": "CustomizationSpec",
     "identity": {
      "_typeName": "CustomizationSysprep",
      "guiUnattended": {
       "_typeName": "CustomizationGuiUnattended",
       "autoLogon": true,
       "autoLogonCount": 1,
       "password": {
        "_typeName": "CustomizationPassword",
        "value": "#######",
        "plainText": true
       },
       "timeZone": 35
      },
      "userData": {
       "_typeName": "CustomizationUserData",
       "fullName": "Administrator",
       "orgName": "Example",
       "computerName": {
        "_typeName": "CustomizationVirtualMachineName"
       },
       "productId": "ignored"
      },
      "guiRunOnce": {
       "_typeName": "CustomizationGuiRunOnce",
       "commandList": [
        "echo \"Hello there\" > C:\\test.txt"
       ]
      }
     },
     "globalIPSettings": {
      "_typeName": "CustomizationGlobalIPSettings",
      "dnsSuffixList": [
       "sampledns.example.com",
       "example.com"
      ]
     },
     "nicSettingMap": [
      {
       "_typeName": "CustomizationAdapterMapping",
       "adapter": {
        "_typeName": "CustomizationIPSettings",
        "ip": {
         "_typeName": "CustomizationDhcpIpGenerator"
        }
       }
      }
     ]
    },

Environment

VMware vCenter Server 8.0.3

Cause

The CloneVM_Task API request with Guest Customization was missing the vim.vm.customization.Identification field under the identity section.
When the identification parameter is not specified (either Workgroup or Domain configuration), vpxd fails to handle the missing field correctly, leading to a crash.

Resolution

Issue is resolved in future vCenter release

 

To workaround the issue, include the identification parameter in the Guest Customization specification.

Example for Workgroup:

<identification>
  <_type>vim.vm.customization.Identification</_type>
  <joinWorkgroup>WORKGROUP</joinWorkgroup>
</identification>

Example for Domain:

<identification>
  <_type>vim.vm.customization.Identification</_type>
  <domainAdmin>administrator</domainAdmin>
  <domainAdminPassword>
    <_type>vim.vm.customization.Password</_type>
    <plainText>false</plainText>
    <value>********</value>
  </domainAdminPassword>
  <domainOU/>
  <joinDomain>example.com</joinDomain>
</identification>