"Error loading Data" error appeared when editing VM settings, caused by out-of-sync VDS configuration between vCenter Server and ESXi

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• "Error loading Data" error appeared after pending for 2 minutes when editing VM settings
  
  
  
  
• In an enhanced-linked mode environment, this issue may be encountered by all virtual machines in the inventory.
  
  
• Click "Error Details" to expand the error message, the vCenter Server FQDN or IP will be displayed, which is masked in the capture below. This FQDN or IP will be used for further troubleshooting:
  
  
  
  
  
• Login the vCenter Server in the error message and check /var/log/vmware/vpxd/vpxd.log, there are two types of the error:
  
  • DvsApplyOperationFault error as below:
    
    YYY-MM-DDTHH:MM:SS info vpxd[#####] [Originator@6876 sub=vmomi.soapStub[6067]] SOAP request returned HTTP failure; <<io_obj p:0x00007f5e4cf633f8, h:323, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-<host MoID>/vpxa, method: applyDVPort; code: 500 (Internal Server Error); fault: (vim.fault.DvsApplyOperationFault) { 
-->     faultCause = (vmodl.MethodFault) null, 
-->     faultMessage = <unset>,
-->     objectFault = (vim.fault.DvsApplyOperationFault.FaultOnObject) [ 
-->         (vim.fault.DvsApplyOperationFault.FaultOnObject) { 
-->                 objectId = "xxxxxxxx", 
-->                 type = "vim.dvs.DistributedVirtualPort", 
-->                 fault = (vim.fault.PlatformConfigFault) { 
-->                     faultCause = (vmodl.MethodFault) null, 
-->                     faultMessage = <unset>, 
-->                     text = "Failed to update port persistence location" 
-->                     msg = "An error occurred during host configuration: Failed to update port persistence location." 
-->              } 
-->         } 
-->     ] 
-->     msg = "Received SOAP response fault from [<<io_obj p:0x00007f5e4cf633f8, h:323, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-<host MoID>/vpxa]: applyDVPort 
-->     Received SOAP response fault from [<<io_obj p:0x000000704367fb20, h:21, <TCP '127.0.0.1 : 60265'>, <TCP '127.0.0.1 : 8307'>>, /sdk]: applyDVPort 
--> "
--> }


Based on host MoID in vpxd log, check /var/log/hostd.log on the ESXi host, you can see following error:



    YYYY-MM-DDT:HH:MM:SS info hostd[#######] [Originator@6876 sub=Solo.Vmomi opID=########-#### user=vpxuser] Result:
--> (vim.fault.DvsApplyOperationFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    objectFault = (vim.fault.DvsApplyOperationFault.FaultOnObject) [
-->       (vim.fault.DvsApplyOperationFault.FaultOnObject) {
-->          objectId = "xxxx",
-->          type = "vim.dvs.DistributedVirtualPort",
-->          fault = (vim.fault.PlatformConfigFault) {
-->             faultCause = (vmodl.MethodFault) null,
-->             faultMessage = <unset>,
-->             text = "Failed to update port persistence location"

YYYY-MM-DDTHH:MM:SS.XXXX warning hostd[#######] [Originator@6876 sub=Hostsvc.NetworkProvider opID=########-#### user=vpxuser] Skip saving dvport <DVS_NAME>-####  to /vmfs/volumes/vsan:###############-###########/########-####-####-####-#############/.dvsData/## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##/####: failed to create dir



  • 500 (Internal Server Error) when calling "deletePorts" method

YYY-MM-DDTHH:MM:SS info vpxd[2760669] [Originator@6876 sub=vmomi.soapStub[18070]] SOAP request returned HTTP failure; <<io_obj p:0x00007f5e488eac08, h:233, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-<host MoID>/vpxa, method: deletePorts; code: 500 (Internal Server Error); fault: (vmodl.fault.SystemError) { 
-->     faultCause = (vmodl.MethodFault) null, 
-->     faultMessage = <unset>, 
-->     reason = "" 
-->     msg = "Received SOAP response fault from [<<io_obj p:0x00007f5e488eac08, h:233, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-<host MoID>/vpxa]: deletePorts 
-->     Received SOAP response fault from [<<io_obj p:0x000000ab4ff94e00, h:24, <TCP '127.0.0.1 : 34432'>, <TCP '127.0.0.1 : 8307'>>, /sdk]: deletePorts 
--> A general system error occurred: "" 
--> }


    Based on host MoID in vpxd log, check /var/log/hostd.log on the ESXi host, you can see following error:
    
    YYY-MM-DDTHH:MM:SS Wa(164) Hostd[#######]: [Originator@6876 sub=Hostsvc.NetworkProvider opID=########-#### sid=######## user=vpxuser:<no user>] Error deleting dvport ## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##-#### : Unable to delete DVPort "" that is in use, use list: vmkX

     

Environment

vCenter Server 8.0

ESXi 8.0

Cause

• DvsApplyOperationFault error:
  • Hosts need a datastore to write in it .dvsdata and configurations from DVS.
  • If the datastore is vSAN then you may meet this issue. 
    
    
• 500 (Internal Server Error) when calling "deletePorts" method
  • This error was caused by out-of-sync VDS configuration between vCenter Server and ESXi host

Resolution

• DvsApplyOperationFault error:
  • Apply resolution in DVS out of Sync for a host. Cannot create directory .dvsdata when using vSAN, which need to open a support case with Broadcom Support  
    
    
    
• 500 (Internal Server Error) when calling "deletePorts" method:
  • Restart vCenter Server can fix this issue.
  • In an enhanced-linked mode environment, only the vCenter Server whose FQDN/IP in the error message need to restart.