Adding an ESX/ESXi host to Nexus 1000v vDS fails with the error: vDS operation failed on host , got (vmodl.fault.SystemError) exception

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Unable to add an ESX/ESXi host to Cisco Nexus 1000v VDS
• Adding an ESX/ESXi host to Cisco Nexus 1000v VDS fails
• Adding the ESX/ESXi host to Nexus 1000v VDS with incorrect version of VEM vib module, which is not as per the support matrix of Cisco Nexus 1000v upgrade guidance, fails
• You see the error
  
  Cannot complete a vNetwork Distributed Switch operation for one or more host members. vDS operation failed on host <hostname>, got (vmodl.fault.SystemError) exception.
  
• Even after installing the VEM manually on the ESX/ESXi host, the task fails with the same error message at 74% after invoking Update Manager.
• vCenter Server log file contains entries similar to:
  
  <TIMESTAMP> error 'App' opID=D40E5368-000003B1] vim.fault.DvsOperationBulkFault
  
  ------ In-memory logs end --------
  <TIMESTAMP> info 'App' opID=D40E5368-000003B1] [VpxLRO] -- FINISH task-123803 -- dvs-288 -- vim.DistributedVirtualSwitch.reconfigure -- 3B3A7374-####-####-####-########D82(AE8F22F7-####-####-####-########0CD)
  <TIMESTAMP> info 'App' opID=D40E5368-000003B1] [VpxLRO] -- ERROR task-123803 -- dvs-288 -- vim.DistributedVirtualSwitch.reconfigure: vim.fault.DvsOperationBulkFault:
  Result:
  (vim.fault.DvsOperationBulkFault) {
  dynamicType = <unset>,
  faultCause = (vmodl.MethodFault) null,
  faultMessage = (vmodl.LocalizableMessage) [
  (vmodl.LocalizableMessage) {
  dynamicType = <unset>,
  key = "com.vmware.vim.vpxd.dvs.dvsBulkHostFault.label",
  arg = (vmodl.KeyAnyValue) [
  (vmodl.KeyAnyValue) {
  dynamicType = <unset>,
  key = "fault",
  value = "vmodl.fault.SystemError",
  },
  (vmodl.KeyAnyValue) {
  dynamicType = <unset>,
  key = "msg",
  value = "",
  },
  (vmodl.KeyAnyValue) {
  dynamicType = <unset>,
  key = "host",
  value = "<HOSTNAME>",
  }
  ],
  message = <unset>,
  }
  ],
  hostFault = (vim.fault.DvsOperationBulkFault.FaultOnHost) [
  (vim.fault.DvsOperationBulkFault.FaultOnHost) {
  dynamicType = <unset>,
  host = 'vim.HostSystem:host-1013',
  fault = (vmodl.fault.SystemError) {
  dynamicType = <unset>,
  faultCause = (vmodl.MethodFault) null,
  reason = "vmodl.fault.SystemError",
  
  
• In vpxd log file, you see this message related to Update Manager:
  <TIMESTAMP>error 'App' opID=3ECDC342-0000019E] [MoDVSwitch::PushBundleToHostInt] failed to install dvs [NAME] host-component on host [host-1073]: got exception:
  vmodl.fault.SystemError : (vmodl.fault.SystemError) {
  dynamicType = <unset>,
  faultCause = (vmodl.MethodFault) null,
  reason = "VUM API stage failed: ",
  msg = "",
  }
  
  
• In the Update Manager logs, you see entries similar to:
  
  <TIMESTAMP>'httpDownload' 1436 INFO] [httpDownload, 556] Downloading http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-%3CVERSION%3E.vib
  <TIMESTAMP> 'httpDownload' 1436 INFO] [httpDownload, 791] Status code: 404
  <TIMESTAMP> 'httpDownload' 1436 ERROR] [httpDownload, 827] Error retrieving document 'http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib. Unhandled status code: 404
  <TIMESTAMP> 'httpDownload' 1436 ERROR] [httpDownload, 929] Connect to http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vibwith proxy off failed: Error retrieving document http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib. Unhandled status code: 404
  <TIMESTAMP> 'httpDownload' 1436 WARN] [httpDownload, 420] Download http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vibfailed, err: Error retrieving document 'http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib. Unhandled status code: 404

 

Cause

This issue occurs if the VIB requested is not a part of the current VSM.

This is not a VMware issue and may occur either due to a user error or problems with N1K components.

Resolution

To install the VEM module on the ESX/ESXi host:

1. Stop the VMware Update Manager service.
2. Install the VEM module manually on the ESX/ESXi host.
   
   For information on options for installing the VEM module manually, see:
   
   • ESX/ESXi 4.x
     
     • Installing Cisco Virtual Ethernet Module (VEM) on an ESX host using the command line (1013551)
       
       
   • ESXi 5.0
     
     • Installing async drivers on ESXi 5.x (2005205)
       
       
3. Run these commands to verify that this version was installed on the ESX/ESXi host:
   
   #vem status
   #vemcmd show version
   #vemcmd show card
   #esxupdate query --vib-view | grep cisco
   
   On ESxi 5.0 you can use the command:
   
   #esxcli software vib list|grep -i vem
   
   
4. Add the ESX/ESXi host to Nexus 1000v vDS in vCenter Server.
5. Run this command to verify that the VEM module exists on the Nexus 1000v VSM:
   
   show module
   
   
6. Start the VMware Update Manager service.

Additional Information

Installing Cisco Virtual Ethernet Module (VEM) on an ESX/ESXi host using the command line

How to download and install async drivers in ESXi 5.x/6.x