Adding an ESX/ESXi host to Nexus 1000v vDS fails with the error: vDS operation failed on host , got (vmodl.fault.SystemError) exception

Products

• VMware

Issue/Introduction

Symptoms:

• Unable to add an ESX/ESXi host to Cisco Nexus 1000v VDS
• Adding an ESX/ESXi host to Cisco Nexus 1000v VDS fails
• Adding the ESX/ESXi host to Nexus 1000v VDS with incorrect version of VEM vib module, which is not as per the support matrix of Cisco Nexus 1000v upgrade guidance, fails
• You see the error
  
  Cannot complete a vNetwork Distributed Switch operation for one or more host members. vDS operation failed on host <hostname>, got (vmodl.fault.SystemError) exception.
  
• Even after installing the VEM manually on the ESX/ESXi host, the task fails with the same error message at 74% after invoking Update Manager.
  
• vCenter Server log file contains entries similar to:
  
  <TIMESTAMP> error 'App' opID=D40E5368-000003B1] vim.fault.DvsOperationBulkFault
  
  ------ In-memory logs end --------
  <TIMESTAMP> info 'App' opID=D40E5368-000003B1] [VpxLRO] -- FINISH task-123803 -- dvs-288 -- vim.DistributedVirtualSwitch.reconfigure -- 3B3A7374-9856-4D1D-A6CB-C0EC9F833D82(AE8F22F7-6533-496A-AA15-7600F3D540CD)
  <TIMESTAMP> info 'App' opID=D40E5368-000003B1] [VpxLRO] -- ERROR task-123803 -- dvs-288 -- vim.DistributedVirtualSwitch.reconfigure: vim.fault.DvsOperationBulkFault:
  Result:
  (vim.fault.DvsOperationBulkFault) {
  dynamicType = <unset>,
  faultCause = (vmodl.MethodFault) null,
  faultMessage = (vmodl.LocalizableMessage) [
  (vmodl.LocalizableMessage) {
  dynamicType = <unset>,
  key = "com.vmware.vim.vpxd.dvs.dvsBulkHostFault.label",
  arg = (vmodl.KeyAnyValue) [
  (vmodl.KeyAnyValue) {
  dynamicType = <unset>,
  key = "fault",
  value = "vmodl.fault.SystemError",
  },
  (vmodl.KeyAnyValue) {
  dynamicType = <unset>,
  key = "msg",
  value = "",
  },
  (vmodl.KeyAnyValue) {
  dynamicType = <unset>,
  key = "host",
  value = "<HOSTNAME>",
  }
  ],
  message = <unset>,
  }
  ],
  hostFault = (vim.fault.DvsOperationBulkFault.FaultOnHost) [
  (vim.fault.DvsOperationBulkFault.FaultOnHost) {
  dynamicType = <unset>,
  host = 'vim.HostSystem:host-1013',
  fault = (vmodl.fault.SystemError) {
  dynamicType = <unset>,
  faultCause = (vmodl.MethodFault) null,
  reason = "vmodl.fault.SystemError",
  
  
• In vpxd log file, you see this message related to Update Manager:
  <TIMESTAMP>error 'App' opID=3ECDC342-0000019E] [MoDVSwitch::PushBundleToHostInt] failed to install dvs [NAME] host-component on host [host-1073]: got exception:
  vmodl.fault.SystemError : (vmodl.fault.SystemError) {
  dynamicType = <unset>,
  faultCause = (vmodl.MethodFault) null,
  reason = "VUM API stage failed: ",
  msg = "",
  }
  
  
• In the Update Manager logs, you see entries similar to:
  
  <TIMESTAMP>'httpDownload' 1436 INFO] [httpDownload, 556] Downloading http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-%3CVERSION%3E.vib
  <TIMESTAMP> 'httpDownload' 1436 INFO] [httpDownload, 791] Status code: 404
  <TIMESTAMP> 'httpDownload' 1436 ERROR] [httpDownload, 827] Error retrieving document 'http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib. Unhandled status code: 404
  <TIMESTAMP> 'httpDownload' 1436 ERROR] [httpDownload, 929] Connect to http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vibwith proxy off failed: Error retrieving document http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib. Unhandled status code: 404
  <TIMESTAMP> 'httpDownload' 1436 WARN] [httpDownload, 420] Download http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vibfailed, err: Error retrieving document 'http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib. Unhandled status code: 404

Cause

This issue occurs if the VIB requested is not a part of the current VSM.

This is not a VMware issue and may occur either due to a user error or problems with N1K components.

Resolution

To install the VEM module on the ESX/ESXi host:

1. Stop the VMware Update Manager service.
2. Install the VEM module manually on the ESX/ESXi host.
   
   For information on options for installing the VEM module manually, see:
   
   • ESX/ESXi 4.x
     
     • Installing async drivers on ESXi 4.x and ESX 4.x (1032936)
     • Configuring patch source in vCenter Update Manager for Cisco Nexus 1000V Virtual Ethernet Module (VEM) (1013134)
     • Installing Cisco Virtual Ethernet Module (VEM) on an ESX host using the command line (1013551)
       
       
   • ESXi 5.0
     
     • Installing async drivers on ESXi 5.x (2005205)
       
       
3. Run these commands to verify that this version was installed on the ESX/ESXi host:
   
   #vem status
   #vemcmd show version
   #vemcmd show card
   #esxupdate query --vib-view | grep cisco
   
   On ESxi 5.0 you can use the command:
   
   #esxcli software vib list|grep -i vem
   
   
4. Add the ESX/ESXi host to Nexus 1000v vDS in vCenter Server.
5. Run this command to verify that the VEM module exists on the Nexus 1000v VSM:
   
   show module
   
   
6. Start the VMware Update Manager service.

Additional Information

Configuring patch source in vCenter Update Manager for Cisco Nexus 1000V Virtual Ethernet Module (VEM)
Installing Cisco Virtual Ethernet Module (VEM) on an ESX/ESXi host using the command line
Installing async drivers on ESXi 4.x and ESX 4.x
How to download and install async drivers in ESXi 5.x/6.x