Adding an ESX/ESXi host to Nexus 1000v vDS fails with the error: vDS operation failed on host , got (vmodl.fault.SystemError) exception
book
Article ID: 301635
calendar_today
Updated On:
Products
VMware vSphere ESXi
Show More
Show Less
Issue/Introduction
Symptoms:
Unable to add an ESX/ESXi host to Cisco Nexus 1000v VDS
Adding an ESX/ESXi host to Cisco Nexus 1000v VDS fails
Adding the ESX/ESXi host to Nexus 1000v VDS with incorrect version of VEM vib module, which is not as per the support matrix of Cisco Nexus 1000v upgrade guidance, fails
You see the errorCannot complete a vNetwork Distributed Switch operation for one or more host members. vDS operation failed on host <hostname>, got (vmodl.fault.SystemError) exception.
Even after installing the VEM manually on the ESX/ESXi host, the task fails with the same error message at 74% after invoking Update Manager.
vCenter Server log file contains entries similar to:<TIMESTAMP> error 'App' opID=D40E5368-000003B1] vim.fault.DvsOperationBulkFault ------ In-memory logs end -------- <TIMESTAMP> info 'App' opID=D40E5368-000003B1] [VpxLRO] -- FINISH task-123803 -- dvs-288 -- vim.DistributedVirtualSwitch.reconfigure -- 3B3A7374-####-####-####-########D82(AE8F22F7-####-####-####-########0CD) <TIMESTAMP> info 'App' opID=D40E5368-000003B1] [VpxLRO] -- ERROR task-123803 -- dvs-288 -- vim.DistributedVirtualSwitch.reconfigure: vim.fault.DvsOperationBulkFault: Result: (vim.fault.DvsOperationBulkFault) { dynamicType = <unset>, faultCause = (vmodl.MethodFault) null, faultMessage = (vmodl.LocalizableMessage) [ (vmodl.LocalizableMessage) { dynamicType = <unset>, key = "com.vmware.vim.vpxd.dvs.dvsBulkHostFault.label", arg = (vmodl.KeyAnyValue) [ (vmodl.KeyAnyValue) { dynamicType = <unset>, key = "fault", value = "vmodl.fault.SystemError", }, (vmodl.KeyAnyValue) { dynamicType = <unset>, key = "msg", value = "", }, (vmodl.KeyAnyValue) { dynamicType = <unset>, key = "host", value = "<HOSTNAME>", } ], message = <unset>, } ], hostFault = (vim.fault.DvsOperationBulkFault.FaultOnHost) [ (vim.fault.DvsOperationBulkFault.FaultOnHost) { dynamicType = <unset>, host = 'vim.HostSystem:host-1013', fault = (vmodl.fault.SystemError) { dynamicType = <unset>, faultCause = (vmodl.MethodFault) null, reason = "vmodl.fault.SystemError",
In vpxd log file, you see this message related to Update Manager:
<TIMESTAMP>error 'App' opID=3ECDC342-0000019E] [MoDVSwitch::PushBundleToHostInt] failed to install dvs [NAME] host-component on host [host-1073]: got exception: vmodl.fault.SystemError : (vmodl.fault.SystemError) { dynamicType = <unset>, faultCause = (vmodl.MethodFault) null, reason = "VUM API stage failed: ", msg = "", }
In the Update Manager logs, you see entries similar to: <TIMESTAMP>'httpDownload' 1436 INFO] [httpDownload, 556] Downloading http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-%3CVERSION%3E.vib <TIMESTAMP> 'httpDownload' 1436 INFO] [httpDownload, 791] Status code: 404 <TIMESTAMP> 'httpDownload' 1436 ERROR] [httpDownload, 827] Error retrieving document ' http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib . Unhandled status code: 404 <TIMESTAMP> 'httpDownload' 1436 ERROR] [httpDownload, 929] Connect to http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib with proxy off failed: Error retrieving document http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib . Unhandled status code: 404 <TIMESTAMP> 'httpDownload' 1436 WARN] [httpDownload, 420] Download http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib failed, err: Error retrieving document ' http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib . Unhandled status code: 404
Cause
This issue occurs if the VIB requested is not a part of the current VSM.
This is not a VMware issue and may occur either due to a user error or problems with N1K components.
Resolution
To install the VEM module on the ESX/ESXi host:
Stop the VMware Update Manager service.
Install the VEM module manually on the ESX/ESXi host. For information on options for installing the VEM module manually, see:
Run these commands to verify that this version was installed on the ESX/ESXi host: #vem status #vemcmd show version #vemcmd show card #esxupdate query --vib-view | grep cisco On ESxi 5.0 you can use the command: #esxcli software vib list|grep -i vem
Add the ESX/ESXi host to Nexus 1000v vDS in vCenter Server.
Run this command to verify that the VEM module exists on the Nexus 1000v VSM:show module
Start the VMware Update Manager service.
Feedback
thumb_up
Yes
thumb_down
No