Adding an ESX/ESXi host to Nexus 1000v vDS fails with the error: vDS operation failed on host , got (vmodl.fault.SystemError) exception
search cancel

Adding an ESX/ESXi host to Nexus 1000v vDS fails with the error: vDS operation failed on host , got (vmodl.fault.SystemError) exception

book

Article ID: 301635

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • Unable to add an ESX/ESXi host to Cisco Nexus 1000v VDS
  • Adding an ESX/ESXi host to Cisco Nexus 1000v VDS fails
  • Adding the ESX/ESXi host to Nexus 1000v VDS with incorrect version of VEM vib module, which is not as per the support matrix of Cisco Nexus 1000v upgrade guidance, fails
  • You see the error

    Cannot complete a vNetwork Distributed Switch operation for one or more host members. vDS operation failed on host <hostname>, got (vmodl.fault.SystemError) exception.
  • Even after installing the VEM manually on the ESX/ESXi host, the task fails with the same error message at 74% after invoking Update Manager.
  • vCenter Server log file contains entries similar to:

    <TIMESTAMP> error 'App' opID=D40E5368-000003B1] vim.fault.DvsOperationBulkFault

    ------ In-memory logs end --------
    <TIMESTAMP> info 'App' opID=D40E5368-000003B1] [VpxLRO] -- FINISH task-123803 -- dvs-288 -- vim.DistributedVirtualSwitch.reconfigure -- 3B3A7374-####-####-####-########D82(AE8F22F7-####-####-####-########0CD)
    <TIMESTAMP> info 'App' opID=D40E5368-000003B1] [VpxLRO] -- ERROR task-123803 -- dvs-288 -- vim.DistributedVirtualSwitch.reconfigure: vim.fault.DvsOperationBulkFault:
    Result:
    (vim.fault.DvsOperationBulkFault) {
    dynamicType = <unset>,
    faultCause = (vmodl.MethodFault) null,
    faultMessage = (vmodl.LocalizableMessage) [
    (vmodl.LocalizableMessage) {
    dynamicType = <unset>,
    key = "com.vmware.vim.vpxd.dvs.dvsBulkHostFault.label",
    arg = (vmodl.KeyAnyValue) [
    (vmodl.KeyAnyValue) {
    dynamicType = <unset>,
    key = "fault",
    value = "vmodl.fault.SystemError",
    },
    (vmodl.KeyAnyValue) {
    dynamicType = <unset>,
    key = "msg",
    value = "",
    },
    (vmodl.KeyAnyValue) {
    dynamicType = <unset>,
    key = "host",
    value = "<HOSTNAME>",
    }
    ],
    message = <unset>,
    }
    ],
    hostFault = (vim.fault.DvsOperationBulkFault.FaultOnHost) [
    (vim.fault.DvsOperationBulkFault.FaultOnHost) {
    dynamicType = <unset>,
    host = 'vim.HostSystem:host-1013',
    fault = (vmodl.fault.SystemError) {
    dynamicType = <unset>,
    faultCause = (vmodl.MethodFault) null,
    reason = "vmodl.fault.SystemError",

  • In vpxd log file, you see this message related to Update Manager:
    <TIMESTAMP>error 'App' opID=3ECDC342-0000019E] [MoDVSwitch::PushBundleToHostInt] failed to install dvs [NAME] host-component on host [host-1073]: got exception:
    vmodl.fault.SystemError : (vmodl.fault.SystemError) {
    dynamicType = <unset>,
    faultCause = (vmodl.MethodFault) null,
    reason = "VUM API stage failed: ",
    msg = "",
    }

  • In the Update Manager logs, you see entries similar to:

    <TIMESTAMP>'httpDownload' 1436 INFO] [httpDownload, 556] Downloading http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-%3CVERSION%3E.vib
    <TIMESTAMP> 'httpDownload' 1436 INFO] [httpDownload, 791] Status code: 404
    <TIMESTAMP> 'httpDownload' 1436 ERROR] [httpDownload, 827] Error retrieving document '
    http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib
    . Unhandled status code: 404
    <TIMESTAMP> 'httpDownload' 1436 ERROR] [httpDownload, 929] Connect to
    http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vibwith proxy off failed: Error retrieving document http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib
    . Unhandled status code: 404
    <TIMESTAMP> 'httpDownload' 1436 WARN] [httpDownload, 420] Download
    http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vibfailed, err: Error retrieving document 'http://%3cvsm%20ip%3e/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v122-4.0.4.1.3.2.0-2.0.1.vib
    . Unhandled status code: 404

 

Cause

This issue occurs if the VIB requested is not a part of the current VSM.
This is not a VMware issue and may occur either due to a user error or problems with N1K components.

Resolution

To install the VEM module on the ESX/ESXi host:
  1. Stop the VMware Update Manager service.
  2. Install the VEM module manually on the ESX/ESXi host.

    For information on options for installing the VEM module manually, see:
  3. Run these commands to verify that this version was installed on the ESX/ESXi host:

    #vem status
    #vemcmd show version
    #vemcmd show card
    #esxupdate query --vib-view | grep cisco

    On ESxi 5.0 you can use the command:

    #esxcli software vib list|grep -i vem

  4. Add the ESX/ESXi host to Nexus 1000v vDS in vCenter Server.
  5. Run this command to verify that the VEM module exists on the Nexus 1000v VSM:

    show module

  6. Start the VMware Update Manager service.



Additional Information