Creation of a logical switch using the NSX REST API POST:/api/v1/logical-switches shows a "failed" state before eventually transitioning to "success."

Products

• VMware NSX
• VMware vCenter Server

Issue/Introduction

When creating logical switches via the NSX API using POST /api/v1/logical-switches, the API responds immediately indicating that the logical switch object has been created

However, if we check the state of the logical switch using the endpoint: GET /api/v1/logical-switches/<logical-switch-id>/state, the state may initially show as "FAILED".

{
    "logical_switch_id": "<new-id>",
    "state": "failed",
    "details": []
}

The status of the logical switch creation eventually resolves within 5-10 seconds, and the response changes to "success":

GET:/api/v1/logical-switches/<new-id>/state

{     
     "logical_switch_id": "<new-id>",
     "state": "success",
     "details": [] 
}

Environment

VMware vCenter Server 8.x

VMware NSX 4.2.1

Cause

The cause is that one or more ESXi hosts are disconnected from vCenter Server, which leads vCenter to report a DvsOperationBulkFault error when attempting to create or reconfigure an NSX portgroup.

From NSX Manager logs (/var/log/proton/nsxapi.log), you may observe the following entries:

ERROR LogicalSwitchStateOperationScheduler2 VcUtilities 98886 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP40407" level="ERROR" subcomp="manager"] Wait for completion failed with 'Cannot complete a vSphere Distributed Switch operation for one or more host members.'
com.vmware.vim.binding.vim.fault.DvsOperationBulkFault: Cannot complete a vSphere Distributed Switch operation for one or more host members.
at jdk.internal.reflect.GeneratedConstructorAccessor458.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
at java.lang.Class.newInstance(Unknown Source) ~[?:?]

 

From vCenter Server logs (/var/log/vmware/vpxd.log), you may observe the following error message:

info vpxd[07985] [Originator@6876 sub=vpxLro opID=eb93883] [VpxLRO] -- BEGIN session[########-####-####-####-############]########-####-####-####-############ -- dvs-######-- vim.DistributedVirtualSwitch.addNsxPortgroups -- ########-####-####-####-############(########-####-####-####-############)

info vpxd[07768] [Originator@6876 sub=vmomi.soapStub[144611] opID=eb93883] SOAP request returned HTTP failure; <<io_obj p:0x00007fa640aa3e68, h:267, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#####/vpxa>, method: applyDVPortgroup; code: 503(Service Unavailable); fault: (null)

error vpxd[07768] [Originator@6876 sub=hostMethod opID=######] Host call [applyDVPortgroup] for host [[vim.HostSystem:host-####,[HOSTNAME]] failed with exception [HTTP error response: Service Unavailable]

error vpxd[07985] [Originator@6876 sub=DvsCore opID=######] API [applyDVPortgroup] on host [vim.HostSystem:host-####] failed with exception [HTTP error response: Service Unavailable]

warning vpxd[07985] [Originator@6876 sub=Vmomi opID=######] VMOMI activation LRO failed; <<########-####-####-####-############, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 33836'>>, dvs-######, vim.DistributedVirtualSwitch.addNsxPortgroups, <vim.version.v8_0_3_0, official, 8.0.3.0>, (null)>, N3Vim5Fault21DvsOperationBulkFault9ExceptionE(Fault cause: vim.fault.DvsOperationBulkFault
info vpxd[07985] [Originator@6876 sub=vpxLro opID=######] [VpxLRO] -- FINISH session[########-####-####-####-############]########-####-####-####-############

error vpxd[07985] [Originator@6876 sub=Default opID=eb93883] [VpxLRO] -- ERROR session[########-####-####-####-############]########-####-####-####-############ -- ########-####-####-####-############(########-####-####-####-############) -- dvs-######-- vim.DistributedVirtualSwitch.addNsxPortgroups: :vim.fault.DvsOperationBulkFault

 

 

Resolution

This is a known issue, and the fix is available in vCenter Server version 8.0.U3e and vCenter Server version 9.0.