Unable to deploy third party security machine (svm) in NSX

Products

• VMware NSX

Issue/Introduction

• Service VM deployments in NSX fails with error:

• Issue persists after restarting EAM service.
• You see the following errors at /var/log/proton/nsxapi.log

<Date> ERROR http-nio-127.0.0.1-7443-exec-4 VcQuerier 4151 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP40500" level="ERROR" reqId="###-###-###-###" subcomp="cm-inventory" username="admin"] Unable to create connection to cm with id: <CM-ID>
com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:56) ~[vlsi-client-7.0.2.8786438.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:236) ~[vlsi-client-7.0.2.8786438.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54) ~[vlsi-client-7.0.2.8786438.jar:?]
<Date>  WARN http-nio-127.0.0.1-7443-exec-4 ComputeManagerQueryServiceImpl 4151 - [nsx@6876 comp="nsx-manager" level="WARNING" reqId="###-###-###-###" subcomp="cm-inventory" username="admin"] Unable to get managed object from reference for compute manager <CM-ID> with target type ClusterComputeResource and target id domain-#### due to the following errors Unknown error occured. Error in processing CM request: Unable to create connection to cm with id: <CM-ID>.
com.vmware.nsx.management.cminv.common.exceptions.InventoryException: Error in processing CM request: Unable to create connection to cm with id: <CM-ID>
        at com.vmware.nsx.management.cminv.vcplugin.query.VcQuerier.getObjectFromResourceReference(VcQuerier.java:78) ~[libcm-inv-container.jar:?]
        at com.vmware.nsx.management.cminv.service.ComputeManagerQueryServiceImpl.getObjectFromResourceReference(ComputeManagerQueryServiceImpl.java:267) ~[libcm-inv-container.jar:?]

<Date>  WARN http-nio-127.0.0.1-7443-exec-4 ComputeManagerQueryServiceImpl - [nsx@6876 comp="nsx-manager" level="WARNING" reqId="###-###-###-###" subcomp="cm-inventory" username="admin"] Removing entry from cache for key <CM-ID>:domain-###:UNIFORMLY_ACCESSIBLE_RESOURCES
<Date>  INFO http-nio-127.0.0.1-7443-exec-4 NsxBaseRestController SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Error in API /cm-inventory/api/v1/fabric/compute-collections/<CM-ID>%3Adomain-###/storage-resources?uniform_cluster_access=true&source=realtime caused by exception com.vmware.nsx.management.cminv.common.exceptions.InventoryException:  {"moduleName":"inventory-mgmt","errorCode":7069,"errorMessage":"Unable to get managed object from reference for compute manager <CM-ID> with target type ClusterComputeResource and target id domain-### due to the following errors : Unknown error occured. .","details":"Unable to get managed object from reference for compute manager <CM-ID> with target type ClusterComputeResource and target id domain-### due to the following errors : Unknown error occured. ."}

• Compute Manager connection from NSX UI looks green.
• Looking at vCenter /var/log/vmware/vpxd/vpxd.log you see the error:

<Date> error vpxd[####] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
<Date> error vpxd[####] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000
<Date> error vpxd[####] [Originator@6876 sub=HTTP session map] Out of HTTP sessions: Limited to 2000

Cause

This is an issue with vCenter vpxd service running out of sessions

Resolution

• Reboot the vCenter appliances to reset the sessions.
• Investigate the cause for vCenter to run out of sessions to prevent the issue from reappearing.