New Segment creation is stuck on NSX Manager UI due to an exception in the grouping workflow.

Products

• VMware NSX

Issue/Introduction

Symptoms:

1. NSX UI is inaccessible.
2. When creating a new segment from the NSX Manager UI, the segment creation cannot go past the "In progress" status and sometimes fails. When trying to delete these segments, they are stuck in the "mark for delete" status. You will need to run NSX APIs to delete the segments.

Log Locations which reflect the issue
/var/log/cloudnet/nsx-ccp.log

• Year-MM-Date:XX:XX.XXXZ INFO nsx-rpc:CCP-AphProvider-<UUID>:user-executor-1 NsxRpcChannel 1487 - [nsx@68 76 comp="nsx-manager" level="INFO" subcomp="ccp"] ServerChannel(vmware.nsx.grouping.GroupingMembershipService, <UUID>.doClose(closeStream=false, status=Status(code=UNKNOWN, msg=leaked exception: java.util.ConcurrentModificationException

/var/log/proton/nsxapi.log

• Year-MM-Date:XX:XX.XXXZ WARN intelligence-agent-delta-config-producer PolicyProducerUtils 4239 INTELLIGENCE [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Failed to retrieve all related compute members of group Group/<UUID>
• com.vmware.nsx.rpc.StatusRuntimeException: UNKNOWN: leaked exception: java.util.ConcurrentModificationException
• Year-MM-Date:XX:XX.XXXZ INFO UfoDelayedIndexer-0 ConsolidatedRealizedStateServiceImpl 4265 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updated consolidated state for intentPath:/infra/segments/LS_TEST8 to:IN_PROGRES

Environment

VMware NSX 4.0.0.1

Cause

• CCP encountered an known issue in 3.2.1 and threw a ConcurrentModificationException when Policy query CCP via Grouping RPC service.
• There is another issue in NSX RPC Lib, which prevented RPC client (in this case it is Policy Grouping) from receiving stub down event, and caused Policy Grouping to continue query over the already destroyed RPC stub
• There is a dependency on L2 Policy over Policy grouping, and because of issue #2, it blocks the new Segment creation/realization on UI

Resolution

This is a known issue in versions 3.2.1 and 4.0.0.
However, 

• CCP exception is fixed in versions 3.2.1.1, 3.2.2, 4.0.1
• RPC Lib issue is fixed in 3.2.2, 4.0.2, 4.1.0

Workaround:
Temporary workaround is to restart Controller service on all three Unified Appliance nodes.

Additional Information

Impact/Risks:
New Segment creation is stuck on UI due to Exception in Grouping workflow