NSX-T DFW rule filter returns "Error: No server's available: Error code 98 (local manager) or Error code 100 (global manager)"

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX 3.x and 4.x
• When filtering NSX-T DFW rules from a Global Manager or a Local Manager, it takes 5-10 minutes to filter the rules or it fails to load and reports error
• On the NSX Manager, /var/log/proton/nsxapi.log

ERROR http-nio-127.0.0.1-7440-exec-43 NsxBaseRestController 4506 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="manager"] java.util.concurrent.ExecutionException: com.vmware.nsx.management.policy.policyframework.exceptions.EnforcementPointOperatoinException: Error while getting group associations from enforcement point : INVALID_ARGUMENT: Invalid group association ID type.

• On the NSX Manager, /var/log/cloudnet/nsx-ccp.log

2023-XX-XXTXX:XX:XX.XXXZ INFO nsx-rpc:CCP-AphProvider-XXXXXXXX-e2c6-4a70-9721-XXXXXXXXXXX:user-executor-3 RetryUtil 1596 - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="common"] Failed command on attempt 10 of 10 retries. java.lang.RuntimeException: Entry is not ready yet
2023-XX-XXTXX:XX:XX.XXXZ WARN nsx-rpc:CCP-AphProvider-XXXXXXX-e2c6-4a70-9721-XXXXXXXXX:user-executor-3 GroupingMembershipServiceImpl 1596 - [nsx@6876 comp="nsx-controller" level="WARNING" subcomp="Grouping API"] Exception occurred
java.lang.RuntimeException: Entry is not ready yet
    at com.vmware.nsx.rooster.service.GroupingMembershipServiceImpl.lambda$getEntryFromCache$73(GroupingMembershipServiceImpl.java:3385) ~[libnsx_ccp_rooster.jar:?]
    at com.vmware.nsx.ccp.common.RetryUtil.withRetries(RetryUtil.java:26) ~[libnsx_ccp_core.jar:?]
    at com.vmware.nsx.rooster.service.GroupingMembershipServiceImpl.getEntryFromCache(GroupingMembershipServiceImpl.java:3373) ~[libnsx_ccp_rooster.jar:?]
    at com.vmware.nsx.rooster.service.GroupingMembershipServiceImpl.lambda$checkRoosterForIpRelatedContainers$40(GroupingMembershipServiceImpl.java:2454) ~[libnsx_ccp_rooster.jar:?]
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[?:1.8.0_342]
    at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580) ~[?:1.8.0_342]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_342]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_342]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_342]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_342]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_342]

Environment

VMware NSX-T

Resolution

This issue has been resolved in NSX 3.2.3.2 and 4.1.1.

Workaround:
There is no workaround to this issue.
The error may temporarily go away on retry.

Additional Information

Impact/Risks:
It is not possible to filter DFW rules from Global Manager or Local Manager in a Federation environment.