In NSX-T 3.2 versions, these same AD groups could have been integrated successfully with NSX-T. But post upgrade to 4.x, this is getting failed.
The AD group name for which this operation is failing has a name which is prefix of another group name. Ex: You have following AD groups, "pg-nsx-r" and "pg-nsx-ro". You are able to add "pg-nsx-ro" successfully in NSX but operation for "pg-nsx-r" fails with the aforementioned error. Here "pg-nsx-r" name is a prefix of "pg-nsx-ro".
Environment
VMware NSX-T Data Center 4.x VMware NSX-T Data Center
Cause
VMware NSX does a search in the AD server to validate if the group exists. In the affected versions this search function uses the logic "starts with", rather than "exact match".
Resolution
This issue has been resolved in versions 4.1.2.4, 4.2.0, and later.
Workaround: You can rename the group in AD so that it's name doesn't become prefix of another group.
Additional Information
Impact/Risks: Unable to add AD group having same name prefix of another group