In NSX-T 3.2 versions, these same AD groups could have been integrated successfully with NSX-T. But post upgrade to 4.x, this is getting failed.
The AD group name for which this operation is failing has a name which is prefix of another group name. Ex: You have following AD groups, "pg-nsx-r" and "pg-nsx-ro". You are able to add "pg-nsx-ro" successfully in NSX but operation for "pg-nsx-r" fails with the aforementioned error. Here "pg-nsx-r" name is a prefix of "pg-nsx-ro".
Environment
VMware NSX-T Data Center 4.x VMware NSX-T Data Center
Cause
VMware NSX does a search in the AD server to validate if the group exists. In the affected versions this search function uses the logic "starts with", rather than "exact match".
Resolution
This is a known issue impacting VMware NSX 4.x. This will be fixed in a future version.
Workaround: You can rename the group in AD so that it's name doesn't become prefix of another group.
Additional Information
Impact/Risks: Unable to add AD group having same name prefix of another group