Unable to Register a CM [vcenter-#####.org] Failed due to [Unknown error in processing request for CM vcenter-#####.org with id f24fad70-d4d2-4cec-####-##########] "errorCode":40200

Products

• VMware NSX

Issue/Introduction

• From NSX-T under System>Compute manager --> the following can be seen: 
  
  Unknown error in processing request for CM vcenter-xxx.xxxx.org with id f24fad70-d4d2-4cec-#####-###########
• SecurityError exception will be coming from VC while registering NSXT extension.
  
  Log location --- var/log/cm-inventory/cm-inventory.log 
  
  2021-06-11T16:34:05.207Z  WARN http-nio-127.0.0.1-7443-exec-2 VcExtensionManagerImpl 29151 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="cm-inventory"] 
  com.vmware.vim.binding.vmodl.fault.SecurityError: Access to perform the operation was denied.
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_281]
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_281]
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_281]
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_281]
   at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_281]
   at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core-7.0.1.8343824.jar:?]
   at
  truncated...
  com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195) ~[vlsi-client-7.0.1.8343824.jar:?]
   at com.sun.proxy.$Proxy157.registerExtension(Unknown Source) ~[?:?]
   at com.vmware.nsx.management.cminv.vcplugin.VcExtensionManagerImpl.registerVcPluginExtension(VcExtensionManagerImpl.java:131) ~[libcm-inv-container.jar:?]
  
  
  Log location --- /var/log/proton/nsxapi.log 
  
  2021-06-11T16:34:05.211Z  INFO http-nio-127.0.0.1-7443-exec-2 VcPlugin 29151 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Registration of extension to CM vcenter-wl1.##.com failed: ErrorCode{id=40200, reason='Unknown error in processing request for CM {0} with id {1}'}
  2021-06-11T16:34:05.255Z  INFO http-nio-127.0.0.1-7443-exec-2 NsxBaseRestController 29151 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Error in API /cm-inventory/api/v1/cm-plugin-container/cm-plugins?action=register caused by exception com.vmware.nsx.management.cminv.exceptions.CmPluginRegistrationFailedException:  {"moduleName":"cm-inventory","errorCode":40200,"errorMessage":"Unknown error in processing request for CM vcenter-wl1.##.com with id f24fad70-d4d2-4cec-####-#############"}
  
  
  From VC under  var/log/vmware/vpxd/vpxd-###.log
  
  2021-06-11T16:34:05.154Z warning vpxd[05585] [Originator@6876 sub=LSClient opID=cm-inventory-#.#.#.#-80] Service registration stub privilege error during lookup service RPC: N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.SecurityError
  --> )
  --> [context]########################################################QYbGlidm1vbWkuc28AARFoGAEekw8BccoUASZaFAGFgg0BLX0NAu+MAmxpYmxvb2t1cC10eXBlcy5zbwADb8RndnB4ZAADM8lnA7oGZAQ/vexsaWJ2aW0tdHlwZXMuc28AAx3MdgOayHUD#################################################/NQ9sa#########[/context]
  2021-06-11T16:34:05.155Z info vpxd[05585] [Originator@6876 sub=LSClient opID=cm-inventory-#.#.#.#-80] Refreshing lookup service token
  2021-06-11T16:34:05.204Z info vpxd[05585] [Originator@6876 sub=vpxLro opID=cm-inventory-#.#.#.#-80] [VpxLRO] -- FINISH lro-30801921
  2021-06-11T16:34:05.204Z info vpxd[05585] [Originator@6876 sub=Default opID=cm-inventory-#.#.#.#-80] [VpxLRO] -- ERROR lro-30801921 -- ExtensionManager -- vim.ExtensionManager.registerExtension: vmodl.fault.SecurityError:
  --> Result:
  --> (vmodl.fault.SecurityError) {
  -->    faultCause = (vmodl.MethodFault) null, 
  -->    faultMessage = <unset>
  -->    msg = "Received SOAP response fault from [<cs p:00007f25c45c11c0, TCP:vcenter-wl1.##.com:443>]: create
  --> "
  --> }
  --> Args:
  --> 
  --> Arg extension:
  --> (vim.Extension) {
  -->    description = (vim.Description) {
  -->       label = "NSX-T Manager", 
  -->       summary = "NSX-T networking and security solutions"
  -->    }, 
  -->    key = "com.vmware.nsx.management.nsxt", 
  -->    company = "VMware", 
  --> 
  
  Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.
  
  

Environment

VMware NSX-T
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

This occurs after SSO domain is repointed or changed, VC can not be registered as compute manager in NSXT because of failure during registering NSXT extension on VC.

Resolution

Workaround:
Add the vCenter VPXD solution user directly to the Administrators group.

There is an script to take care of such :
 

-----BEGIN SCRIPT-----
#!/bin/sh
HOST=localhost
ADMIN=cn=Administrator,cn=Users,dc=vsphere,dc=local
/opt/likewise/bin/ldapmodify -h "$HOST" -D "$ADMIN" -W <<EOM
dn: cn=Administrators,cn=Builtin,dc=vsphere,dc=local
changetype: modify
add: member
member: CN=vpxd-42a9751e-4839-427b-####-###########,CN=ServicePrincipals,dc=vsphere,dc=local
EOM
-----END SCRIPT-----

NOTES: 

You would just have to find the particular VPXD solution user name for this vCenter instance and replace that in the script above (i.e. replace 'vpxd-42a9###-####-...' with the real VPXD solution user name). 

This solution user can be found by running the following on VC

/usr/lib/vmware-vmafd/bin/dir-cli group list --name SolutionUsers 

Also keep in mind that your Domain may not be [vsphere.local] as example could be [###.vi]

Example of steps to run the WA :

#0-  I ran the following from the VC with the aim of finding the  VPXD solution ID  but I see 3 possible VPXD solution IDS 

root@vcenter-wl1 [ ~ ]# /usr/lib/vmware-vmafd/bin/dir-cli group list --name SolutionUsers

Enter password for administrator@###.vi:
CN=machine-6f562e3d-ad1e-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=vsphere-webclient-6f562e3d-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=vpxd-6f562e3d-ad1e-42f0-####-####################,CN=ServicePrincipals,DC=###,DC=vi        ###1####
CN=vpxd-extension-6f562e3d-ad1e-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=hvc-6f562e3d-ad1e-42f0-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=wcp-6f562e3d-ad1e-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=workload_storage_management-6f562e3d-ad1e-####-#####################,cn=ServicePrincipals,dc=###,dc=vi
CN=hvc-a5f26514-f634-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=wcp-a5f26514-f634-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=workload_storage_management-a5f26514-f634-####-#####################,cn=ServicePrincipals,dc=###,dc=vi
CN=machine-f04b80a3-b97b-4ebb-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=vsphere-webclient-f04b80a3-b97b-4ebb-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=vpxd-f04b80a3-b97b-4ebb-####-####################,CN=ServicePrincipals,DC=###,DC=vi         ###2###
CN=vpxd-extension-f04b80a3-b97b-4ebb-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=hvc-f04b80a3-b97b-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=wcp-f04b80a3-b97b-4ebb-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=workload_storage_management-f04b80a3-b97b-4ebb-####-#####################,cn=ServicePrincipals,dc=###,dc=vi
CN=machine-4bd6131a-c5cf-4cdf-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=vsphere-webclient-4bd6131a-c5cf-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=vpxd-4bd6131a-c5cf-4cdf-####-####################,CN=ServicePrincipals,DC=###,DC=vi              ###3###
CN=vpxd-extension-4bd6131a-c5cf-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=hvc-4bd6131a-c5cf-4cdf-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=wcp-4bd6131a-c5cf-4cdf-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=workload_storage_management-4bd6131a-####-#####################,cn=ServicePrincipals,dc=###,dc=vi
CN=nsxt_834ac160-40cb-####-#####################,cn=ServicePrincipals,dc=###,dc=vi
CN=VMware_CWP_SOLUTION_d6aa2a05-4201-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=VMware_CWP_SOLUTION_9f744faf-82bc-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=VMware_CWP_SOLUTION_6cb5cc54-fe4f-4208-####-#####################,CN=ServicePrincipals,DC=###,DC=vi
CN=nsxt_698d01e8-5a20-4d84-####-#####################,cn=ServicePrincipals,dc=###,dc=vi
root@vcenter-wl1 [ ~ ]#


From the output above we took the 3 possible VPXD solution IDS 

CN=vpxd-6f562e3d-ad1e-####-#####################,CN=ServicePrincipals,DC=###,DC=vi    ####1####
CN=vpxd-f04b80a3-b97b-####-#####################,CN=ServicePrincipals,DC=###,DC=vi   ####2####
CN=vpxd-4bd6131a-c5cf-####-#####################,CN=ServicePrincipals,DC=###,DC=vi        ####3####


#1- SSH into VC

#2- Create 3 separated scripts

vim user1solution.sh

#!/bin/sh
HOST=localhost
ADMIN=cn=Administrator,cn=Users,dc=###,dc=vi
/opt/likewise/bin/ldapmodify -h "$HOST" -D "$ADMIN" -W <<EOM
dn: cn=Administrators,cn=Builtin,dc=###,dc=vi
changetype: modify
add: member
member: CN=vpxd-6f562e3d-ad1e-42f0-####-#####################,CN=ServicePrincipals,dc=###,dc=vi
EOM

vim user2solution.sh

 

#!/bin/sh
HOST=localhost
ADMIN=cn=Administrator,cn=Users,dc=###,dc=vi
/opt/likewise/bin/ldapmodify -h "$HOST" -D "$ADMIN" -W <<EOM
dn: cn=Administrators,cn=Builtin,dc=###,dc=vi
changetype: modify
add: member
member: CN=vpxd-f04b80a3-b97b-####-#####################,CN=ServicePrincipals,dc=###,dc=vi
EOM

vim user3solution.sh
 

#!/bin/sh
HOST=localhost
ADMIN=cn=Administrator,cn=Users,dc=###,dc=vi
/opt/likewise/bin/ldapmodify -h "$HOST" -D "$ADMIN" -W <<EOM
dn: cn=Administrators,cn=Builtin,dc=###,dc=vi
changetype: modify
add: member
member: CN=vpxd-4bd6131a-c5cf-####-#####################,CN=ServicePrincipals,dc=###,dc=vi
EOM

#3-- Give execute rights to all 3 files

chmod a+x user1solution.sh
chmod a+x user2solution.sh
chmod a+x user3solution.sh

# --Run the Scripts

./user1solution.sh
./user2solution.sh
./user3solution.sh

#4-- Go back to the failed Compute Manager registration and you should click on error on Registration status and resolve the error.