NOTE: (UUID 1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 is a fake example, UUID will be different case-by-case)
Symptoms:
- When deploying TKGS, the following error is observed during the supervisor deployment phase:
Configured Control Plane VMs
Unable to get TLS certificate for vCenter HTTP Reverse Proxy. Details HTTP request failed; GET, url: https://vcsa.example.com/rest/vcenter/certificate-management/vcenter/tls, code: 500, body: '{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vcenter.certificate_management.vcenter.tls.get"],"default_message":"Could not validate permission information for operation com.vmware.vcenter.certificate_management.vcenter.tls.get invocation.","id":"com.vmware.vapi.authorization.permission.error"}]}}'
- Accessing Certificate Management from vCenter web-console through administration > certificate management and a similar error to this (type of certificate may differ i.e. machine certificates vs. trusted root certificates):
- Checking /var/log/vmware/certificatemanagement/certificatemanagement-svcs.log
2023-09-28T15:10:52.394Z [tomcat-exec-4 [] ERROR com.vmware.vapi.authz.impl.AuthorizationFilter opId=] Could not validate permission information for operation com.vmware.vcenter.certificate_management.vcenter.trusted_root_chains.list invocation.
com.vmware.vim.binding.vmodl.fault.SecurityError: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_362]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_362]
at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_362]
- Checking /var/log/vmware/vpxd-svcs/vpxd-svcs.log
2023-09-28T15:10:52.326Z [authz-service-3 [] WARN com.vmware.cis.authorization.impl.AclPrivilegeValidator opId=eb8c657d-26dc-4492-b436-57775e1ba86a IS] User SSO.DOMAIN\cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 does not have privileges [System.View] on object urn%3Aacl%3Aglobal%3Apermissions
2023-09-28T15:10:52.327Z [authz-service-0 [] WARN com.vmware.cis.authorization.impl.AclPrivilegeValidator opId=eb8c657d-26dc-4492-b436-57775e1ba86a IS] User SSO.DOMAIN\cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 does not have privileges [System.View] on object urn%3Aacl%3Aglobal%3Apermissions
From the logs we can see that user cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 does not have privileges.
On service start, there is a prestart script that should generate an ephemeral certificate, recreate the user, set the password in it's password file, and correct permissions if necessary. In this case, setting/correcting permissions failed. At this time, root cause has not been determined.
Add account cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 to CAAdmin group
/usr/lib/vmware-vmafd/bin/dir-cli group modify --name CAAdmins --add cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14
Check if it is added:
/usr/lib/vmware-vmafd/bin/dir-cli group list --name CAAdmins