Unable to view Certificate Management page and/or errors during TKGS supervisor deployment

Products

• VMware vCenter Server

Issue/Introduction

NOTE: (UUID 1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 is a fake example, UUID will be different case-by-case)

 

Symptoms:

- When deploying TKGS, the following error is observed during the supervisor deployment phase:

Configured Control Plane VMs

Unable to get TLS certificate for vCenter HTTP Reverse Proxy. Details HTTP request failed; GET, url: https://vcsa.example.com/rest/vcenter/certificate-management/vcenter/tls, code: 500, body: '{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vcenter.certificate_management.vcenter.tls.get"],"default_message":"Could not validate permission information for operation com.vmware.vcenter.certificate_management.vcenter.tls.get invocation.","id":"com.vmware.vapi.authorization.permission.error"}]}}'

-  Accessing Certificate Management from vCenter web-console through administration > certificate management and a similar error to this (type of certificate may differ i.e. machine certificates vs. trusted root certificates):

-  Checking /var/log/vmware/certificatemanagement/certificatemanagement-svcs.log 

2023-09-28T15:10:52.394Z [tomcat-exec-4 [] ERROR com.vmware.vapi.authz.impl.AuthorizationFilter opId=] Could not validate permission information for operation com.vmware.vcenter.certificate_management.vcenter.trusted_root_chains.list invocation.
com.vmware.vim.binding.vmodl.fault.SecurityError: null
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_362]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_362]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_362]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_362]
        at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_362]

 

-  Checking /var/log/vmware/vpxd-svcs/vpxd-svcs.log

2023-09-28T15:10:52.326Z [authz-service-3 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=eb8c657d-26dc-4492-b436-57775e1ba86a IS] User SSO.DOMAIN\cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 does not have privileges [System.View] on object urn%3Aacl%3Aglobal%3Apermissions
2023-09-28T15:10:52.327Z [authz-service-0 [] WARN  com.vmware.cis.authorization.impl.AclPrivilegeValidator  opId=eb8c657d-26dc-4492-b436-57775e1ba86a IS] User SSO.DOMAIN\cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 does not have privileges [System.View] on object urn%3Aacl%3Aglobal%3Apermissions

Cause

From the logs we can see that user cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 does not have privileges.  

On service start, there is a prestart script that should generate an ephemeral certificate, recreate the user, set the password in it's password file, and correct permissions if necessary.  In this case, setting/correcting permissions failed.  At this time, root cause has not been determined.

Resolution

Add account cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14 to CAAdmin group 

/usr/lib/vmware-vmafd/bin/dir-cli group modify --name CAAdmins --add cms-1a2b3c4d-5e6f-7g8h-9a10-b11c12d13e14

Check if it is added:

/usr/lib/vmware-vmafd/bin/dir-cli group list --name CAAdmins