Updating certificates using certificate manager on vCenter Server or PSC 6.0 Update 1b fails
search cancel

Updating certificates using certificate manager on vCenter Server or PSC 6.0 Update 1b fails

book

Article ID: 335964

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

After replacing solution user certificates using the certificate manager, you experience these symptoms:

  • In the %ALLUSERSPROFILE%\VMWare\vCenterServer\logs\invsvc.log file, you see entries similar to:

    [YYYY-MM-DDTHH:MM:SS] [pool-12-thread-1 INFO com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Failed trying to retrieve token: ns0:RequestFailed: Error occured looking for solution user :: More than one solution user found
    [YYYY-MM-DDTHH:MM:SS] [pool-12-thread-1 ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper opId=] Hit ServiceCommunicationException while fetching admin group for the SSO Admin user : [email protected]
    com.vmware.vim.query.server.ssoauthentication.exception.ServiceCommunicationException: com.vmware.vim.sso.client.exception.InternalError: Failed trying to retrieve token: ns0:RequestFailed: Error occured looking for solution user :: More than one solution user found
     
  • In the %ALLUSERSPROFILE%\VMWare\vCenterServer\logs\vsphere-client.log file, you see entries similar to:

    [YYYY-MM-DDTHH:MM:SS] [INFO ] usage-data-collector-thread com.vmware.vise.vim.security.sso.impl.SsoUtilInternal Preparing the STS configuration for https://psc.domain.com/sts/STSService/domain.local
    [YYYY-MM-DDTHH:MM:SS] [INFO ] usage-data-collector-thread com.vmware.vise.vim.security.sso.impl.SsoUtilInternal Requesting all STS trusted root certificates from https://psc.domain.com/sso-adminserver/sdk/domain.local
    [YYYY-MM-DDTHH:MM:SS] [WARN ] usage-data-collector-thread .c.h.i.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase Shutting down the connection monitor.
    [YYYY-MM-DDTHH:MM:SS] [ERROR] usage-data-collector-thread com.vmware.vim.sso.client.impl.SoapBindingImpl SOAP fault javax.xml.ws.soap.SOAPFaultException: Error occurred looking for solution user :: More than one solution user found

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
     
  • In the /var/log/vmware/vapi/endpoint.log file, you see entries similar to:

    com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.sso.client.exception.InternalError: Failed trying to retrieve token: ns0:RequestFailed: Error occurred looking for solution user :: More than one solution user found
    at com.vmware.vapi.endpoint.cis.StsBuilder.createToken(StsBuilder.java:178)
    at com.vmware.vapi.endpoint.cis.StsBuilder.rebuild(StsBuilder.java:73)
    at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:52)
    at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:349)
    at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:176)
    at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: com.vmware.vim.sso.client.exception.InternalError: Failed trying to retrieve token: ns0:RequestFailed: Error occurred looking for solution user :: More than one solution user found
    Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start vapi-endpoint services. Error: Operation timed out.
     
  • The vSphere Web Client fails with the error:

    A server error occurred.
    [500] SSO error: null
    Check the vSphere Web Client server logs for details.
     
  • Navigating to the https://fqdn/psc/ fails with the error:

    HTTP Status 400 - An error occurred while sending an authentication request to the PSC Single Sign-On server - null
    type Status report
    message An error occurred while sending an authentication request to the PSC Single Sign-On server - null
    description The request sent by the client was syntactically incorrect.
    VMware vFabric tc Runtime 2.9.7.RELEASE/7.0.55.A.RELEASE



Environment

VMware vCenter Server 6.0.x

Cause

This issue is caused by a change in the certificate-manager in vCenter Server Update 1b. New options are present for processing the certool.cfg file correctly, as well as processing config files for each individual solution user. If these config files do not have unique information for each solution user, the generated certificates have the same Subject.

For example, in the C:\ProgramData\VMware\vCenterServer\logs\vmca\certificate-manager.log file, you see entries similar to:

[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager Selected operation: Replace Solution user certs with VMCA Certificate
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager Please configure machine.cfg with proper values before proceeding to next step.
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager Press Enter key to skip optional parameters or use Default value.
[YYYY-MM-DDTHH:MM:SS]INFO certificate-manager machine.cfg file contents.
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager Country = US
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager Name = vSphere
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager Organization = VMware
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager OrgUnit = Support
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager State = Colorado
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager Locality = Denver
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager #IPAddress =
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager Email = [email protected]
[YYYY-MM-DDTHH:MM:SS] INFO certificate-manager Hostname = vcsa.domain.com

The same information will be seen for these options in the other config files (vsphere-webclient.cfg, vpxd.cfg, vpxd-extension.cfg) which causes the certificates not to be unique.

Resolution

This issue is resolved in vCenter Server 6.0 Update 3,  To download go to - Download Broadcom products and software

 

Workaround:
To workaround this issue, re-generate new Solution User Certificates, ensuring that each certificate is given a unique subject.
This can typically be achieved by making the Name: value unique for each Solution user.
Using the Certificate Manager > Select Option 6 to re-generate new VMCA issue SOlution User Certificate.

Additional Information

Powered by Wolken Software