Content library creation failed with error message "AuthenticationException"

Products

• VMware vCenter Server

Issue/Introduction

• All the content libraries went missing after custom machine SSL certificate of vcenter was renewed.
• Content library creation fails with the error : "com.vmware.vcenter.apigw.security.AuthenticationException: Failed to log into [uri=http://localhost:1080/cls/, sessionMgr=SessionManagerInfo [_sessionMgrSvcId=com.vmware.cis.session, _loginOpId=create, _logoutOpId=delete], ssoDomain=vsphere.local(c4beb1f0-ce40-11e7-bf3a-000c29c2bb9c)]: com.vmware.vapi.std.errors.unauthenticated => {data=<unset>, error_type=UNAUTHENTICATED, messages=[com.vmware.vapi.std.localizable_message => {args=[], default_message=This method requires authentication., localized=<unset>, id=vapi.method.authentication.required, params=<unset>}]}"
• Below errors are observed in /var/log/vmware/content-library/cls.log:

  Caused by: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
  Caused by: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint doesn't match
  Caused by: java.security.cert.CertificateException: Unable to construct a valid chain
  Caused by: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found

• Accessing lifecycle manager fails with an error message "An unexpected error has occurred."

Environment

VMware vCenter Server 8.0.3

Cause

• The root and intermediate certificates signing the machine SSL certificate was missing in VECS although present in VMDIR. 

Resolution

• List the trusted certificates published to the VMware Directory Service: /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list
• List the trusted certificates published in VECS using vecs-cli : /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
• Identify the certificates missing in VECS. These certificates are seen in the vSphere client.
• VECS refresh using the command fails with the below error message: /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh  

  Error: Failed to trigger root cert refresh

  vecs-cli failed. Error 11: Possible errors:
LDAP error: Administrative limit exceeded
Win Error: Operation failed with error ERROR_BAD_FORMAT (11)

• Publishing the missing certificates using the below command fails:
  • /usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id ################ --login User@vSphere_Domain_Name.local --password <PASSWORD> --outcert /tmp/certificate _name.cer
  • /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /var/core/certificate_name.cer --login administrator

Followed the below steps to fix the issue:

• Deleted the unnecessary certificates from the VECS and VMDIR by following the steps in KB : Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)
• Unpublished the intermediate and root certificate from the VMDIR:  
  • /usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id ################ --login User@vSphere_Domain_Name.local --password <PASSWORD> --outcert /tmp/certificate _name.cer
  • /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /tmp/certificate _name.cer
• Published both the certificates :
  • /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /var/core/certificate_name.cer --login administrator
• Force a refresh of VECS: /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh