vCenter Server LDAPS authentication fails with "java.security.SignatureException: Bad signature length" due to Certificate Pinning

Products

• VMware vCenter Server

Issue/Introduction

• Users are unable to authenticate to the vCenter Server using their Active Directory credentials.
• Attempts to validate, edit, or reconfigure an existing Active Directory over LDAP (LDAPS) Identity Source within the Single Sign-On (SSO) configuration fail.
• Reviewing the vCenter Server SSO log (/var/log/vmware/sso/ssoAdminServer.log) reveals that the LDAPS connection is failing during the TLS handshake, specifically throwing a java.security.SignatureException error:

ERROR ssoAdminServer[...] [...] Server SSL certificate verification failed for [Subject: ] [SHA1 Fingerprint: ...].: Server SSL certificate not a trusted certificate nor signed by a trusted certificate...  Issuer: CN=IssuingCA-01, DC=domain, DC=local...Trusted certificate 0: [...  SubjectAlternativeName [  DNSName: ###.###.###...java.security.SignatureException: Bad signature length: got 512 but was expecting 256ERROR ssoAdminServer[...] [...] [com.vmware.identity.interop.ldap.OpenLdapClientLibrary] Server SSL certificate not trusted: Subject ()ERROR ssoAdminServer[...] [...] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldaps://###.###.###:636] because [com.vmware.identity.interop.ldap.ServerDownLdapException] with reason [Can't contact LDAP server]

Environment

^{vCenter Server 7.x and 8.x}

^{AD over LDAPS}

Cause

This issue occurs when an individual Domain Controller's leaf certificate is uploaded to vCenter as the "Trusted Certificate" (a practice known as certificate pinning), rather than uploading the organization's complete Certificate Authority (CA) chain.

In a highly available Active Directory environment, vCenter may query multiple Domain Controllers. If the pinned certificate belongs to a Domain Controller utilizing a 2048-bit RSA key, but vCenter attempts to connect to a different Domain Controller utilizing a 4096-bit RSA key, the cryptographic key lengths will not mathematically align.

Comparing a 4096-bit signature payload (512 bytes) against a pinned 2048-bit trust anchor (256 bytes) is a violation of X.509 RFC standards. The Java runtime environment inside vCenter treats this mathematical discrepancy as a potential Man-in-the-Middle (MITM) downgrade attack and intentionally drops the connection to maintain strict security boundaries.

Resolution

To permanently resolve this issue and ensure proper cryptographic trust across all Active Directory nodes, vCenter must be configured to trust the full Active Directory Certificate Authority (CA) chain instead of an individual Domain Controller's server certificate.

Step 1: Obtain the CA Certificates Export the public certificates for your domain's Root CA and Issuing CA (Intermediate CA) from your Active Directory Certificate Services.

• The certificates must be exported in Base-64 encoded X.509 format (.cer, .crt, or .pem).
• Note: Do not export or use the individual server/leaf certificates from the Domain Controllers themselves.

Step 2: Create the Certificate Chain You must manually combine the CA certificates into a single PEM file for vCenter to process them correctly.

1. Open a plain text editor (such as Notepad or Notepad++).
2. Paste the text of the Issuing (Intermediate) CA certificate first. Ensure you include the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----
3. Paste the text of the Root CA certificate directly underneath it, with no blank lines in between.
4. Save the combined file as ca-chain.pem.

Example of a valid chain format:

-----BEGIN CERTIFICATE-----<Base64 Encoded Issuing CA Certificate Data>-----END CERTIFICATE----------BEGIN CERTIFICATE-----<Base64 Encoded Root CA Certificate Data>-----END CERTIFICATE-----

Step 3: Update the vCenter SSO Identity Source

1. Log in to the vSphere Client as an administrator with SSO privileges (e.g., [email protected]).
2. Navigate to Administration > Single Sign On > Configuration.
3. Select the Identity Providers tab, then click Identity Sources.
4. Select the affected LDAPS Identity Source and click Edit.
5. Next to the SSL certificate field, click Browse (or Choose File) and upload the newly created ca-chain.pem
6. Save the configuration.

By supplying the full CA trust chain, vCenter can now dynamically validate the certificate of any Domain Controller in the environment, regardless of individual RSA key sizes, restoring LDAPS authentication.

Additional Information

^{For further details regarding LDAP with SSL requirements in vCenter Server, please refer to: Broadcom KB 2041378}