Replacing vCenter 3rd Party CA certificates from the SDDC Manager fails with error "Unexpected data detected in stream"

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

The certificate replacement task fails with errors similar to below in the SDDC UI:

500 Internal Server Error: "{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["Unexpected data detected in stream"],"default_message":"Exception found (Unexpected data detected in stream)","id":"com.vmware.certificatemanagement.error"}]}}" Error Message: Failed to replace certificate for myvCenter.acme.com due to: 500 Internal Server Error: "{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["Unexpected data detected in stream"],"default_message":"Exception found (Unexpected data detected in stream)","id":"com.vmware.certificatemanagement.error"}]}}" Remediation Message: Reference Token:

Cause:

SDDC Manager: /var/log/vmware/vcf/operationsmanager/operationsmanager.log:

2022-05-04T10:08:40.659+0000 DEBUG [vcf_om,5feb9d47bd8f4c1b,3338] [c.v.v.c.s.f.i.CertificateOperationsFacadeImpl,http-nio-127.0.0.1-7300-exec-6] DomainCertificateOperation: {"workflowId":"3e15e9e9-d5fb-4fb4-aa58-9d824cdfa54a","domainName":"EUC-WLD","operationType":"REPLACE_CERTIFICATE","operationStatus":"*****","

resourceCertificateOperations":[{"resource":{"hostName":"<vCenter FQDN>","resourceType":"vcenter","master":false},"result":{"status

":"FAILED","message":"{\"code\":\"CERTIFICATE_REPLACEMENT_FAILED_WITH_ERROR\",\"args\":[\"*****\",\"500 Internal Server Error: [{\\\"type\\\":\\\"com.vmware.

vapi.std.errors.error\\\",\\\"value\\\":{\\\"error_type\\\":\\\"ERROR\\\",\\\"messages\\\":[{\\\"args\\\":[\\\"Unexpected data detected in stream\\\"],\\\"de

fault_message\\\":\\\"Exception found (Unexpected data detected in stream)\\\",\\\"id\\\":\\\"com.vmware.certificatemanagement.error\\\"}]}}]\"]}"},"creation Timestamp":1651658707878,"updateTimestamp":1651658860269}]}

The upload of the myDomain.tar.gz file may well succeed but the implementation of the certificate will fail within a few minutes.

Environment

VMware Cloud Foundation 4.x

Cause

This issue can be caused by extraneous words/strings/characters in the rootca chain.

For example, when reviewing the Root CA certificate chain in Notepad you see the following:

The text in RED is sometimes included in certificate chains as a descriptor and does not affect the reading of the cert in cryptoshell.

However, such text will cause the certificate replacement to fail on the vCenter side.

NOTE: There may be other forms of text, numbers, etc...

Resolution

Edit the certificate chain and remove the extra lines. The chain should look something like below: