vCenter 9.1 certificate replacement is consistently failing in VCF Operations with error code: vmsp.cert.replacement.error

Products

• VCF Operations

Issue/Introduction

While attempting to replace vCenter certificate, it consistently fails after approximately 30 minutes with following error message:

Error Code: vmsp.cert.replacement.error
Certificate task REPLACE_CERTIFICATE for <vCenter FQDN> has failed. Error message: Certificate replacement error - VMSP Task status with task ID <task_id> is Failed. Details: ERROR: Internal error occurred. Please retry the operation or contact support if the issue persists. [{*}VCFMS-MACHINE-###{*}]

Environment

VCF Operations 9.1.x

Cause

This condition arises when a user tries to rotate vCenter certificate twice within 30 minutes.

Resolution

To perform manual remediation to recover the cluster, please follow the steps mentioned below:

1.  Log into one of the VCF Service Runtime control plane using vmware-system-user credentials and switched to root using sudo su - command.

2. Executed following command to get the new certificate:

vcenterCertificate=$(
    echo | openssl s_client -showcerts -connect <VCENTER_IP_OR_FQDN>:443 2> /dev/null |
    sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | base64 | tr -d '\n'
)

 

3. Extract thumbprint from the new certificate using following command:

thumbprint=$(
    echo $vcenterCertificate | base64 -d | openssl x509 -noout -fingerprint -sha1 | cut -d= -f2
)

 

4.  Patch the vmsp-platform PackageDeployment with the new certificate and its thumbprint:

kubectl patch pd vmsp-platform -n vmsp-platform --type=merge --patch-file /dev/stdin <<EOF
{
  "spec": {
    "values": {
      "provider": {
        "vsphere": {
          "thumbprint": "${thumbprint}"
          "tlsCertificate": "${vcenterCertificate}"
        }
      }
    }
  }
}
EOF

 

5. Patch the platform-trust secret with the new certificate:

kubectl patch secret platform-trust -n vmsp-platform --type=merge \
  -p "$(jq -n --arg cert "$vcenterCertificate" '{data: {"ca.crt": $cert}}')"

 

6. Wait for the above operations to complete and the vmsp-platform PackageDeployment to become ready:

kubectl wait packagedeployment/vmsp-platform -n vmsp-platform --for=condition=Ready

 

7. Restart the vsphere-csi-controller:

kubectl logs deploy/vsphere-csi-controller -n kube-system -c vsphere-csi-controller

After performing the above-mentioned steps, retry the attempt to replace the certificate and it should succeed.