SSL is out of sync in VCF Automation and Identity Broker 9.0.x running on vCenter 8.0.x

Products

• VCF Operations
• VCF Automation

Issue/Introduction

VCF Automation (VCFA) and Identity Broker (VIDB) require the vCenter certificate and thumbprint in order to securely communicate. If the VCFA/VIDB cluster is running on a vSphere 8.0 environment, there is no automatic synchronization of the vCenter certificate when it is rotated. This causes communication issues between VCFA/VIDB and vCenter, and can cause impact to the running application.

Environment

VCF Automation 9.0, VCF Identity Broker 9.0 running on vCenter server 8.0.x

Cause

VCFA and VIDB (underlying Kubernetes platform) can use vSphere NDC (Non-Destructive Certificate rotation) endpoint to update its copy of the vCenter certificate and thumbprint periodically when available. However, the vCenter NDC is available from vSphere 9.0 onwards. When the vCenter certificate is rotated on vSphere 8.0, the copy maintained by VCFA/VIDB is not updated automatically.

Resolution

Note: Emphasize to secure a snapshot of the Fleet Management node before making any changes.

 

Manually patch the VCFA/VIDB with the new certificate and thumbprint.

 

1. Prerequisites

a. Access to the Ops LCM (Fleet) appliance using SSH
You must have the admin and root credentials and IP/hostname of the Fleet/LCM appliance running in the VCF instance.

 

b. kubeconfig

To get kubeconfig, use the following guide: Fetch Kubeconfig From Locker - env_id can be seen from the Fleet LCM UI.

/var/lib/vrlcm/fetch-kubeconfig.py localhost admin@local lcm_pwd ssh_root_pwd env_id

Once you have kubeconfig, you can use it to get a 'debug' container running against one of the nodes using the following techniques

export KUBECONFIG=/path/to/kubeconfig

 

2. Steps

a. Get the new certificate

vcenterCertificate=$(
    echo | openssl s_client -showcerts -connect ${host}:443 2> /dev/null |
    sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | base64 | tr -d '\n'
)

b. Extract thumbprint from the new certificate

thumbprint=$(
    echo $vcenterCertificate | base64 -d | openssl x509 -noout -fingerprint -sha1 | cut -d= -f2
)
c. Patch the vmsp-platform PackageDeployment with the new certificate and its thumbprint

kubectl patch pd vmsp-platform -n vmsp-platform --type=merge --patch-file /dev/stdin <<EOF
{
  "spec": {
    "values": {
      "provider": {
        "vsphere": {
          "thumbprint": "${thumbprint}"
          "tlsCertificate": "${vcenterCertificate}"
        }
      }
    }
  }
}
EOF

d. Patch the platform-trust secret with the new certificate

kubectl patch secret platform-trust -n vmsp-platform --type=merge \
  -p "$(jq -n --arg cert "$vcenterCertificate" '{data: {"ca.crt": $cert}}')"

 

3. Validation

a. Wait for the above operations to complete and the vmsp-platform PackageDeployment to become ready

kubectl wait packagedeployment/vmsp-platform -n vmsp-platform --for=condition=Ready

b. Restart the vsphere-csi-controller
Restarting this service is non-impacting for any workloads and will ensure that the storage system is immediately remediated following the certificate update.

kubectl rollout restart deployments/vsphere-csi-controller -n kube-system

c. Check the logs of the vsphere-csi-controller pod

kubectl logs deploy/vsphere-csi-controller -n kube-system

Ensure that there are no log messages that indicate a thumbprint mismatch.