VCFA and VIDB 9.0.x and vCenter 8.0.x SSL out of sync
search cancel

VCFA and VIDB 9.0.x and vCenter 8.0.x SSL out of sync

book

Article ID: 428829

calendar_today

Updated On:

Products

VCF Automation

Issue/Introduction

VCFA and VIDB require the vCenter certificate and thumbprint in order to securely communicate.

If the VCFA/VIDB cluster is running on a vSphere 8.0 environment, there is no automatic synchronization of the vCenter certificate when it is rotated.

This causes communication issues between VCFA/VIDB and vCenter, and can cause impact to the running application.

Environment

VCF Automation 9.0 and VCF Identity Broker 9.0 running on vCenter 8.0.x

Cause

VCFA and VIDB (actually the underlying Kubernetes platform) can use vSphere NDC (Non-Destructive Certificate rotation) endpoint to update its copy of the vCenter certificate and thumbprint periodically when available. However, the vCenter NDC is available from vSphere 9.0 onwards. When the vCenter certificate is rotated on vSphere 8.0, the copy maintained by VCFA/VIDB is not updated automatically.

Resolution

Manually patch the VCFA/VIDB with the new certificate and thumbprint

1. Prerequisites

1.1. Access to the Ops LCM (Fleet) appliance using SSH

You must have the admin and root credentials and IP/hostname of the Fleet/LCM appliance running in the VCF instance.

1.2. kubeconfig

To get kubeconfig, use either option below:

  • SSH into one of the cluster nodes and locate the file at /etc/kubernetes/admin.conf
  • Obtain it from LCM locker by SSH to LCM node and run the /var/lib/vrlcm/fetch-kubeconfig.py with the following argument: 
    /var/lib/vrlcm/fetch-kubeconfig.py localhost admin@local lcm_pwd ssh_root_pwd env_id

The env_id can be seen from the Fleet LCM UI. Once you have kubeconfig, you can use it to get a debug container running against one of the nodes using the following technique:

export KUBECONFIG=/path/to/kubeconfig

2. Steps

2.1. Get the new certificate

vcenterCertificate=$(echo | openssl s_client -showcerts -connect ${host}:443 2> /dev/null | sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | base64 | tr -d '\n')

2.2. Extract thumbprint from the new certificate

thumbprint=$(echo $vcenterCertificate | base64 -d | openssl x509 -noout -fingerprint -sha1 | cut -d= -f2)

2.3. Patch the vmsp-platform PackageDeployment with the new certificate and its thumbprint

kubectl patch pd vmsp-platform -n vmsp-platform --type=merge --patch-file /dev/stdin <<EOF
{
  "spec": {
    "values": {
      "provider": {
        "vsphere": {
          "thumbprint": "${thumbprint}",
          "tlsCertificate": "${vcenterCertificate}"
        }
      }
    }
  }
}
EOF

2.4. Patch the platform-trust secret with the new certificate

kubectl patch secret platform-trust -n vmsp-platform --type=merge -p "$(jq -n --arg cert "$vcenterCertificate" '{data: {"ca.crt": $cert}}')"

3. Validation

3.1. Wait for operations to complete

Wait for the above operations to complete and the vmsp-platform PackageDeployment to become ready:

kubectl wait packagedeployment/vmsp-platform -n vmsp-platform --for=condition=Ready

3.2. Restart the vsphere-csi-controller

Restarting this service is non-impacting for any workloads and ensures that the storage system is immediately remediated following the certificate update.

kubectl rollout restart deployments/vsphere-csi-controller -n kube-system

3.3. Check the logs of the vsphere-csi-controller pod

kubectl logs deploy/vsphere-csi-controller -n kube-system -c vsphere-csi-controller

Ensure there are no log messages that indicate a thumbprint mismatch. Below is an example error log:

{"level":"error","time":"2026-02-04T17:50:18.550826357Z","caller":"vsphere/virtualcenter.go:690","msg":"failed to connect to VirtualCenter host: \"vcenter.fqdn\". Err: Post \"https://vcenter.fqdn:443/sdk\": host \"vcenter.fqdn:443\" thumbprint does not match \"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##\"","TraceId":"","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.GetVirtualCenterInstanceForVCenterConfig\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:690\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).Init\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/vanilla/controller.go:216\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:194\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:208\nmain.main\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:98\nruntime.main\n\t/build/mts/release/bora-25031886/compcache/cayman_go/ob-24618579/linux64/src/runtime/proc.go:272"}
{"level":"error","time":"2026-02-04T17:50:18.550938628Z","caller":"vanilla/controller.go:218","msg":"failed to get vCenterInstance for vCenter \"vcenter.fqdn\"err=Post \"https://vcenter.fqdn:443/sdk\": host \"vcenter.fqdn:443\" thumbprint does not match \"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##\"","TraceId":"","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).Init\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/vanilla/controller.go:218\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:194\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:208\nmain.main\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:98\nruntime.main\n\t/build/mts/release/bora-25031886/compcache/cayman_go/ob-24618579/linux64/src/runtime/proc.go:272"}
{"level":"error","time":"2026-02-04T17:50:18.551013386Z","caller":"service/driver.go:195","msg":"failed to init controller. Error: failed to get vCenterInstance for vCenter \"vcenter.fqdn\"err=Post \"https://vcenter.fqdn:443/sdk\": host \"vcenter.fqdn:443\" thumbprint does not match \"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##\"","TraceId":"","TraceId":"","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:195\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:208\nmain.main\n\t/build/mts/release/bora-25031886/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:98\nruntime.main\n\t/build/mts/release/bora-25031886/compcache/cayman_go/ob-24618579/linux64/src/runtime/proc.go:272"}
Powered by Wolken Software