Backups and startup of deployment fail after updating the CA certificate of vCenter

Products

• VMware Integrated OpenStack

Issue/Introduction

Several symptoms may appear due to this:

• Backups will fail.
• Startup of VMware Integrated OpenStack may fail due to failure to mount the persistent volume containers for the database, RabbitMQ, or Nova console logs.

In the following logs we'll see untrusted certificate errors to vCenter:

• The kube-controller-manager log in the kube-system namespace.
• The manager log in the cluster-api-system namespace.

Environment

7.x

Cause

Even if the certificate is already trusted and present in the vCenter resource and by OpenStack services two custom resources must be manually updated. These custom resources are for the management plane and not OpenStack itself.

Resolution

We need to update specific custom resources with the new management vCenter certificate. We will need the CA certificate specifically.

1. Replace the CA certificate on the VIO manager in the /etc/kubernetes/pki/ directory. You can find the current point as ca-file in /etc/kubernetes/cloud-config/cloud-config.yaml. For example this would be: /etc/kubernetes/pki/sc2-10-186.0-214.vmware.com.crt.
2. Also copy the contents of the new CA certificate. Place the values in the cluster CR. Edit the cluster CR with: osctl edit cluster. Under vsphereCertificate: | replace the old CA certificate with the new one.
3. The kube-controller-manager pods must be restarted as a final step. To do this we need to perform two actions:

mv /etc/kubernetes/manifests/kube-controller-manager.yaml /tmp
mv /tmp/kube-controller-manager.yaml /etc/kubernetes/manifests/