Apply changes failing with error "The URL https://vcenter-fqdn/sdk/vimService does not have a valid SSL certificate."

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

When running apply changes on a deployment it results in an error like shown below:

 L Error: Unknown CPI error 'Unknown' with message 'The URL https://vcenter-fqdn/sdk/vimService does not have a valid SSL certificate. You may use the manifest property 'vcenter.connection_options.ca_cert' in the global CPI config to set a trusted CA CERTIFICATE, PEM-encoded. Exception: SSL_read: shutdown while in init' in 'delete_vm' CPI method (CPI request ID: 'cpi-122156')

Environment

Bosh with Vsphere and NSX

Cause

This error may occur for different reasons. The following Kbs have more information https://knowledge.broadcom.com/external/article/293395/vsphere-cpi-ssl-errors.html  or https://knowledge.broadcom.com/external/article/373332/error-unknown-cpi-error-unknown-with-mes.html 

Resolution

In this KB the reason for the error was incorrect certificate set at Bosh Director tile -> vCenter Config --> NSX CA Certificate. To resolve the issue make sure the certificate returned by NSX matches the certificate set at Bosh. You can use the following commands to check the certificate that is being returned by nsx fqdn:

openssl s_client -showcerts -connect nsx-fqdn:443
openssl s_client -showcerts -connect vcenter-fqdn:443

When rotating the certificate on NSX side, it has to be updated in Bosh director aswell to ensure communication is intact. 

Additional Information

KB How to rotate the NSX Manager CA Certificate for TKGI https://knowledge.broadcom.com/external/article/308609/how-to-update-nsx-manager-ca-certificate.html