vsphere_CPI error "SSL peer certificate or SSH remote key was not OK" in set_vm_metadata CPI method

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Following the creation or regeneration of the NSX API server self-signed certificate, the Bosh Director vSphere CPI could fail during VM creation with the error below:

Error: Unknown CPI error 'Unknown' with message 'SSL peer certificate or SSH remote key was not OK' in 'set_vm_metadata' CPI method (CPI request ID: 'cpi-151736')

This error indicates some problem with the self-signed certificate. Even though basic curl test using the cert as CA may result in a successful SSL connection, the CPI would still fail with the specific error.

It appears that the error is caused by a certificate that was created using a Certificate Signing Request (CSR) that doesn't follow the example in the NSX Management TLS Certificate documentation

As an example of a CSR that is not according to the doc, here is one that, when used to create the certificate, resulted in the said CPI error:

    [ req ]
    default_bits = 2048
    distinguished_name = req_distinguished_name
    req_extensions = req_ext
    prompt = no
    [ req_distinguished_name ]
    countryName = US
    stateOrProvinceName = California
    localityName = CA
    organizationName = NSX
    commonName = nsxt.domain.com
    [ req_ext ]
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = 10.###.###.11
    DNS.2 = 10.###.###.12
    DNS.3 = 10.###.###.13
    DNS.4 = 10.###.###.9

The correct CSR (with the above values) should look like the following, which follows the doc example:

    [ req ]
    default_bits = 2048
    default_md = sha256
    prompt = no
    distinguished_name = req_distinguished_name
    x509_extensions = SAN

    [ req_distinguished_name ]
    countryName = US
    stateOrProvinceName = California
    localityName = CA
    organizationName = NSX
    commonName = nsxt.domain.com

    [ SAN ]
    subjectAltName = IP:10.###.###.11,IP:10.###.###.12,IP:10.###.###.13,IP:10.###.###.9

Environment

Product Version: 1.9+

Resolution

The resolution would be to regenerate a new self-signed certificate by ensuring that the CSR closely follows the example in the NSX Management TLS Certificate documentation

Once the new self-signed certificate has been regenerated accordingly, then import it to NSX via the NSX UI

Then, register that cert with the NSX-T API Server

Afterwards, "Apply Changes" can be started.