"RefreshSupervisorClusterStatusActivity | [Activity Execution] Unable to create a VcRestClient for VC" message in vcloud-container-debug.log on Cloud Director Cells after performing a vCenter Server Reconnect

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• In the Cloud Director Provider UI under Resources > Cloud Resources > Provider VDCs the Provider VDC does not show the blue Kubernetes icon which would show that it supports Kubernetes.
• After performing a Reconnect on a vCenter from Cloud Director the Provider VDC stops showing the blue Kubernetes icon which would show that it supports Kubernetes.
• The /opt/vmware/vcloud-director/logs/vcloud-container-debug.log on one of the Cloud Director Cells shows a message of the following form:

| DEBUG    | Backend-activity-pool-246228 | RefreshSupervisorClusterStatusActivity | [Activity Execution] Unable to create a VcRestClient for VC: <VC_UUID>.  - Handle: urn:uuid:<HANDLE_UUID>, Current Phase: RefreshSupervisorClusterStatus
Activity$RefreshKubernetesStatusPhase | activity=(com.vmware.vcloud.vimproxy.internal.impl.PCEventProcessingActivity,urn:uuid:<ACTIVITY_UUID>) activity=(com.vmware.ssdc.backend.RefreshSupervisorClusterStatusActivity,urn:uuid:<ACTIVITY_UUID>)
com.vmware.vapi.client.exception.SslException: https://192.168.1.10:443/api invocation failed with "javax.net.ssl.SSLPeerUnverifiedException: Certificate for <192.168.1.10> doesn't match any of the subject alternative names: [vcenter.example.com]"
        at com.vmware.vapi.internal.protocol.client.rpc.http.HttpClient.send(HttpClient.java:212)
        at com.vmware.vapi.internal.protocol.client.msg.json.JsonApiProvider.sendRequest(JsonApiProvider.java:127)
        at com.vmware.vapi.internal.protocol.client.msg.json.JsonApiProvider.invoke(JsonApiProvider.java:314)
        at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:225)
        at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:206)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethodAsync(Stub.java:170)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:138)
        at com.vmware.cis.SessionStub.create(SessionStub.java:44)
        at com.vmware.cis.SessionStub.create(SessionStub.java:37)
        at com.vmware.vcloud.vcenter.VcVApiClientImpl.createStub(VcVApiClientImpl.java:103)
        at com.vmware.vcloud.vcenter.VcVApiClientImpl.createStub(VcVApiClientImpl.java:67)
        at com.vmware.vcloud.vcenter.VCRestClientImpl.createStub(VCRestClientImpl.java:51)
        at com.vmware.ssdc.backend.RefreshSupervisorClusterStatusActivity$RefreshKubernetesStatusPhase.invoke(RefreshSupervisorClusterStatusActivity.java:104)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <192.168.1.10> doesn't match any of the subject alternative names: [vcenter.example.com]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:467)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.vmware.vapi.internal.protocol.client.rpc.http.HttpClient.invoke(HttpClient.java:165)
        at com.vmware.vapi.internal.protocol.client.rpc.http.HttpClient.send(HttpClient.java:189)
        ... 19 more

Environment

VMware Cloud Director 10.x

Cause

Cloud Director will attempt to verify the Kubernetes status for an ESXi cluster in vSphere which has been enabled as a Supervisor Cluster.
In order to do this it will attempt to communicate with the vCenter Server's API.
If the certificate presented by vCenter does not have a Subject Alternate Name matching the IP or FQDN that Cloud Director uses to connect then the API call will fail.
The message will be present in the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log on one of the Cloud Director Cells:

javax.net.ssl.SSLPeerUnverifiedException: Certificate for <192.168.1.10> doesn't match any of the subject alternative names: [vcenter.example.com]

When this occurs Cloud Director cannot confirm the Kubernetes status for an ESXi cluster and will not set the Provider VDC with the blue Kubernetes icon which would show that it supports Kubernetes.

Resolution

Examine the vCenter Server's certificate and confirm the details of the Subject Alternate Name field.
Ensure that Cloud Director is configured to connect to the vCenter Server using an IP or FQDN that is listed as a Subject Alternate Name.
To change this we can follow the documentation here, Modify vCenter Server Settings.
Example steps would be as follows:

1. Open the Cloud Director Provider UI as a System Administrator and navigate to Resources > Infrastructure Resources > vCenter Server Instances.
2. Click on the name of the vCenter to open its details.
3. Under General > vCenter Server Info click Edit.
4. Change the Url field to a valid IP or FQDN for this vCenter that is listed as a Subject Alternate Name in the vCenter Server's certificate and save the changes.
5. Next perform a Reconnect in Cloud Director on this vCenter.
6. After the reconnect operation completes wait a short period for Cloud Director to resync the vCenter inventory.
7. Navigate to Resources > Cloud Resources > Provider VDCs and confirm that the Provider VDC now shows the blue Kubernetes icon.