Kubernetes Policies in VCD 10.2 with vCenter 7.0 and later, Tanzu are non-functional

Products

• VMware Cloud Director

Issue/Introduction

Cloud Director unable to reach vSphere for Kubernetes, you see message similar to:

Could not connect to vSphere for Kubernetes infrastructure

Environment

VMware Cloud Director 10.x

Cause

This issues is caused due to the certificate structure of Tanzu Kubernetes in vCenter, The certificate(s) of the Supervisor Cluster is not automatically trusted by VCD. Calls made to the Supervisor Cluster by VCD fail due to SSL errors.

Resolution

To resolve this issue:

Note: This currently applies to VMware Cloud Director 10.2 with vCenter 7.0 and later environments with vSphere with Tanzu feature enabled.

The Supervisor Cluster certificates can be manually trusted and accepted by VCD.
Follow the below steps to manually set CA certificates to trusted are:

1. Login to the vCenter UI.
2. Navigate to Menu > Developer Center > API Explorer > GET /api/vcenter/namespace-management/clusters/{cluster}
3. In the value for the cluster parameter, input the moref of the vCenter Cluster containing the Kubernetes Supervisor Cluster. Note: The moref is located in the URL of the H5 UI when clicking on the cluster. It is in the format: domain-xx
4. Click execute to execute the API:

Copy the curl command displayed in the VC UI into a linux shell:

Example:

curl -X GET 'https://<URL>/api/vcenter/namespace-management/clusters/domain-c8' -H 'vmware-api-session-id: <API_SESSION_UUID>'

    

Copy and amend this curl command by adding "|jq '.tls_management_endpoint_certificate'|sed 's/\\n/\n/g'" to a linux shell.
Optionally, you may add the curl option "--insecure" if vCenter's endpoint is associated with a self-signed certificate (non production use case).

           
          Output:  

curl -X GET 'https://<URL>/api/vcenter/namespace-management/clusters/domain-c8' -H 'vmware-api-session-id: <API_SESSION_UUID>' | jq '.tls_management_endpoint_certificate'|sed 's/\\n/\n/g'
  
 Executing this command should output something like:
   
   "-----BEGIN CERTIFICATE-----
MIIC5zCCAc+gxxxxxxx1234xxxkiG9w0BAQsFADAVMRMwEQYDVQQDEw
...

...

5ByXjSEd5ZBC9JkSfSjGbP8TVtLQ9Sal3Bzb
-----END CERTIFICATE-----
"

     Note: You may need to install the following package in your linux distribution: "curl", "sed", "jq" to execute the above command.

5. Copy this output content, without the double-quotes into a file with suffix ".pem".
6. Open a browser to vCD provider UI, and navigate to tab "Administration", and select menu "Trusted Certificates"
7. Click on "IMPORT", then click on "SELECT FILE", and input the file created at step 5 above.
8. Verify the details displayed and click the "IMPORT" button.

Note: Optionally, If you're using vCenter Server version before 7u1c you'll also need to perform the following additional steps to trust the workload management certificate.

To do so, repeat steps 4 to 6 but adjust the curl command as follow, example:

curl -X GET 'https://<URL>/api/vcenter/namespace-management/clusters/domain-c8' -H 'vmware-api-session-id: <API_SESSION_UUID>' | jq '.tls_endpoint_certificate'|sed 's/\\n/\n/g'

(as you can see this replaces the string "tls_management_endpoint_certificate" with "tls_endpoint_certificate")

9. In the VCD provider UI, navigate to the Tanzu Kubernetes vCenter.
10. Go to Infrastructure Resources > vCenter Server Instances > (specific vCenter). Click on the vCenter Server.
11. Click "REFRESH" to refresh the vCenter Server.
12. Wait a short period of time (about a minute) and confirm that the policy was generated (or refreshed) by navigating to vCD provider UI "Monitor > Events".

Note: Observe an audit event such as:

• "Provider VDC Kubernetes Policy '[name-of-PVDC]-[name-of-host-cluster]-KubernetesPolicy' (<POLICY_UUID>) modification"
• Then navigate to "Provider VDCs", select your PVDC, then select "Policies / Kubernetes", and confirm that the corresponding policy was auto generated.