Replacing Custom Signed Supervisor VIP Certificate fails with error "Failed to decode PEM-encoded certificate for Kubernetes API Server"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When attempting to update the Custom/Private Supervisor VIP certificate, the process fails with the following symptoms:

• The vSphere Client UI displays the error: Failed to decode PEM-encoded certificate for Kubernetes API Server

• Validation fails specifically when the certificate chain lacks an email address field.

• The replacement fails after a CSR was generated using an external CLI tool rather than the vSphere UI.

Cause

The error occurs when using the GUI to replace a certificate that was originally generated via the CLI because of the fundamental difference in how the private key is handled. When you generate a CSR using the GUI, it keeps the private key securely stored internally and does not export it. This means the GUI only allows you to replace the certificate chain that matches the existing private key.

In contrast, when you generate a CSR with the CLI, the private key is exported separately as a separate file. During certificate replacement using the vSphere client GUI, the private key cannot be provided or imported, as the GUI expects to find the matching key in its internal store. Since the CLI-generated key is external, the GUI detects a key mismatch and throws an error.

Resolution

The resolution follows a tiered approach based on your organizational certificate requirements.

Method 1: Standard UI Replacement (Recommended)
If your Certificate Authority (CA) allows the inclusion of an email address, use the vSphere Client UI to generate the CSR. This ensures the private key remains in the internal VECS store, allowing for a seamless upload. For detailed steps refer: Replace the VIP Certificate from vSphere Client

Method 2: DCLI Lifecycle Management (For CSRs without Email)
If you are unable to include an email address, the alternative approach is to use the DCLI (Datacenter Command-Line Interface) method to generate the CSR and perform certificate replacement entirely from the vCenter Server CLI. This allows you to specify both the certificate and the corresponding private key, bypassing the GUI limitations.

Syntax to generate a CSR without email address:

dcli +show-unreleased-apis com vmware vcenter namespacemanagement certificatemanagement tlscsr create --cluster <cluster> --organization-unit-name <copy from existing vip cert> --common-name <copy from existing vip cert> --key-size <optional keysize> --country <country> --email-address "" --locality <locality> --organization-name <org name> --state-or-province <state>

Steps:

1. 1. SSH to vCenter server and login as root
   2. Enter shell with the command shell
   3. List the clusters with Namespaces:
      1. dcli +show-unreleased-apis com vmware vcenter namespacemanagement clusters list
      2. Enter the local administrator credentials such as [e-mail]
      3. Save the credentials (Optional)
         |---------|------------|---------------------------------------------------------------------------------|-----------------|-------------|
         |cluster  |cluster_name|stats                                                                            |kubernetes_status|config_status|
         |---------|------------|---------------------------------------------------------------------------------|-----------------|-------------|
         |domain-c## |example_name ||--------|----------------|-----------|------------|---------------|------------||READY            |RUNNING      |
         |         |            ||cpu_used|storage_capacity|memory_used|cpu_capacity|memory_capacity|storage_used||                 |             |
         |         |            ||--------|----------------|-----------|------------|---------------|------------||                 |             |
         |         |            ||0       |0               |0          |0           |0              |2##8       ||                 |             |
         |         |            ||--------|----------------|-----------|------------|---------------|------------||                 |             |
         |---------|------------|---------------------------------------------------------------------------------|-----------------|-------------|
         
         CRITICAL: Ensure the Supervisor Cluster status is READY and RUNNING before proceeding. If the cluster is in any other state, contact Broadcom Support to prevent further environment instability.
         
         
   4. Generate a CSR, change the variables or as necessary:
      
      dcli +show-unreleased-apis com vmware vcenter namespacemanagement certificatemanagement tlscsr create --cluster domain-c## --organization-unit-name <example org> --common-name kube-apiserver-domain-c## --key-size 2048 --country <country name> --email-address "" --locality <example locality> --organization-name <example org> --state-or-province <example state>


   5. Copy the above generated CSR, get it signed by the preferred Private Certificate Authority
   6. Update the Signed Private Supervisor VIP Certificate for the Supervisor Cluster:
      1. Using the pem file generated in previous step using the command:

dcli com vmware vcenter namespacemanagement clusters update --cluster example_name --tls-endpoint-certificate "<Certificate-data>"

or 

1. 1. 1. In the vSphere Client, navigate to Workload Management.
      2. Select Supervisors and then select the Supervisor from the list.
      3. Click Configure and select Certificates.
      4. On the vSphere client, navigate to the Workload Platform Management, select Actions > Replace Certificate.
      5. Upload the newly generated certificate file (full chain) and click on Replace Certificate.