Supervisor Cluster Upgrade Failure from 1.27 to 1.28 with "System Error Occurred on Master Node

Products

• VMware vSphere with Tanzu

Issue/Introduction

^{During the upgrade of a Supervisor Cluster from vSphere with Tanzu Kubernetes Grid (TKG) version 1.27 to 1.28, the upgrade fails, and an error message is encountered stating "System error occurred on master node with identifier."}

^{Error adding certificate extensions from config section server_cert_ext #####:error:#####:X509 V3 routines:v2i_AUTHORITY_KEYID:unable to get issuer keyid:crypto/x509/v3_akid.c:177: ######:error:######:X509 V3 routines:X509V3_EXT_nconf_int:error in extension:crypto/x509/v3_conf.c:48:section=server_cert_ext, name=authorityKeyIdentifier, value=keyid}

^{Additionally, the pod description for pinniped-concierge-kube-cert-agent shows the following error:}

^{Back-off pulling image "localhost:5000/vmware.io/pinniped:v0.22.0_vmware.1"}

 

 

Environment

VMware vSphere with Tanzu

Cause

The issue occurs when a ControlPlane VM does not have the necessary issuer information configured in the OpenSSL configuration. This results in the failure to retrieve the keyid from the CA certificate, leading to the upgrade failure.

Resolution

1. Delete the Affected ControlPlane VM: Follow the steps outlined in the Broadcom Knowledge Base article for troubleshooting vSphere with Tanzu TKGs:
   Troubleshooting vSphere with Tanzu TKGS

2. Deploy a New ControlPlane VM: After deleting the affected VM, automatically a new ControlPlane VM will be deployed wait till IP address is assigned.

3. SSH into the New ControlPlane VM: Use SSH to log into the newly deployed ControlPlane VM.

4. Update the OpenSSL Configuration: Edit the /etc/vmware/wcp/openssl.conf file on the newly deployed ControlPlane VM and append the following line under the relevant section:

    

   authorityKeyIdentifier = keyid,issuer

5. Repeat for All Newly Deployed ControlPlane VMs: If there are multiple ControlPlane VMs in the cluster, repeat the above step for all newly deployed VMs to ensure the issuer is correctly configured.

6. This configuration change allows OpenSSL to fall back to using the issuer information if the keyid cannot be retrieved from the CA certificate, resolving the issue and allowing the upgrade to complete successfully.

Additional Information

This issue has been identified and fixed in the upcoming release of vCenter Server 8.0 U3. However, the release timelines are not yet published.