Rotating vSphere with Tanzu spherelet certifficate fails with "Failed to sign client CSR" and "Put URL: Forbidden"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Running "certmgr certificates rotate" by following steps provided in KB Replace vSphere with Tanzu / vSphere Kubernetes Service Supervisor Certificates fails with:
  • Results: "Failed to sign client CSR"
  • Error: "PUT "https://##.##.##.##:443/guestFile?[...]" Forbidden"
• A proxy server is configured and enabled in the vCenter Server Appliance Management Interface (VAMI) settings.

Environment

VMware vSphere with Kubernetes
VMware vCenter Server
VMware ESXi

Cause

This issue occurs due to incomplete proxy bypass configurations in the vCenter Server VAMI.

When a proxy is enabled, outgoing network traffic from the vCenter Server is routed through the proxy by default. If the NO_PROXY configuration does not explicitly exclude the ESXi hosts, or more specifically their management network IPs/CIDR, vCenter routes internal API requests directed at the ESXi hosts through the proxy server. This interferes with the expected direct communication and causes the required API requests to fail, ultimately resulting in the failure to sign the client Certificate Signing Request (CSR).

Resolution

To resolve this issue, the IP subnet of the ESXi hosts' management network must be added to the proxy exclusion list (NO_PROXY) within the vCenter Server Appliance Management Interface (VAMI). This ensures that API requests destined for the ESXi hosts bypass the proxy server and route directly.

For instructions on modifying the proxy settings and updating the bypass list in the vCenter Server VAMI, refer to Broadcom Knowledge Base Article 370265: How to configure proxy settings for vCenter Server.

Once the proxy settings are successfully updated to exclude the ESXi management network, rerun the certmgr certificates rotate command.