"certmgr" tool for supervisor certificate rotation from vCenter fails with error "Proxy Authentication Required"

Products

• VMware vSphere ESXi

Issue/Introduction

Attempting to run the certmgr tool on the vCenter Server by following steps in KB Replace vSphere with Tanzu / vSphere Kubernetes Service Supervisor Certificates to list or rotate certificates for a Supervisor Cluster fails. The output displays a fatal error regarding proxy authentication

# ./certmgr certificates list -c domain-c#:########-####-####-####-############
time="[YYYY-MM-DDTHH:MM]" level=info msg="[/root/certmgr certificates list -c domain-c#:########-####-####-####-############]"
time="[YYYY-MM-DDTHH:MM]" level=error msg="STS Issue HOK request failedPost \"https://<vcenter-fqdn>/sts/STSService/vsphere.local\": Proxy Authentication Required"
time="[YYYY-MM-DDTHH:MM]" level=error msg="Failed to get STS token: Post \"https://<vcenter-fqdn>/sts/STSService/vsphere.local\": Proxy Authentication Required"
time="[YYYY-MM-DDTHH:MM]" level=fatal msg="Error creating VC client: failed to login to VC: Post \"https://<vcenter-fqdn>/sts/STSService/vsphere.local\": Proxy Authentication Required"

Environment

vSphere with Tanzu

VMware vCenter Server

Cause

The certmgr tool attempts to communicate with the vCenter Server. This failure occurs because the vCenter's system-level proxy configuration is intercepting internal traffic.

Specifically, the NO_PROXY (or no_proxy) variable in the proxy configuration file /etc/sysconfig/proxy does not include the vCenter FQDN, or the format of the variable is incorrect, causing the request to be routed through an external proxy that requires authentication.

Resolution

To resolve this issue, you must ensure that internal vCenter traffic bypasses the proxy by correctly configuring the NO_PROXY settings.

1. Log in to the vCenter Server Appliance via SSH as root.

2. Backup the existing proxy configuration file: cp /etc/sysconfig/proxy /etc/sysconfig/proxy.bak

3. Edit the proxy configuration file: vi /etc/sysconfig/proxy

4. Update the NO_PROXY variable to ensure it includes the following:

   • The FQDN and IP address of the vCenter.

   • The CIDR range for the vCenter and ESXi host management network 

   • Localhost (127.0.0.1, localhost).

   • Right domain name (ensure right usage of '*' in the domain name)

5. Save the file and exit (press ESC, type :wq!, press Enter)

6. Reboot the vCenter to ensure the environment variables are fully applied across all processes.
   Refer How to configure proxy settings for vCenter Server and Troubleshooting vCenter Server Proxy Configuration

7. Retry to run the certmgr tool to list or rotate certificates for a Supervisor Cluster