Failed to rotate vSphere with Tanzu Supervisor certificate when a HTTPS Proxy is configured

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:

• Failed to execute certmgr binary from "Replace vSphere with Tanzu Supervisor Certificates"

  time="YYYY-MM-DDThh:mm:ssZ" level=info msg="[/root/certmgr certificates rotate]"
  time="YYYY-MM-DDThh:mm:ssZ" level=error msg="STS Issue HOK request failedPost \"https://<vCenter Server PNID>/sts/STSService/vsphere.local\": Forbidden"
  time="YYYY-MM-DDThh:mm:ssZ" level=error msg="Failed to get STS token: Post \"https://<vCenter Server PNID>/sts/STSService/vsphere.local\": Forbidden"
  time="YYYY-MM-DDThh:mm:ssZ" level=fatal msg="Failed to obtain VC client: %sfailed to login to VC: Post \"https://<vCenter Server PNID>/sts/STSService/vsphere.local\": Forbidden"

  Note: Messages may vary depending on your environment.

• When you run the env command in the shell executing the certmgr binary, you can see that environment variables such as http_proxy or https_proxy are defined.
  

  root@vcsa [ ~ ]# env | grep -i proxy 
  https_proxy=http://proxy_ip_address:port/ 
  http_proxy=http://proxy_ip_address:port/ 
  no_proxy=localhost, 127.0.0.1 
  NO_PROXY=localhost, 127.0.0.1 
  ...

Environment

vSphere with Tanzu, vSphere Supervisor

Cause

The certmgr script is affected by the HTTPS proxy settings on the vCenter Server.
A proxy can cause certmgr to fail access to the vCenter Server itself.

Resolution

Workaround 1: Configure the vCenter Server to not use a proxy when accessing itself
See resolution of "How to configure Proxy Settings for vCenter Server"

Workaround 2: Set NO_PROXY when excuting certmgr

no_proxy="localhost, 127.0.0.1, <vCenter Server PNID>" ./certmgr certificates rotate

or

NO_PROXY="localhost, 127.0.0.1, <vCenter Server PNID>" ./certmgr certificates rotate