Error "vCenterUnreachable" with 503 Service Unavailable while running Tanzu CLI Command

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• When executing the "tanzu management-cluster get" command in Tanzu Kubernetes Grid multicloud (TKGm), the cluster status returns a vCenterUnreachable error.
• Logs will display the following message:

unable to create tags manager: POST https://<vCenter-IP>/rest/com/vmware/cis/session: 503 Service Unavailable

• Direct network connectivity to the vCenter API endpoint is functional 

#curl -k -v -X POST https://<vcenter-ip>/rest/com/vmware/cis/session returns a "401 Unauthorized", confirming the network path is open).

• The issue may appear isolated to specific user accounts or specific management clusters, while others operating on different clusters function normally.

Environment

• Tanzu Kubernetes Grid multicloud (TKGm) : 1.6.1
• VMware vCenter Server : 8.0 U1

 

Cause

• This issue occurs when the vCenter vAPI endpoint service (vmware-vapi-endpoint) exhausts its API thread pool or reaches its rate limit for specific automated CAPV (Cluster API Provider vSphere) workflows.
• Because the thread pool is locked up, vCenter actively rejects new session creation requests from the management cluster's controllers with a "503 Service Unavailable" error, leading to the vCenterUnreachable status.

Resolution

To resolve this issue, you must clear the exhausted thread pools on the vCenter server and force the Tanzu controllers to initiate fresh API sessions.

Step 1: Restart the vCenter vAPI Endpoint Service

     
1. Log in to the vCenter Server Appliance (vCSA) via SSH as root and restart the endpoint service.

2. Stop the service:

    #service-control --stop vmware-vapi-endpoint

3. Start the service:

   #service-control --start vmware-vapi-endpoint

4. Wait 3 to 5 minutes for the service to fully initialize before proceeding to the next step.

Step 2: Restart Tanzu CAPv Controller Pods

1. SSH from the jumpbox/bootstrap machine where the Tanzu CLI is installed

2. Set your kubectl context to the affected management cluster (replace <management-cluster-name> with your actual cluster name):

  # kubectl config use-context <management-cluster-name>-admin@<management-cluster-name>

3. Restart the CAPv controller deployment:

 # kubectl rollout restart deployment/capv-controller-manager -n capv-system


Step 3: Verify the Resolution

     1. Once the pods have successfully restarted and the vAPI endpoint is fully online, verify the cluster status:

        #tanzu management-cluster get
   
    2. The vCenterUnreachable error should no longer be present, and the cluster status should display correctly.