Supervisor Deployment Stuck in "Configuring" State Due to Untrusted ALB Endpoint Certificate

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Supervisor deployment in a VCF Workload Domain remains stuck in the Configuring state.

• The vCenter Server UI displays the warning: Configured Load Balancer fronting the kubernetes API Server. Timed out waiting for LB service update.
• The wcpsvc.log on vCenter contains the following error: timed out waiting for kube-apiserver-lb-svc Service update. Err: context deadline exceeded
• The NCP pod logs, shows the below error:
  

  YYYY-MM-DDTHH:MM:SS.MS+HH:MM stderr F [ncp GreenThread-39 I] nsx_ujo.common.controller AviSecretController worker 2 failed to sync Bootstrap due to retryable exception: Failed to get Avi auth token: Unexpected error from backend manager (['<Domain>:443', '<Domain>:443', '<Domain>:443', '<Domain>:443']) for PUT policy/api/v1/infra/alb-auth-token: 401 UNAUTHORIZED

• Service accounts nsxt-alb and nsxt-ako are missing from the Avi Controller.

Environment

VMware Cloud Foundation 9.1

vCenter Server 9.1

Cause

The issue occurs because the alb-endpoint on the NSX Manager is registered with an outdated or incorrect certificate. This prevents NSX Manager from authenticating with the Avi Controller to provision the required nsxt-alb and nsxt-ako service accounts. Without this, the required VIPs cannot be created, causing the Supervisor configuration to stall in the "Configuring" state.

Resolution

To resolve this issue, you must force the NSX Manager to re-synchronize the ALB endpoint and recreate the required user accounts.

Step 1: Validate Certificate

1. Identify the current certificate for the ALB Endpoint: 
   

   curl -k -u 'admin:<password>' -X GET https://<NSX-Mgr>/policy/api/v1/infra/sites/default/enforcement-points/alb-endpoint

   
   Example output:

   curl -k -u 'admin:########' --request GET <URL>
   {
     "connection info : {
        "username":"\u000\u000\u000\u000',
        "tenant" : "admin",
        "expires_at" : "",
        "managed_by" : "VCE",
        "status" : "DEACTIVATE PROVIDER",
        "certificate" : "-----BEGIN CERTIFICATE-----
   <CERTIFICATE >-----END CERTIFICATE-----
        "is_default_cert" : false,
        "enforcement _point_address" : "###.##.##.##",
        "resource type" :
        "AviConnectionInfo"
   },
   "auto_enforce" : true,
   "esource_type" ; "EnforcementPoint",
   "id": "alb-endpoint",
   "display_name" : "alb-endpoint",
   "path" : "/infra/sites/default/enforcement-points/alb-endpoint",
   "relative_path" : "alb-endpoint",
   "parent _path" : "/infra/sites/default",
   "remote_path" : "'",
   "unique_id" : "########-####-####-####-############",
   "realization_id" : "########-####-####-####-############",|
   "owner id" : "########-####-####-####-############".
   "marked for_delete" : false,
   "overridden" : false,
   "-system_owned" : false,
   " _protection" : "NOT_PROTECTED",
   "_create_time" : ##########
   "_crate_user": "<User Account>"
   "_last_modified_time : #########,
   "_last_modified_user" : "system",
   "_revision" : 2
   }

   
   
   
2.  Paste the entire certificate (including the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- lines) into a text file and save it as cert.pem.
3.  Run the OpenSSL command: openssl x509 -in cert.pem -text -noout
4. Validate certificate details to the current certificate on the AVI Load Balancer.
5. If certificate details do not match, proceed to Step 2: Validate Service Accounts

Step 2: Validate Service Accounts

1. SSH to the AVI Controller using the admin account. ssh admin@<avi-controller-ip>
2. Enter the Avi shell by typing: shell
3. Check user. User accounts should not exist. If user accounts do exist, manually clean up the user accounts. 
   

   > show user nsxt-alb
   No user object named 'nsxt-alb' found.
   
   > show user nsxt-ako
   No user object named 'nsxt-ako' found.

Step 3: Delete the alb-endpoint.

1. Backup the current configuration.
   

   curl -k -u 'admin:<password>' -X GET https://<NSX-Mgr>/policy/api/v1/infra/sites/default/enforcement-points/alb-endpoint > /root/alb-endpoint.api

2. Delete the alb-endpoint.
   

   curl -k -X DELETE https://<nsx-mgr-ip>/policy/api/v1/infra/sites/default/enforcement-points/alb-endpoint -u admin:'<password>'

 

Step 4: Wait approximately 5–10 minutes. The alb-endpoint will automatically get recreated. 

Validate the new certificate by following instructions in Step 1: Validate certificate
After the alb-endpoint has been recreated, the Supervisor will be deployed successfully. This can take some time, approximately 10 to 30 minutes.

 

Additional Information

Registration Issues for Avi on NSX Manager Due to Certificate Validation Failure