Avi Load Balancer cluster status not available from NSX UI with error "some error has occurred"
search cancel

Avi Load Balancer cluster status not available from NSX UI with error "some error has occurred"

book

Article ID: 427503

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

In VCF 9, NSX, Avi Load Balancer, AKO and other products are orchestrated and deployed through VCF 9 SDDC manager. 

NSX manager may show Avi Load Balancer status as "Unavailable" under "Systems", "Appliances" accompanying an error message of "Some error has occurred". 

This does not impact Avi Load Balancer functionality. However, NSX is unable to manage this Avi cluster. 

In NSX syslog you may find:

2026-01-13T04:21:51.474Z  INFO http-nio-127.0.0.1-7440-exec-460 AlbControllerTokenServiceImpl 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] Generating token for user nsxt-lcm for 1.08 hours for ip <IPv4>
026-01-13T04:21:51.476Z  INFO http-nio-127.0.0.1-7440-exec-460 AlbControllerRestClient 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] AVI API POST https://<IPv4>/api/authtoken is called with Version 30.2.1
2026-01-13T04:21:51.476Z  INFO http-nio-127.0.0.1-7440-exec-460 AlbControllerAuthInterceptor 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] __INIT__ Inside Interceptor..
2026-01-13T04:21:51.476Z  INFO http-nio-127.0.0.1-7440-exec-460 AlbControllerRestUtils 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] Inside authentication session for admin
2026-01-13T04:21:51.476Z  INFO http-nio-127.0.0.1-7440-exec-460 AlbControllerRestUtils 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] Building http client for rest handler
2026-01-13T04:21:51.476Z  INFO http-nio-127.0.0.1-7440-exec-460 AlbControllerRestUtils 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] BuildHttpClient completed
2026-01-13T04:21:51.626Z  INFO http-nio-127.0.0.1-7440-exec-460 AlbControllerRestUtils 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] Authentication session success for admin
2026-01-13T04:21:51.651Z  INFO http-nio-127.0.0.1-7440-exec-460 AlbControllerAuthInterceptor 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] __DONE__ Interceptor
2026-01-13T04:21:51.651Z  INFO http-nio-127.0.0.1-7440-exec-460 AlbControllerConfigurationUtils 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] Error occurred Got response code 500
2026-01-13T04:21:51.651Z  WARN http-nio-127.0.0.1-7440-exec-460 AlbControllerTokenServiceImpl 83508 POLICY [nsx@4413 comp="nsx-manager" level="WARNING" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] Got exception while refreshing token: org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: "{"error":"record not found"}"
2026-01-13T04:21:51.651Z  WARN http-nio-127.0.0.1-7440-exec-460 AlbControllerConfigUtils 83508 POLICY [nsx@4413 comp="nsx-manager" level="WARNING" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] [Clustering id null] Error occurred while getting token for the user nsxt-lcm. Error is org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: "{"error":"record not found"}"
2026-01-13T04:21:51.651Z  INFO http-nio-127.0.0.1-7440-exec-460 PolicyALBControllerDeploymentFacadeImpl 83508 POLICY [nsx@4413 comp="nsx-manager" level="INFO" reqId="<UUID>" subcomp="manager" username="<username>"] [ALB Controller] Error code is 94598 with error msg Some error has occurred.

 

Checking from Avi Load Balancer UI, no nsxt-lcm and/or nsxt-infra-admin users exist. 

Environment

VCF 9

Cause

This is caused by users used for NSX-Avi integration being removed from Avi Load Balancer

Resolution

To resolve this issue, the missing nsxt-lcm and nsxt-infra-admin users need to be recreated from Avi Load Balancer.

nsxt-infra-admin user password needs to match with NSX. The existing password saved in NSX can be fetched by the following method:

  • SSH into a NSX manager node as root
  • Issue command:  
    /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t ALBControllerAdminCreds > /tmp/albcreds.dump
  • View the /tmp/albcreds.dump to find the password
    • Example output in the file:
    • {"Key":
{
  "stringId": "InfraAdminCreds"
}}

,{"Payload":
{
  "managedResource": {
    "displayName": "InfraAdminCreds"
  },
  "password": "<password-here>",
  "username": "nsxt-infra-admin"
}}
    • Recreate the nsxt-infra-admin user in Avi load balancer with the same password

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

  • NSX Manager log bundles
  • Avi Load Balancer manager support bundle
  • Text of any error messages seen in NSX GUI or command lines pertinent to the investigation
  • Handling Log Bundles for offline review with Broadcom support

Collect Support Bundles for Troubleshooting NSX-T
Uploading files to cases on the Broadcom Support Portal
Creating and managing Broadcom support cases

Powered by Wolken Software