Load balancer status unknown after upgrade to VCF 5.2 version
search cancel

Load balancer status unknown after upgrade to VCF 5.2 version

book

Article ID: 394959

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • After upgrading the VCF to version 5.2, customer observed all the LB were in unknown state and traffic was impacted towards the virtual pool members.

>>>>The Certificate import flow is:
           1) Import Certificate in UI or Policy API.
          2) In policy layer, do certificate CA chain verification and store it in Policy layer. This Certificate is marked by policy path, such as /infra/certificates/<ca_cert_name>
          3) After Certificate is stored in Policy layer, Manager layer will read Certificate content in Policy layer and do certificate CA chain verification and store it in Manager layer. This Certificate is marked by UUID,  such as :7####-3##-4###-8###-a4######

>>>>LB component workflow:
          1) Create LB in Policy layer with Certificate : km##s_c#_c##t
          2) Manager layer will read content in Policy layer and create LB component in Manager. For Certificate, it will find the relative UUID based on Certificate path km##s_c#_c##t . Then 7####-3##-4###-8###-a4###### is found out. In fact, no relative certificate in Manager layer. It cannot find out the certificate.

  • Edge Logs will show below :
     /var/log/syslog
2025-03-19T20:22:15.569Z a###### NSX 373816 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="WARN"] [93####-###-a1##-#######] cfg: can't find load_balancer_certificate with uuid: 7#####-3###-41###-8##a4#####
2025-03-19T20:22:15.827Z al#####.pcsalp.local NSX 373816 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="INFO"] [93####-######-a1##-#######] cfg: aborting config generation,will wait for further config updates
  • NSX Manager logs will show below:
    /var/log/proton/nsxapi.log
2025-03-19T14:44:53.759Z WARN providerTaskExecutor-0 CertBaseProvider 70530 POLICY [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Object /infra/certificates/km##s_##_###has realization path /infra/realized-state/enforcement-points/default/certificates/km##s_c##_c## and realization object id 7####-3##-4###-8###-a#######, but the realized object cannot be found on MP.

2025-03-19T14:44:53.762Z ERROR providerTaskExecutor-0 CertificateUtil 70530 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP2076" level="ERROR" subcomp="manager"] Certificate chain validation failed. Make sure a valid chain is provided in order leaf,intermediate,root certificate.
2025-03-19T14:44:53.763Z WARN providerTaskExecutor-0 CertBaseProvider 70530 POLICY [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Error in processResourceChange for /infra/certificates/km##s_c#_c##t
com.vmware.nsx.management.truststore.exceptions.InvalidDataException: null

To validate certificate realize in MP plane or not :

<ca_cert_name> = km##s_c##_c## 

From any API Client (example : Postman) ->> GET /policy/api/v1/infra/realized-state/realized-entities?intent_path=/infra/certificates/<ca_cert_name>  >>> it has the realization error: "state": "ERROR"

 

Environment

VMware NSX

Cause

In this case customer had a successful LB configuration (old configuration without invalid certificate) in NSX-T 3.2 version before upgrade, when customer imported invalid certificate, as per the bug the certificate was created in policy and in manager layer when it tried to validate the certificate chain, the certificate failed to be created in manager layer. So in this situation this invalid certificate UUID stayed in NSX database and also edge nestdb configuration. NSX UI did not throw any error for this invalid certificate. The LB engine was running with the old configuration. It was not using the invalid certificate. That is why monitoring was successful and also LB configuration was up and running before upgrade. After upgrade to NSX-t 4.2 version, edge got restarted and that edge didn't hold any old configuration so it followed the steps to push LB configuration when it found the certificate invalid it stuck in LB unknown state.

Resolution

This issue is resolved in VMware NSX 4.2, available at Broadcom downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:
The workaround is to update the LB monitor and remove the invalid CA from LB monitor via Policy API or UI then LB will be recovered. 

Navigate to  >> Networking > Load Balancing > Monitor > Trusted CA certificate

Powered by Wolken Software