Error "new conf not applied, generation mismatch" when running "get load-balancer <UUID> diagnosis"

Products

• VMware NSX

Issue/Introduction

• The traffic is hitting the Virtual Server IP, but is not transferred to any of the pool members with the correct port number.
• Newly added Virtual Servers are not working as expected
• When we run."get load-balancer <LB_UUID> diagnosis" In the NSX-T Edge cli (logged as Admin), we see the following error
  
  Checking
Action : checking system
Result : passed

Action : checking crash
Result : passed

Action : checking daemon status
Result : passed

Action : checking configuration
Result : new conf not applied, generation mismatch


• In the NSX-T Edge log (/var/log/syslog) the following errors are seen
  YYYY-MM-DDTHH:MM:SS.msZ <Edge_name> NSX 36####3 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="FATAL"] [<LB_UUID>] cannot load certificate "/config/vmware/edge/lb/etc/<LB_UUID>/certs/client_ssl_<Certfificate_UUID>.crt": PEM_read_bio_X509() failed (SSL: error:04800066:PEM routines::bad end line)

Environment

VMware NSX

Cause

The Load Balancer/Virtual Server Certificate imported into the NSX was not in the correct format. There were a lot of space lines in the certificate data.

Resolution

Workaround:

   The following workarounds will use this NSX-T Edge log error (/var/log/syslog) as a reference to obtain the LB_UUID and the CertID, which have incorrect formats in the certificate Data.

 

YYYY-MM-DDTHH:MM:SS.msZ <Edge_name> NSX 36####3 LOAD-BALANCER [nsx@6876 comp="nsx-edge" subcomp="lb" s2comp="lb" level="FATAL"] [<LB_UUID>] cannot load certificate "/config/vmware/edge/lb/etc/<LB_UUID>/certs/client_ssl_<Certfificate_UUID>.crt": PEM_read_bio_X509() failed (SSL: error:04800066:PEM routines::bad end line)

• From NSX-T Edge Root access:
  
  

  • SSH in the NSX-T Edge (Where the T1 Load-balancer service is active).
  • Navigate to the Load-Balancer certificates folder: cd /config/vmware/edge/lb/etc/<Load-Balancer ID>/certs/
  • Identify the certificate(s) which is/are referred in the error.
  • Review the certificate content: In this example, use the command "less client_ssl_####-####_####.crt".
    

    A good certificate will have the following format:

    -----BEGIN CERTIFICATE-----
    <data>
    <data>
    <data>
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    <data>
    <data>
    <data>
    -----END CERTIFICATE-----

    The certificate format causing this issue will be:

    -----BEGIN CERTIFICATE-----
    <data>
    <data>
    <data>
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    <data>
                        -------> Extra space or blank line present in the certificate Data
    <data>
                       -------> Extra space or blank line present in the certificate Data 
    <data>
    -----END CERTIFICATE-----

• The above indicates that the Certificate imported into NSX was not imported in the correct format there are lots of spaces in the Certificate data.
  • Identify the certificate name with the above file. Identify on which Virtual Server(s) the certificate is applied.
  • Edit the Virtual Server(s) and remove the Certificate (In the SSL configuration) that was imported with an incorrect format.
  • Re-import the Certificate with the correct format without spaces or any invalid characters. Als,o when you are importing SSL certificate into NSX, it should have the complete Certificate Chain (Leaf SSL certificate(Client/Server), intermediate certificate(s), root certificate) also include the private key for the CA signed certificate if the CSR was not generated in NSX.
  • Apply the newly imported certificate to the required Virtual Server(s)
  • Repeat the above with the other certificates (if any).

Additional Information

Similar issue is also seen same certificate with different names is applied to the same Virtual Server. https://knowledge.broadcom.com/external/article?articleNumber=318329