"The TN certificate ... failed validation: The certificate has expired" upgrade precheck fails however no TN cert is expired

Products

• VMware NSX

Issue/Introduction

• This can be observed when upgrading to NSX 4.2.3 or higher 
• During the Pre-checks of NSX Manager the following error is displayed Pre-upgrade checks failed for MP: The TN certificate with ID 1f49f77b-####-####-############ failed validation: The certificate has expired. Please delete or replace this certificate prior to upgrading.  See KB article https://knowledge.broadcom.com/external/article?articleId=369034
• The Certificate Analyzer, Results and Recovery (CARR) Script does not display any expired certificates for Transport Nodes 
• When running the curl command no node id is returned  curl -k -X GET -H 'X-Nsx-Username:admin' "http://localhost:4321/nsxapi/api/v1/messaging/clients/distribution"

   {   
        "client_ids": [
          "[cvn-edge]:50a7cec9-####-####-############",
          "[cvn-hv]:7d79a697-####-####-############",
          "[cvn-mp-mpa]:a65fa708-####-####-############"
        ],  
        "node_id": "b8e7d107-####-####-############"
      },  
      {   
        "client_ids": [
          "[cvn-hv]:7d25113d-####-####-############",
          "[cvn-hv]:1f49f77b-####-####-############"
        ],  
        "node_id": "00000000-0000-0000-0000-000000000000". <---- No node_id for above 2 nodes.
      }    
    ]   
  }


Environment

VMware NSX 4.2.3 or higher

Cause

The Transports nodes have been failed to be removed from the messaging client's list.

Resolution

This is a known issue impacting VMware NSX.

Workaround: 
Ensure that you have a recent, valid backup of your NSX managers and ensure that you know the passphrase for your backups.

• For the Client_ids such as 7d25113d-####-####-############ & 1f49f77b-####-####-############" that are reporting no node_id, these  can deleted nodes manually from messaging clients by executing below command and repeat for both Clients ids:
  curl -k -X DELETE -H 'X-Nsx-Username:admin' http://localhost:4321/nsxapi/api/v1/messaging/clients/7d25113d-####-####-############
• Confirm that these Client_ids have been removed by checking the below output 
  curl -k -X GET -H 'X-Nsx-Username:admin' http://localhost:4321/nsxapi/api/v1/messaging/clients