Error: "Client certificate not found in trust store ErrorCode:99" returned from NSX Manager after appliance failover

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware NSX-T Data Center
• VMware Tanzu Application Service

Issue/Introduction

• TKGI clusters are inaccessible via VIP. Clusters might show stuck "in progress" state.
• The NSX Manager appliances are configured in a cluster with more than 1 node.
• The problem occurs after a failover of NSX Manager VIP from one node to another.
• The following error messages in TKGI tasks and on NSX manager might appear: "Error: Error processing bind parameters: [GET /search/query][500] searchByTagInternalServerError  &{RelatedAPIError:{Details:Client certificate not found in trust store ErrorCode:99 ErrorData:<nil> ErrorMessage:Internal server error has occurred. ModuleName:common-services} RelatedErrors:[]}"
• From SSH to NSX Manager node, the following will appear in the reverse-proxy.log (when using TKGI): "level="WARNING" subcomp="http"] Client certificate 'CN=pks-nsx-t-superuser' not found in trust store"

Environment

This issue might impact services that utilize NCP to manage NSX-T environments running below version 4.1.2, including TKGI, and TAS.

Cause

Certain conditions in multi-node NSX Manager configuration prevent the PrincipalIndentity certificate from replicating to all 3 NSX manager appliances. Failover from the primary node that has the certificate to a node that doesn't contain the certificate leads to a failure for services requiring authentication to NSX Manager using the PrincipalIdentity certificate.

Resolution

This is resolved in NSX 4.1.2 and NSX 4.2.0 releases. If upgrade is not possible, workaround the condition by failing over the NSX manager nodes until the original NSX Manager appliance is leader.