NSX-T "nsxa down" Error and vMotion Failure Due to APH_TN Certificate Sanitization

Products

VMware NSX

Issue/Introduction

Connectivity issues occur where virtual machines (VMs) lose tags and Distributed Firewall (DFW) rules fail to apply, often resulting in vMotion failures. This behavior is linked to a configuration hash mismatch in the Management Plane Agent (MPA) communication between the NSX Manager nodes and ESXi hosts.

vMotion fails with:

On affected host's in /var/run/log/nsx-syslog.log, the following configuration hash mismatch error is observed, indicating a discovery loop on the ESXi host:

In(182) nsx-proxy[2101198]: NSX 2101198 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="mpa-proxy-lib" tid="2101198" level="INFO"] HeartbeatManager: configuration hash mismatch in heartbeat callback. Old hash - #############################################################1234, New hash - ############################################################5678. Invoking RESET on Forwarding Engine

Check which host is the master for the transport node, in the host nsxcli run get manager, it will return an entry like, the one with the connected and Asterix indicates it is the master:
```
- <NSX-Manager-IP-A>     Connected (NSX-RPC) *
```

On the impacted ESXi host, check which manager node where last discovery was received, look in /var/run/log/nsx-syslog.log for:

In(182) nsx-proxy[2101198]: NSX 2101198 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="mpa-proxy-lib" tid="2101253" level="INFO"] Received discovery response on: fqdn - , ip - <NSX-Manager-IP-B>, port - 1234.

Checking on the master node and last discovery node, in the NSX managers logs /var/log/messaging-manager/messaging-manager.log, for both managers, looking at around the same time as the discovery response log entry above, we see a difference in the certificate PEM between the 2 nodes, for port 1234:

Example Manager A - Master node:

INFO MessagingStreamListenerImpl-1-1 MessagingStreamListenerImpl - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="messaging"] Corfu StreamListener entry UfoTable ClusterNodeConfigModel, (########-9ab4-41ac-8158-############) : payload managed_resource {
......
  appliance_conn_info {
    ip_address {
      value: "<NSX-Manager-IP-A>"
    }
    port: 1234
    certificate {
      value: "-----BEGIN CERTIFICATE-----\n<PEM details>\n-----END CERTIFICATE-----"
    }

Example Manager B-discovery node:

INFO MessagingStreamListenerImpl-1-1 MessagingStreamListenerImpl - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="messaging"] Corfu StreamListener entry UfoTable ClusterNodeConfigModel, (########-1234-acdb-99oo-############) : payload managed_resource {
......
  appliance_conn_info {
    ip_address {
      value: "<NSX-Manager-IP-B>"
    }
    port: 1234
    certificate {
      value: "-----BEGIN CERTIFICATE-----\n<PEM details>\n-----END CERTIFICATE-----\n"
    }

Environment

VMware NSX 4.1.x
VMware NSX 4.2.x
VCF NSX 9.x

Cause

The issue is triggered by a configuration hash mismatch in the Management Plane Agent (MPA) heartbeat. One or more NSX Manager nodes contain an APH_TN certificate with a trailing newline (\n) character. This causes a mismatch during heartbeat verification with the master node, leading to continuous connectivity instability.

Resolution

Preparation: Take a full NSX Manager backup before proceeding.
Node 1 (Non-VIP Leader):
- SSH as admin to the first non-VIP manager node.
- Run get cluster status to confirm system health.
- If all services are healthy, execute: restart service messaging-manager.
- Confirm status with get service messaging-manager and get cluster status. Ensure all services are healthy before proceeding.
Node 2 (Non-VIP Leader):
- SSH as admin to the second non-VIP manager node.
- Run get cluster status to confirm system health.
- Execute: restart service messaging-manager.
- Confirm status with get service messaging-manager and get cluster status. Ensure all services are healthy before proceeding.
Node 3 (VIP Leader):
- SSH as admin to the manager node currently acting as the VIP leader.
- Run get cluster status to confirm system health.
- Execute: restart service messaging-manager.
- Confirm status with get service messaging-manager and get cluster status.
Post-Resolution Verification:
- Monitor the ESXi log file /var/run/log/nsx-syslog.log (or your centralized logging server) to ensure the "nsxa down" errors have ceased.

Note: For defects and enhancements related to this issue, subscribe to this article at Subscribe to Article Updates to receive notifications regarding future patches or official resolutions.

Additional Information

There is another KB with similar log entries, but due to a version hash mismatch VMs configured with NSX-T DFW rules become unreachable on the network
If the issue persists after performing the service restarts, please collect support bundles from the NSX Managers and affected ESXi hosts.
To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.