Transport node configuration state has failed is reported for hosts but the issue resolves itself
search cancel

Transport node configuration state has failed is reported for hosts but the issue resolves itself

book

Article ID: 446026

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • vRealize Operations (vROps) or Aria Operations reports the alert: Transport node configuration state has failed.
  • The alert typically auto-clears within 5 to 10 minutes without administrative intervention.
  • In the NSX Manager /var/log/syslog there will be similar log messages indicating that Logical Switch full-sync to vCenter has failed:

    NSX 77434 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Host configuration failed. Detailed error message :  LogicalSwitch full-sync: Failed to execute the LogicalSwitch full-sync to VC. [TN = TransportNode/<TN-UUID>]. Reason: RuntimeException

  • This can happen due to various reasons while trying to update Logical Switch State - Corfu stale revision update exception, TransactionAbortedException etc:

    WARN L2HostConfigTaskExecutor5 IdentifiableProxyObjectDao 77434 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] corfuDBDataStore.staleRevisionUpdateException(): org
    .corfudb.runtime.exceptions.StaleRevisionUpdateException: Update on stale revision. Correct revision = 709 Given = 708
            at com.vmware.nsx.persistence.UfoTxn$MergeCallbackImpl.doMerge(UfoTxn.java:663)
            at org.corfudb.runtime.collections.TxnContext.merge(TxnContext.java:273)
            at com.vmware.nsx.persistence.UfoTxn.putRecord(UfoTxn.java:569)
            at com.vmware.nsx.persistence.UfoTxn.putRecord(UfoTxn.java:141)
    ..........................................................

    WARN L2HostConfigTaskExecutor3 ObjectsView 77434 TXEnd[TX[4c69]] Aborted Exception
    org.corfudb.runtime.exceptions.TransactionAbortedException: TX ABORT  | Snapshot Time = Token(epoch=376, sequence=3832995976) | Failed Transaction ID = <UUID> | Offending Address = <ADDRESS> | Conflict Key = <CONFLICT_KEY> | Conflict Stream = nsx$LogicalSwitchState | Cause = CONFLICT | Time = 102 ms
            at org.corfudb.runtime.view.StreamsView.append(StreamsView.java:180) ~[?:?]
            at org.corfudb.runtime.view.StreamsView.append(StreamsView.java:233) ~[?:?]

  • Preceding this failure, there is a log in the NSX Manager /var/log/proton/nsxapi.log indicating that the TZs(Transport Zones) are changing as part of the configuration request:

    INFO L2HostConfigTaskExecutor3 TransportNodeAsyncUtils 77434 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Found TZs have changed for TN TransportNode/<TN-UUID> HostSwitch <ID>. DesiredState TZs [TZ UUID] , RealizedState TZs [TZ-UUID, TZ-UUID, TZ-UUID]

Environment

  • VMware NSX

Cause

The issue is triggered by an incorrect delta detection during a configuration full-sync.

  1. NSX incorrectly identifies an internal security Transport Zone (TZ)—created for DFW configurations—as being removed during a full-sync process.
  2. When multiple Transport Nodes (TNs) attempt to update their realization state simultaneously based on this false delta, they compete to update the Logical Switch (LS) state in the underlying Corfu database.
  3. If two nodes attempt to update the same LS at the exact same micro-second, one succeeds while others fail with a TransactionAbortedException or StaleRevisionUpdateException because their view of the LS version becomes stale.
  4. This causes the Transport node configuration realization to be marked as "Failed" until the next automatic retry.

Please note: The larger the number of hosts in the cluster and the larger the number of DVPGs on the DVS used by NSX, the more likely it is to run into this race condition and transaction conflict.

Resolution

  • This issue is transient and self-healing. NSX is designed with automatic retry logic that successfully completes the realization loop once the database thread clears.
    • Validation Steps:
      1. Log in to the NSX Manager UI.
      2. Navigate to System > Fabric > Hosts > Expand relevant cluster.
      3. Verify if the affected hosts have returned to a Success NSX Configuration state.
      4. If the error persists for more than 15 minutes, manually trigger a resync:
        • Select the affected host.
        • Click Actions > Sync Transport Node.
  • No manual workaround is required as the system automatically recovers. 
  • The behavior is fixed in NSX version 9.0.
Powered by Wolken Software