Host Post Upgrade check fails after NSX manager upgrade

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• After upgrading NSX Transport Nodes and the NSX Manager(s), you observe Post Check Status errors for upgraded Hosts
• The upgrade was successful
• The errors you observe may include:
  • Failed to execute Check to ensure a host is out of maintenance-mode. Command IsHostInMaintenanceMode failed on host <UUID>.
  • Error while checking if host reboot is pending from a previous action. Please check the configuration state of host <UUID> and ensure that services are operating normally.
  • Failed to execute ESXi post upgrade dataplane check. Failed to execute ESXi post upgrade dataplane check. Error occurred while copying the upgrade scripts to host <UUID>. SFHC connectivity may be down.
• Checking logs on NSX Manager: var/log/upgrade-coordinator/upgrade-coordinator.log

####-##-##T##:##:##.###Z  INFO pool-48-thread-1-rpc StubManager 83142 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Unable to reach client ########-####-###-####-########, application UPGRADE, msg GenericUnifiedMsg,  methodName rpcMsgHandler
####-##-##T##:##:##.###Z ERROR ForkJoinPool.commonPool-worker-5 HostSfhcTemplate 14638 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30132" level="ERROR" subcomp="upgrade-coordinator"] Execution of Command <UUID> failed on (###############{})
com.vmware.nsx.messaging.exceptions.MessagingException: Unable to reach client ########-####-###-####-########, application SERVICE_FABRIC_APPLICATION
        at com.vmware.nsx.messaging.nsxrpc.StubManager.runRPC(StubManager.java:278) ~[libmessaging-library.jar:?]
        at com.vmware.nsx.messaging.nsxrpc.StubManager.callRpc(StubManager.java:251) ~[libmessaging-library.jar:?]
        at com.vmware.nsx.messaging.rpc.RpcManager.sendRequest(RpcManager.java:215) ~[libmessaging-library.jar:?]
        at com.vmware.nsx.messaging.service.impl.ClientServiceImpl.sendRpcRequest(ClientServiceImpl.java:105) ~[libmessaging-library.jar:?]
        at com.vmware.nsx.messaging.service.impl.ClientServiceImpl.sendRpcRequest(ClientServiceImpl.java:76) ~[libmessaging-library.jar:?]

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX 4.x
VMware NSX-T Data Center 3.x

Cause

Host Post Upgrade check fails due to NSX RPC call not reaching the host.
This is a condition that may occur in a VMware NSX environment.

Resolution

This issue is resolved in VMware NSX 4.1.2  available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:
Use the 'RUN POST CHECK' feature again at a later time.