NSX Manager upgrade pre-check fails due to upgrade-coordinator processes are stuck

Products

• VMware NSX

Issue/Introduction

• Upgrade pre-check failed with the following message.
  

  Unexpected error while upgrading upgrade unit: NSX Manager upgrade dry run failed. Do not proceed with the upgrade. Please collect the support bundle and contact VMWare GS.

• Retry pre-check also failed.
• You see messages similar to following in /var/log/upgrade-coordinator/logical-migration.log on NSX Manager:
  

  ####-##-##T##:##:##.###Z ERROR main MVOCorfuCompileProxy 31741 Access[ImmutableCorfuTable[f2a]]
  org.corfudb.runtime.exceptions.TrimmedException: getStreamAddressMap[{f2a@-1}] [-1, ##########] stream has been trimmed at address ########## and this space is not covered by the loaded checkpoint with start address -1, while accessing the stream at version ##########. Looking for a new checkpoint.

Environment

VMware NSX

Cause

• upgrade-coordinator spawns multiple processes during the life cycle and processes can get stuck for various reasons. For examples:
  • NSX-T upgrade pre check never completes
  • upgrade-coordinator upgrade progress is stuck in 60%
    
    
• upgrade-coordinator is not robust enough to cleanup stuck processes and cannot recover on its own.

Resolution

• This issue is fixed in VMware NSX 4.2.1.
• Restart upgrade-coordinator service as admin on the NSX Manager where the pre-check invoked to unblock this situation.
  

  nsxmanager> restart service install-upgrade