NSX Global Manager Upgrade 4.1.2.3 -> 4.2.1.X failed during service shutdown

Products

• VMware NSX

Issue/Introduction

NSX Global Manager Upgrade failed during Timed out waiting for shutdown of corfu-server-wrapper 

 

From NSX manager we can see. 

 

get upgrade progress-status 

Timed out waiting for shutdown of corfu-server-wrapper
     Stdout: Starting shutdown script
Shutdown script complete
 
Stderr: /image/VMware-NSX-unified-appliance-4.2.1.0.0.24302016/scripts/shutdown_manager.py:254: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if ret is 0 and os.path.exists(_ORCHESTRATOR_NODE_TOUCH_FILE):
/image/VMware-NSX-unified-appliance-4.2.1.0.0.24302016/scripts/shutdown_manager.py:258: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if ret is 0:
/image/VMware-NSX-unified-appliance-4.2.1.0.0.24302016/scripts/shutdown_manager.py:260: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if ret is 0:
/image/VMware-NSX-unified-appliance-4.2.1.0.0.24302016/scripts/shutdown_manager.py:262: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if ret is 0:

  Shutdown script complete",
        "troubleshooting": "Upgrade has failed. Appliance OS is of old version, however, UI will not be available. Please contact GSS to retry the upgrade."

 

Environment

VMware NSX

Resolution

• Restart/resume upgrade from CLI when orchestrator UI is not accessible
  1. Check which step failed in /var/log/resume-upgrade.log
  2. Run CLI command "get upgrade-bundle <bundle-name> playbook VMware-NSX-manager-<version>-playbook" to get list of steps in playbook of manager. Verify that the failed step is after the "reboot" step in sequence
  3. (Only on orchestrator node) Run the playbook again using CLI command "start upgrade-bundle <bundle-name> playbook VMware-NSX-manager-<version>-playbook". Since the failure is in post-reboot upgrade step, MP will not reboot again
  4. (Only on orchestrator node) Run CLI command "resume upgrade-bundle <bundle-name> playbook" to resume the upgrade
  5. get upgrade progress-status

•  
• Once you observed shutdown_manager failed
  1. stop corfu server "/etc/init.d/corfu-server stop"
  2. wait for update to resume 
     • 
  3. Verify upgrade status  "get upgrade progress-status"
     • 
  4. Wait for update to finish