NSX-T Manager Upgrade Stalls with GUI and API Inaccessible

Products

• VMware NSX

Issue/Introduction

Symptoms:
NSX-T Manager fails to complete upgrade from 2.3.0 to 2.3.1, leaving the GUI and API inaccessible. The failure occurs at the install_os step.   

NSX manager upgrade process goes through 10 different steps and on each step a particular script to executed to perform certain tasks
Following are the steps:

  - name: pre_upgrade_validation
  - name: shutdown_manager
  - name: install_os
  - name: migrate_manager_config
  - name: switch_os
  - name: reboot
  - name: run_migration_tool
  - name: start_manager
  - name: update_upgrade_status
  - name: finish_upgrade

Environment

VMware NSX-T Data Center

Cause

There is a cron job that runs a "du" command on partitions periodically in the NSX-T 2.3 release. This cron job can interfere in the "install_os" scripts that are also running in the background as part of the upgrade. 

Resolution

This issue will be resolved in NSX-T 2.4 release. So, upgrades to 2.4 will not hit this issue. 

To workaround this issue, please contact VMware NSX-T support.