NSX Upgrade stuck. Host component upgrade PAUSED and Post check column for a host shows x issues.

Products

• VMware Cloud Foundation

Issue/Introduction

This article provides information about specific upgrade failure, cause and remedy steps.

Symptoms:

• Error message received - "Cannot continue upgrade. Upgrade coordinator backend is busy in sync operation. Please try again after some time"
• Users have VCF 4.5.1 and NSX-T 3.2.1.2 deployed.
• NSX-T upgrade to version is triggered from SDDC.
• NSX-T manager reports hosts component as PAUSED, specific Host as 100% and Post check column has issues.

Environment

Vmware Cloud Foundation 4.5.1

Cause

During the NSX upgrade workflow, under the Host upgrade phase, some post checks are executed on the host. These checks are executed via nsx-sfhc vib running on the host. nsx-sfhc also contributes in response to TN state API by execution of  "esxcli" to list vibs on the host. The observation is that when "esxcli" takes too long than usual while sfhc remains busy processing those requests, a parallel call to run a post check is not executed in a stipulated time. Hence upgrade mark was checked as failed.
Once "esxcli" call is completed: Retrying post-check works.

Resolution

Timeouts are increased in NSX 4.1.1

Workaround:

To workaround the issue:

1. Re-run the post checks from the UI , after sometime. If successful, Hard refresh the page.

If the above step does not work follow the below mentioned steps

1. Restart upgrade coordinator using NSX root cli as below :

stop service upgrade-coordinator
start service upgrade-coordinator

2. Retry upgrade again.