NSX-T upgrade continue operation failed: Pre-upgrade checks failed for MP: Failed to execute Datastore cluster status check. null

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You are upgrading NSX-T Data Center.
• In NSX-T Manager log /var/log/upgrade-coordinator/catalina.*.log the following messages can be seen:

2021-09-29T08:52:13.044Z ERROR http-nio-127.0.0.1-7442-exec-5 UpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30078" level="ERROR" subcomp="upgrade-coordinator"] Error while continuing the upgrade for component MP
com.vmware.nsx.management.common.exceptions.BaseException: Pre-upgrade checks failed for MP: Failed to execute Datastore cluster status check. null
        at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.handleMandatoryCheckResult(UpgradeServiceImpl.java:963) ~[libuc-core.jar:?]

• In NSX-T Manager log /var/log/upgrade-coordinator/upgrade-coordinator.log the following messages can be seen:

2021-09-29T08:52:13.001Z  INFO http-nio-127.0.0.1-7442-exec-5 UpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Executing check Datastore cluster status check
2021-09-29T08:52:13.003Z  INFO http-nio-127.0.0.1-7442-exec-5 ClusterManagerUtil - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Sending request to cluster manager with api path http://localhost:7441/api/v1/cluster-manager/status
2021-09-29T08:52:13.006Z ERROR http-nio-127.0.0.1-7442-exec-5 ClusterManagerUtil - - [nsx@6876 comp="nsx-manager" errorCode="MP2101" level="ERROR" subcomp="upgrade-coordinator"] Request GET http://localhost:7441/api/v1/cluster-manager/status HTTP/1.1 failed, return code is 403
2021-09-29T08:52:13.006Z ERROR http-nio-127.0.0.1-7442-exec-5 ClusterManagerUtil - - [nsx@6876 comp="nsx-manager" errorCode="MP2121" level="ERROR" subcomp="upgrade-coordinator"] Cluster status retrieved from cluster manager is empty.

• In NSX-T Manager log /var/log/nvpapi/api_access.log the following messages can be seen:

2021-09-29T08:28:35.815Z api_access.80.syslog INFO 127.0.0.1 - netcldadmin [2021-09-29 08:28:35] 'GET /api/v1/cluster-manager/status' 403 212 "https://10.91.252.123/upgrade-coordinator/upgrade.html?baseRouteUrl=systemAdmin/utilities/upgrade" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" 0.001456

2021-09-29T08:29:59.560Z api_access.80 INFO 127.0.0.1 - UC [2021-09-29 08:29:59] 'GET /api/v1/cluster-manager/status' 200 12016 "" "Java/1.8.0_212" 0.033233

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center 3.x

Cause

HTTP error code 403 usually means a client-side error such as authentication failure, lack of permissions to access the requested resource or client certificate issue or the user's session expired. 
This situation is likely to happen when using a vIDM user to upgrade NSX-T.

Resolution

Log in again to the browser.