Symptoms:
- You are upgrading NSX-T Data Center.
- In NSX-T Manager log /var/log/upgrade-coordinator/catalina.*.log the following messages can be seen:
2021-09-29T08:52:13.044Z ERROR http-nio-127.0.0.1-7442-exec-5 UpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30078" level="ERROR" subcomp="upgrade-coordinator"] Error while continuing the upgrade for component MP
com.vmware.nsx.management.common.exceptions.BaseException: Pre-upgrade checks failed for MP: Failed to execute Datastore cluster status check. null
at com.vmware.nsx.management.upgrade.service.impl.UpgradeServiceImpl.handleMandatoryCheckResult(UpgradeServiceImpl.java:963) ~[libuc-core.jar:?]
- In NSX-T Manager log /var/log/upgrade-coordinator/upgrade-coordinator.log the following messages can be seen:
2021-09-29T08:52:13.001Z INFO http-nio-127.0.0.1-7442-exec-5 UpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Executing check Datastore cluster status check
2021-09-29T08:52:13.003Z INFO http-nio-127.0.0.1-7442-exec-5 ClusterManagerUtil - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Sending request to cluster manager with api path http://localhost:7441/api/v1/cluster-manager/status
2021-09-29T08:52:13.006Z ERROR http-nio-127.0.0.1-7442-exec-5 ClusterManagerUtil - - [nsx@6876 comp="nsx-manager" errorCode="MP2101" level="ERROR" subcomp="upgrade-coordinator"] Request GET http://localhost:7441/api/v1/cluster-manager/status HTTP/1.1 failed, return code is 403
2021-09-29T08:52:13.006Z ERROR http-nio-127.0.0.1-7442-exec-5 ClusterManagerUtil - - [nsx@6876 comp="nsx-manager" errorCode="MP2121" level="ERROR" subcomp="upgrade-coordinator"] Cluster status retrieved from cluster manager is empty.
- In NSX-T Manager log /var/log/nvpapi/api_access.log the following messages can be seen:
2021-09-29T08:28:35.815Z api_access.80.syslog INFO 127.0.0.1 - netcldadmin [2021-09-29 08:28:35] 'GET /api/v1/cluster-manager/status' 403 212 "https://10.91.252.123/upgrade-coordinator/upgrade.html?baseRouteUrl=systemAdmin/utilities/upgrade" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" 0.001456
2021-09-29T08:29:59.560Z api_access.80 INFO 127.0.0.1 - UC [2021-09-29 08:29:59] 'GET /api/v1/cluster-manager/status' 200 12016 "" "Java/1.8.0_212" 0.033233