NSX Federation: After upgrading a standby GM cluster to 3.2.x, "Error: Application is not ready (Error code: 280)" is displayed on standby GM UI

Products

• VMware NSX

Issue/Introduction

Symptoms:

- Example of error in Standby GM UI:




- Standby GM upgrade is complete:
get node upgrade status
2022-04-19 22:09:29,863 - Resuming paused playbook /var/vmware/nsx/file-store/VMware-NSX-manager-3.2.0.1.0.19232400-playbook.yml
2022-04-19 22:09:29,931 - Validating playbook /var/vmware/nsx/file-store/VMware-NSX-manager-3.2.0.1.0.19232400-playbook.yml
2022-04-19 22:09:30,028 - Running "run_migration_tool" (step 7 of 13)
2022-04-19 22:15:12,071 - Running "start_manager" (step 8 of 13)
2022-04-19 22:20:33,281 - Running "resume_other_nodes" (step 9 of 13)
2022-04-19 22:25:59,265 - Running "update_events" (step 10 of 13)
2022-04-19 22:26:04,350 - Running "restore_datastore_cluster" (step 11 of 13)
2022-04-19 22:26:54,592 - Running "update_upgrade_status" (step 12 of 13)
2022-04-19 22:26:59,649 - Running "finish_upgrade" (step 13 of 13)
2022-04-19 22:26:59,757 - Playbook finished successfully
{
  "state": 1,
  "state_text": "CMD_SUCCESS",
  "info": "",
  "body": null
}

- "Application not ready" error message is also seen in API responses:

GET https://<Standby GM IP>/global-manager/api/v1/global-infra/global-managers
{
    "module_name": "common-services",
    "error_message": "Application is not ready", <---- This is expected
    "error_code": 280
}

 

GET https://<Standby GM IP>/global-manager/api/v1/global-infra/
{
    "httpStatus": "SERVICE_UNAVAILABLE",
    "error_code": 280,
    "module_name": "common-services",
    "error_message": "Application is not ready"
}

GET https://<Standby GM IP>/global-manager/api/v1/global-infra/domains/default/groups
{
    "httpStatus": "SERVICE_UNAVAILABLE",
    "error_code": 280,
    "module_name": "common-services",
    "error_message": "Application is not ready"
}

 

Error in /var/log/gmanager/gmanager.log after Standby GM upgrade:

20##-##-##T##:##:##.###Z INFO http-nio-127.0.0.1-64440-exec-2 NsxBaseRestController 4519 SYSTEM [nsx@6876 comp="global-manager" level="INFO" subcomp="global-manager"] Error in API /gm/api/v1/infra/global-managers caused by exception com.vmware.nsx.management.common.exceptions.BaseException: {"moduleName":"common-services","errorCode":280,"errorMessage":"Application is not ready"}

20##-##-##T##:##:##.###Z  INFO http-nio-127.0.0.1-64440-exec-2 NsxBaseRestController 4519 SYSTEM [nsx@6876 comp="global-manager" level="INFO" subcomp="global-manager"] Error in API /gm/api/v1/infra/global-managers caused by exception com.vmware.nsx.management.common.exceptions.BaseException: {"moduleName":"common-services","errorCode":280,"errorMessage":"Application is not ready"}

 

Environment

VMware NSX-T Data Center

Cause

This is expected when the Standby GM cluster is first upgraded to 3.2.x.
The replication service expects the Active GM and Standby GM to be on the same NSX-T version.

Resolution

This issue is resolved on later versions of NSX and does not occur after the upgrade to 3.2.x is complete.

Workaround:

Proceed with the Active GM cluster upgrade. The error is resolved once the Active and Standby GMs are on the same version.