vCenter Upgrade from SDDC manager fails with error "No enum constant com.vmware.evo.sddc.lcm.primitive.model.vcenter.VCenterStatusToRemediateSteps.VCENTER_VVS_COMPATIBILITY_FALURE"

Products

• VMware Cloud Foundation

Issue/Introduction

• While upgrading vCenter via SDDC Manager UI, it fails with Error: VxRail Manager status code 504 GATEWAY_TIMEOUT for VxRail /rest/vxm/v1/system API

• Under SDDC Manager's /var/log/vmware/vcf/lcm/lcm-debug.log, below log snippets are found:
  
  YYYY-MM-DDTHH:MM:SS ERROR [vcf_lcm,###########,####,upgradeId=######-####-####-######,resourceType=VCENTER,resourceId=######-####-####-######,bundleElementId=######-####-####-######]
[c.v.e.s.l.p.impl.vxm.VxManagerClient,Upgrade-#] General exception in executeVxManagerHttpRequest Read timed out
YYYY-MM-DDTHH:MM:SS ERROR [vcf_lcm,###########,####,upgradeId=######-####-####-######,resourceType=VCENTER,resourceId=######-####-####-######,bundleElementId=######-####-####-######]
[c.v.e.s.l.p.i.v.VcenterUpgradeCompatibilityPrimitiveHelper,Upgrade-#] Failed to calculate vcenter upgrade compatibility java.net.SocketTimeoutException: Read timed out..
Compatibility checks failed for Vcenter Upgrade at VCenterPatchUpdateState(virtualMachineName=####-###-####, stage=VCENTER_UPGRADE_PRE_VALIDATION, vCenterSpec=com.vmware.evo.sddc.lcm.model.internal.upgrade.PrimitiveSpec@########, isUpgradeStageCompleted=true) .
Due to error: Read timed out Upgrade and interop incompatible product pair details are provided below(if any) :[] Products and Versions that have no compatibility data include(if any) :[], errorType: RECOVERABLE,errorCode: VC_VVS_INCOMPATIBLE_UPGRADE, errorDescription:
Compatibility checks failed for Vcenter Upgrade at VCenterPatchUpdateState(virtualMachineName=####-###-####, stage=VCENTER_UPGRADE_PRE_VALIDATION, vCenterSpec=com.vmware.evo.sddc.lcm.model.internal.upgrade.PrimitiveSpec@########, isUpgradeStageCompleted=true).
Due to error: Read timed out Upgrade and interop incompatible product pair details are provided below(if any) :[] Products and Versions that have no compatibility data include(if any) :[], metadata VC_VVS_INCOMPATIBLE_UPGRADE
####-##-##T##:##:##.###+###ERROR [vcf_lcm,###########,####,upgradeId=######-####-####-######,resourceType=VCENTER,resourceId=######-####-####-######,bundleElementId=######-####-####-######] [c.v.e.s.l.p.i.v.VCenterPatchUpgradeImpl,Upgrade-#] VCenter upgrade failed with VCENTER_VVS_COMPATIBILITY_FALURE

Environment

• VCF on VxRail
• VCF 5.x

Cause

The error stems from a latency issue with the VxRail (GET /rest/vxm/v1/system) API. Because the API failed to respond within the expected timeframe, the server triggered an HTTP 504 Gateway Timeout.

Resolution

Please contact Dell Technical Support to investigate why the VxRail Managers are exceeding established API timeout thresholds.

Additional Information

Note:

• Execute the below command in SDDC Manager SSH, to test the VxRail Manager responses:
  
  time curl -k --user "[email protected]:<password>" --request GET "https://<VxRail_IP_address_>/rest/vxm/v1/system"
  
  
• Execute the below command in SDDC Manager SSH, to rule out any underlying network issue between the SDDC manager and the VxRail Manager and compare both the results:
  
  time curl -k --user "[email protected]:<password>"   --request GET "https://<VxRail_IP_address>/rest/vxm/private/v1/system"