vCenter Upgrade from SDDC manager fails with error "No enum constant com.vmware.evo.sddc.lcm.primitive.model.vcenter.VCenterStatusToRemediateSteps.VCENTER_VVS_COMPATIBILITY_FALURE"

Products

• VMware Cloud Foundation
• VMware Cloud Foundation 5.x

Issue/Introduction

 

Symptoms:

• The environment is VCF on VxRail 5.x

• On the SDDC manager UI we are seeing in the error details a "VxRail Manager status code 504 GATEWAY_TIMEOUT for VxRail /rest/vxm/v1/system API"

• The LCM logs indicate errors similar to the following:
  
  lcm-debug.log
  xxxx-xx-xxTxx:xx:xx.xxx+xxxx ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,resourceType=VCENTER,resourceId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,bundleElementId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] 
[c.v.e.s.l.p.impl.vxm.VxManagerClient,Upgrade-7] General exception in executeVxManagerHttpRequest Read timed out
xxxx-xx-xxTxx:xx:xx.xxx+xxxx ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,resourceType=VCENTER,resourceId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,bundleElementId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] 
[c.v.e.s.l.p.i.v.VcenterUpgradeCompatibilityPrimitiveHelper,Upgrade-7] Failed to calculate vcenter upgrade compatibility
java.net.SocketTimeoutException: Read timed out
..
xxxx-xx-xxTxx:xx:xx.xxx+xxxx ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,resourceType=VCENTER,resourceId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,bundleElementId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] 
[c.v.e.s.l.p.i.v.VcenterUpgradeCompatibilityPrimitiveHelper,Upgrade-7] Failed to calculate vcenter upgrade compatibility
..
xxxx-xx-xxTxx:xx:xx.xxx+xxx ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,resourceType=VCENTER,resourceId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,bundleElementId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] 
[c.v.e.s.l.p.i.v.VcenterUpgradeCompatibilityPrimitiveHelper,Upgrade-7] UpgradeError while calculating VVS compatibility, stage: 
Compatibility checks failed for Vcenter Upgrade at VCenterPatchUpdateState(virtualMachineName=XXXX-XXX-XXXX, stage=VCENTER_UPGRADE_PRE_VALIDATION, vCenterSpec=com.vmware.evo.sddc.lcm.model.internal.upgrade.PrimitiveSpec@XXXXXXXX, isUpgradeStageCompleted=true) . 
Due to error: Read timed out Upgrade and interop incompatible product pair details are provided below(if any) :
[] Products and Versions that have no compatibility data include(if any) :[], errorType: RECOVERABLE,errorCode: VC_VVS_INCOMPATIBLE_UPGRADE, errorDescription: 
Compatibility checks failed for Vcenter Upgrade at VCenterPatchUpdateState(virtualMachineName=XXXX-XXX-XXXX, stage=VCENTER_UPGRADE_PRE_VALIDATION, vCenterSpec=com.
vmware.evo.sddc.lcm.model.internal.upgrade.PrimitiveSpec@xxxxxxxx, isUpgradeStageCompleted=true) . 
Due to error: Read timed out Upgrade and interop incompatible product pair details are provided below(if any) :
[] Products and Versions that have no compatibility data include(if any) :[], metadata VC_VVS_INCOMPATIBLE_UPGRADE
  xxxx-xx-xxTxx:xx:xx.xxx+xxx ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,resourceType=VCENTER,resourceId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,bundleElementId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]
   [c.v.e.s.l.p.i.v.VCenterPatchUpgradeImpl,Upgrade-7] VCenter upgrade failed with VCENTER_VVS_COMPATIBILITY_FALURE

 

Environment

VCF on VxRail 5.x , VCF 4.5.x

Cause

This is a know issue affecting VxRAIL Manager that has been highlighted by Broadcom Engineering to Dell where the specific VxRail API (GET /rest/vxm/v1/system) seemed to have issues responding in time resulting in a HTTP 504 Gateway Timeout server error.

Resolution

This is not a SDDC manager issue.

Engage Dell support to investigate the VxRail Managers that exceeding the API timeout limits.

Note:

To test the VxRail Manager responses use the following command executed from the SDDC manager:

time curl -k --user "[email protected]:<password>" --request GET "https://<VxRail_IP_address_>/rest/vxm/v1/system"

To rule out an underlying network issue between the SDDC manager and the VxRail Manager test the cached version of the API and compare the 2 results:

time curl -k --user "[email protected]:<password>"   --request GET "https://<VxRail_IP_address>/rest/vxm/private/v1/system"