Bypass SDDC Manager upgrade compatibility checks when upgrade is unavailable due to missing compatibility data

Products

• VMware Cloud Foundation

Issue/Introduction

• Upgrade compatibility checks fail with error messages like:

  PRODUCT_VERSIONS_MISSING_FROM_COMPATIBILITY_DATA=Unable to compute compatibility for target SDDC Manager version: 
  {0} due to missing compatibility data for the following set of products on the system: {1}
  
  Target SDDC Manager version: <version> is not compatible with the system.
  
  "COMPATIBILITY_CONTROLLER_EXCEPTION"="Failed to load cache data from file 
  /nfs/vmware/vcf/nfs-mount/compatibility/VmwareCompatibilityData.json into VVS offline cache."
  
  The product XXXX version {"version":"X.X.XXX-XXXXX"} is not compatible with: [VcfProductVersion(productType=XXXX, version=X.X.XXX-XXXXX)]

  Example: 
  ERROR  [c.v.v.c.v.i.CompatibilityControllerAggregator,Scheduled-2] The product VX_MANAGER version {"version":"8.0.100-28093095"} is not compatible with: [VcfProductVersion(productType=VCENTER, version=7.0.3.01500-21784236)]

  Getting Upgradable API call failed. VvsCompatibilityController controller failed with error message: Cannot invoke "com.vmware.vcf.compatibility.controllers.internal.vvs.rest.model.ProductBundles.getProducts()" because "this.offlineCacheData" is null

Environment

VCF 5.x
VCF on VxRail 5.x

Cause

As part of the VCF 5.0 release, there is a new feature that allows users to indefinitely upgrade their SDDC Manager to newer releases, without having to upgrade other BoM components on the system (namely NSX, vCenter and ESXi).

This is contingent on whether or not a target SDDC Manager version is compatible with said BoM components.

If the compatibility data sets are malformed or in an incorrect format the SDDC Manager will not be able to compute compatibility checks. As a result, if the BOM versions found on the system can't be mapped to versions found in the compatibility data, upgrade availability can't be computed. This will cause the upgrades to be blocked.

Resolution

Ensure that the compatibility data sets are up-to-date.

This issue may still occur, because there is the possibility that the data set might not fully reflect certain specific real world environment inventory/upgrade use cases.

In these scenarios, provided we have up-to-date compatibility data sets, the Upgrade Compatibility checks can be disabled using the steps provided in the workaround.

 

Warning: For upgrades to VCF 9.0.0.0 that warn of NSX incompatibility the below workaround must not be applied. Some versions of NSX are currently incompatible for upgrade to VCF 9.0.0.0 and the VCF upgrade process must not be started, see Following for more details NSX Upgrade version is not compatible with the current NSX version

Workaround:

To disable the upgrade compatibility checks, follow the steps below:

1. SSH into SDDC Manager appliance with the vcf user and su to root.
2. Edit the file: /opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag 
3. Update the property vcf.compatibility.controllers.compatibilityCheckEnabled to false
4. Edit the file: /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties
5. Update the property lcm.enable.vvs.compatibility.upgrade.precheck to false
6. Save and close the file.
7. Restart Lifecycle Management by running the command: systemctl restart lcm

Note: Re-enable the compatibility checks once the upgrades are complete to ensure guardrails are in place with respect to Bill-of-Materials compatibility for future upgrades.

Additional Information

To re-enable the compatibility checks, follow the steps below:

1. SSH into SDDC Manager appliance with the vcf user and su to root.
2. Edit the file: /opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag
3. Update the property vcf.compatibility.controllers.compatibilityCheckEnabled to true
4. Save and close the file.
5. Restart Lifecycle Management by running the command: systemctl restart lcm