SDDC Manager upgrade fails with VCF Release 9.1.1.0.25713928 not found in LCM Manifest

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

When performing an upgrade pre-check or attempting to upgrade SDDC Manager in VMware Cloud Foundation (VCF) from version 9.1.0.0 to 9.1.1.0, the Management Domain pre-check fails at the compatibility check stage.

In /var/log/vmware/vcf/lcm/lcm-debug.log entries can be found similar to the following:

<timestamp> ERROR vcf_lcm 2133191 [sddcm@4413 traceId="0000000000000000" spanId="0000" actionExecutionId="########-####-####-####-###########" opId="########-####-####-####-###########" parentTaskId="########-####-####-####-###########" taskId="########-####-####-####-###########" threadId="AsyncTaskEngine10" class="c.v.v.l.p.s.a.CompatibilityPrecheckAction" method="executeStep" line="119"] Error occurred while executing the out of band upgrade precheck
java.lang.RuntimeException: VCF Release 9.1.1.0.25713928 not found in LCM Manifest
        at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.lambda$vmwBomsIdentical$21(LcmManifestServiceImpl.java:1310)
        at java.base/java.util.Optional.orElseThrow(Optional.java:403)
        at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.vmwBomsIdentical(LcmManifestServiceImpl.java:1307)

Or:

<timestamp> INFO TargetVersionVvsCompatibilityValidator: Validate target version: 9.1.1.0.25713928 on domain: <REDACTED_DOMAIN_ID> for VVS compatibility
<timestamp> DEBUG LcmManifestServiceImpl: Get required VMWARE_SOFTWARE type bundles for skip upgrade from {"version":"9.1.0.0"} to {"version":"9.1.1.0.25713928"}
<timestamp> ERROR CompatibilityPrecheckAction: Error occurred while executing the out of band upgrade precheck
java.lang.RuntimeException: VCF Release 9.1.1.0.25713928 not found in LCM Manifest
        at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.lambda$vmwBomsIdentical$21(LcmManifestServiceImpl.java:1310)
        at java.base/java.util.Optional.orElseThrow(Optional.java:403)
        at com.vmware.evo.sddc.lcm.services.impl.LcmManifestServiceImpl.vmwBomsIdentical(LcmManifestServiceImpl.java:1307)

Environment

VMware Cloud Foundation (VCF) 9.1.0.0
SDDC Manager 9.1.0.0

Cause

The Lifecycle Management (LCM) Unified Precheck Framework validator passes the SDDC Manager component build string (9.1.1.0.25713928) as the target release version key instead of the top-level VCF release version (9.1.1.0), resulting in a manifest lookup failure.

Resolution

To prevent this issue, the compatibility pre-check can be skipped by taking the following steps:

Establish an SSH connection with the SDDC Manager and log in with the vcf user.
Elevate the shell to root level by running the following command:
```
# su -
```
and provide the root password when queried for it.

Create a backup copy of the compatibility.flag file:

# cp /opt/vmware/vcf/lcm/lcm-app/conf/compatibility.flag /opt/vmware/vcf/lcm/lcm-app/conf/compatibility.bak

Edit the compatibility.flag file using the VI editor

Replace the line:

vcf.compatibility.controllers.compatibilityCheckEnabled=true

against:

vcf.compatibility.controllers.compatibilityCheckEnabled=false

Save the change and exit the editor (:wq!).
Restart the Lifecycle Manager service:
```
# systemctl restart lcm
```

Once the appliance was successfully upgraded, revert the change by setting the vcf.compatibility.controllers.compatibilityCheckEnabled option back to true, then restart the lcm service again to pickup this change.