Unable to find any patch Binary under Binary Management of VCF Instance to patch SDDC Manager,NSX, ESXi or vCenter

Products

VMware SDDC Manager VCF Operations

Issue/Introduction

The Binary Management tab located at VCF Fleet Manager > Lifecycle > VCF Instances does not display any upgrade binaries. Specifically, binaries required to patch SDDC Manager, NSX, ESXi, or vCenter Server from version 9.0.0 to 9.0.1.0 are missing
The Binary Management tab within the Lifecycle section of an SDDC 9.0.0 instance does not display the expected upgrade binaries. Specifically, binaries required to patch SDDC Manager, NSX, ESXi, or vCenter Server to version 9.0.1.0 are missing.

When reviewing the lcm-debug.log file on the SDDC Manager Appliance (/var/log/vmware/vcf/lcm/lcm-debug.log), entries similar to the following are observed:

INFO  [vcf_lcm,dd3d########438a,efab] [c.v.v.l.r.a.c.v.m.LcmManifestController,http-nio-127.0.0.1-7400-exec-4] In LcmManifestController, save manifest
DEBUG [vcf_lcm,dd3d########438a,efab] [c.v.e.s.l.s.i.LcmManifestServiceImpl,http-nio-127.0.0.1-7400-exec-4] Saving manifest with sequenceNum:94 and version:1
DEBUG [vcf_lcm,dd3d########438a,efab] [c.v.e.s.l.d.c.m.LcmManifestClientImpl,http-nio-127.0.0.1-7400-exec-4] Found manifest with version 1 and sequence number 95
ERROR [vcf_lcm,dd3d########438a,efab] [c.v.v.l.r.a.c.v.m.LcmManifestController,http-nio-127.0.0.1-7400-exec-4] In LcmManifestController, failed to save manifest
com.vmware.evo.sddc.lcm.model.error.LcmException: Lcm Manifest is already updated
ERROR [vcf_lcm,dd3d########438a,efab] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7400-exec-4] [S5CEL4] LCM_MANIFEST_ALREADY_EXISTS Lcm Manifest is already updated
ERROR [vcf_lcm,dd3d########438a,efab] [c.v.v.i.b.i.TranslationMessage,http-nio-127.0.0.1-7400-exec-4] Can't find resource for bundle java.util.PropertyResourceBundle, key LCM_MANIFEST_ALREADY_EXISTS.remedy

When checking the Manifest version stored in the Lifecycle Manager (LCM) database, the Sequence_Number has a higher value than the current version specified in the vcfManifest.json file, run the following command to verify the sequence number mismatch:
```
psql -U postgres -d lcm -h localhost -c "\x" -c "select id,version,sequence_number from manifest;"
```
Sample Output:
```
Expanded display is on.
-[ RECORD 1 ]---+---
id              | 1
version         | 1
sequence_number | 95
```

Environment

VMware Cloud Foundation 9.x

Cause

This issue occurs when the Sequence Number is modified using a Custom Manifest File.

It may also be triggered while performing the remediation steps described in KB: VCF 9.0.0 Deployment fails with VCF Installer 9.0.1 with error - NSX install image validation failed for NSX Version

Resolution

To resolve this discrepancy, perform the following procedure to ensure the Manifest Table for the LCM database reflects the correct sequence number.

Take snapshot of SDDC Manager Appliance.
SSH to SDDC Manager as vcf user and then switch user to root
Truncate the manifest table in the LCM database by running the following command:
```
psql -U postgres -d lcm -h localhost -c "truncate table manifest"
```

Obtain the token using the following command:

TOKEN=$(curl -H 'Content-Type:application/json' https://localhost/v1/tokens -d '{"username" : "admin@local","password":"<admin_password>"}' -k | jq -r '.accessToken')

If admin@local password is not known then follow How to Reset the Admin@local password in SDDC Manager to reset it.

Update the LCM table with the correct Sequence Number using the following command:

curl -k -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" -X POST https://localhost/v1/manifests -d "@/nfs/vmware/vcf/nfs-mount/metadata/vcfManifest.json"

Verify that the database has been updated with the correct Sequence Number by running the following command:

psql -U postgres -d lcm -h localhost -c "\x" -c "select id,version,sequence_number from manifest;"

Sample Output:

Expanded display is on.
-[ RECORD 1 ]---+---
id              | 2
version         | 1
sequence_number | 94

Restart the LCM service by running the following command:
```
systemctl restart lcm
```
Check the Binary Management tab to confirm that the binaries for all SDDC Bill of Materials (BOM) components (SDDC Manager, NSX, ESXi, and vCenter Server) for version 9.0.1.0 are now displayed.

Additional Information

If Issue persists please reach out to Broadcom Support referring this KB and we will be there to assist.