VCF Upgrade to 9.1 failure during Fleet LCM deployement step at "Deploy VCF Management Components" step
search cancel

VCF Upgrade to 9.1 failure during Fleet LCM deployement step at "Deploy VCF Management Components" step

book

Article ID: 452077

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

When upgrading VCF from 9.0.x (e.g., 9.0.2) to 9.1.x, the upgrade fails at the "Deploy VCF Management Components" step.

Snippet in vcf-fleet-build-service-fleetbuild-#########.log:

Couldn't get component (id: <UUID>, type: OPS) from Ops Platform. Reason: Unexpected VCF Operations Suite client error has occurred.

Error occurred while updating component with id: <UUID>

Unable to register component with Id: <UUID> in VCF Ops.

Environment

VCF (VMware Cloud Foundation) 9.1.

Cause

This issue is caused by a race condition if the VCF Operations upgrade overlaps with the Ops LCM daily scheduled inventory sync.

If the daily sync triggers while the Ops 9.1 upgrade is in progress (specifically, after the new 9.1 binaries are updated in Ops), it prematurely records the Ops environment as version 9.1.

Ops LCM Inventory sync /var/log/vrlcm/vmware_vrlcm.log snippet: 

YYYY-MM-DDT03:00:ss.###Z  ProductInventorySyncPlanner - Creating spec for inventory sync for product : vrops in environment : <UUID>
YYYY-MM-DDT03:00:ss.###Z  MachineRegistry - GETTING MACHINE FOR THE KEY :: vropsimportenvironment   (daily sync workflow starts)
YYYY-MM-DDT03:00:ss.###Z  DeploymentsDiscoveryTask - Starting VMware Aria Operations Discovery task.
YYYY-MM-DDT03:00:ss.###Z  VropsConnector - GET /casa/node/config: 200 200        (live call to OPS's CASA API succeeds)
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - Getting version from <Ops_FQDN>
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - version obtained from VMware Aria Operations ::: 9.1.0.0100.25435105
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - executing getVersion
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - version obtained from VMware Aria Operations ::: 9.1.0.0100.25435105
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - Version obtained: 9.1.0

 

Environment updated records .../logs/component-manager-plugin-uuid.log snippet:

YYYY-MM-DDT03:0m:ss.###Z INFO ... [CollectInventoryTask.execute] - Collecting legacy lcm inventory data...
YYYY-MM-DDT03:0m:ss.###Z INFO ... [CollectInventoryTask.printEnvironments] - EnvironmentId : <UUID>, isFleetInstance : true, Product : vrops, Version : 9.1.0.0

Later during the "Deploy VCF Management Components" phase, Fleet LCM queries the Ops version. Because it receives version 9.1 instead of the expected legacy 9.0.x version, the Fleet LCM deployement subsequnetly fails.

 

 

Resolution

To resolve and prevent this issue, the VCF upgrade must not overlap with the Ops LCM daily inventory sync window. Since the scheduled time for the daily inventory sync varies across different environments, the specific sync time must be manually identified before proceeding.

Step 1. Revert the Environment:

Revert the VCF environment (including VCF Ops and LCM) to the pre-upgrade 9.0.x state using available snapshots.

 

Step 2. Identify the Daily Inventory Sync Time

Review the /var/log/vrlcm/vmware_vrlcm.log file to determine when the daily sync workflow is triggered. Search for the ProductInventorySyncPlanner and DeploymentsDiscoveryTask entries.

Example snippet from /var/log/vrlcm/vmware_vrlcm.log:

YYYY-MM-DDT03:00:ss.###Z  ProductInventorySyncPlanner - Creating spec for inventory sync for product : vrops in environment : <UUID>
YYYY-MM-DDT03:00:ss.###Z  MachineRegistry - GETTING MACHINE FOR THE KEY :: vropsimportenvironment   (daily sync workflow starts)
YYYY-MM-DDT03:00:ss.###Z  DeploymentsDiscoveryTask - Starting VMware Aria Operations Discovery task.
YYYY-MM-DDT03:00:ss.###Z  VropsConnector - GET /casa/node/config: 200 200        (live call to OPS's CASA API succeeds)
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - Getting version from <Ops_FQDN>
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - version obtained from VMware Aria Operations ::: 9.1.0.0100.25435105
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - executing getVersion
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - version obtained from VMware Aria Operations ::: 9.1.0.0100.25435105
YYYY-MM-DDT03:00:ss.###Z  VropsEndpoint - Version obtained: 9.1.0

Note: In the example above, the daily inventory sync is scheduled at exactly 03:00 (UTC).

 

Step 3. Re-initiate the Upgrade

Re-initiate the VCF 9.0.x to 9.1 upgrade, ensuring the execution time is scheduled to avoid the identified sync window. It is recommended to schedule the upgrade to start at least 10 minutes after the daily sync has occurred to ensure no overlap.

 

Powered by Wolken Software