"Import VCF Operations in Fleet Lifecycle" stage of "Deploy VCF Management Components" workflow fails or completes with warnings

Products

VCF Operations VMware Cloud Foundation VCF Automation VCF Operations for Networks

Issue/Introduction

This issue occurs during a VCF 9.0 to 9.1 upgrade with the following warnings:

Failed to discover component of type 'VCFA'
Failed to discover component of type 'VIDB'
Failed to discover component of type 'OPS_LOGS'
An unexpected error occurred in step discover_vrni_component_task
Failed to add existing VCF Operations Networks
Failed to add existing Log management

If SDDC Manager, Fleet lifecycle, SDDC lifecycle, and VCF Operations are all at version 9.1.0.0300 or newer, review this section.

After "Deploy VCF Management Component" workflow completes, if one of the components in VCF 9.0 deployment (such as VCF Automation) could not be add to Fleet lifecycle, one or more of the following information is seen.

VCF Operations UI will display banner to prompt users to go to VCF Management tab, as some components are blocked from being upgraded

Top Notification Banner:
"Some components are blocked from being upgraded to 9.1 because they failed to be added to Fleet Lifecycle. GO TO VCF MANAGEMENT"
VCF Operations UI will report an alert for user attention

Alert Name:
Action Required: Fleet Lifecycle task(s) failed

Symptom:
The Critical symptom Some components are blocked from being upgraded to 9.1 because they failed to be added to Fleet Lifecycle. has been observed on vcf-fleet-lcm

(Note: Cancelling this alert from the VCF Operations Alerts panel dismisses both the top warning banner (example 1 above) and the VCF Management warning panel (example 3 below)
VCF Operations Build → Lifecycle → VCF Management→ Components tab shows a warning panel with a list of components that failed to be added to Fleet lifecycle, each linking to the corresponding task detail

Main Warning Box:
1 warning
mgmt: The following components are blocked from being upgraded to 9.1 because they failed to be added to fleet lifecycle. Click each component to view the error and remediation information.
• VCF Operations for networks
• Log management
The task details link shows the cause of the failure and a link to this KB article for remediation steps.

An unexpected error occurred in step discover_vrni_component_task_ref. Reference Code: 127B3D16. Please contact support with this reference code. Detail: I/O error on POST request for "https://[redacted]:443/api/auth/login": Certificate for <[redacted]> doesn't match any of the subject alternative names: []

If SDDC Manager, Fleet lifecycle, SDDC lifecycle, and VCF Operations running versions before 9.1.0.0300, review this section.

After "Deploy VCF Management Component" workflow completes, it will be in a Failed state as shown below.

Upgrade failure in the VCF Operations UI:

Within Build > Tasks > VCF Instances: The Task: Deploy VCF Management Components the Task Status column shows Failed
Example of the failed Import task in the VCF Operations UI:

Within Build > Tasks > VCF Instances > click Deploy VCF Management Components: The Import VCF Operations in Fleet Lifecycle subtask Task Status column shows Failed

Environment

VCF Operations 9.1

Cause

The import step fails for one or more of the following reasons:

Invalid credentials
Invalid certificate
Missing vCenter host
Invalid component IPs

All conditions produce the same symptom. The failed task cannot be retried directly until the invalid data is cleared from VCF Operations.

Resolution

To resolve failures during the "Deploy VCF Management Components" task in a VCF 9.0 to 9.1 upgrade, the attached script must be used to clear invalid metadata and import 9.0 components.

Prerequisites:

Access required:

SSH access to the SDDC Manager appliance

Credentials required:

Credentials	Default Username	Purpose
SDDC Manager admin password	[email protected]	Retrieving component FQDNs
VCF Services Runtime password	[email protected]	Acquiring Fleet lifecycle API JWT
VCF Operations admin password	admin	Discover existing 9.0 components
VCF Automation password	vmware-system-user	Option 2 import
VCF Operations for networks password	admin@local	Option 2 import
Identity Broker password	vmware-system-user	Option 2 import
Log management password	admin	Option 2 import

What the Script Does:

To resolve the issue, use the attached script. The script runs interactively and provides guidance through two options. Only run the option(s) that apply to the version as identified below:

Option 1 — Clear stale data and restart the deploy task
Option 2 — Manually import missing 9.0 components

Only run the option(s) that apply to the version as identified below:

Symptom	Option
Deploy VCF Management Components task completed with warnings	Option 2 only
Deploy VCF Management Components task failed	Option 1, then Option 2

Note: A timestamped log file is written automatically to:

/var/log/vmware/vcf/vcf_remediation_backups/remediation_YYYYMMDD_HHMMSS.log

Option 1 — Clear stale data and restart the deploy task

Run this first when the "Deploy VCF Management Components" task is in a FAILED state.

Notes:

Prompts for credentials once at the start.
Backs up the existing legacy component data before making any changes.
Deletes the invalid legacy LCM data from VCF Operations and waits until the deletion is confirmed.
Triggers a retry of the "Deploy VCF Management Components" task. If the automatic retry does not succeed, the script prints manual instructions.

Option 2 — Manually import missing 9.0 components

Run this if Option 1 is complete (or not needed) and if any 9.0 components are still missing from Fleet.

Shows a status table of which components are already added and which are still missing.
Allows selection of components to add: VCF Automation, VCF Operations for Networks, Identity Broker, and/or Log management.
Runs pre-checks before adding each component and stops if any check fails:
- Verifies the component is reachable over the network.
- For VCF Operations for Networks: checks that the TLS certificate includes a valid SAN entry. If this check fails, the script directs you to KB 440459 to rotate the certificate before proceeding.
- Validates credentials for all components. If the check fails for VCF Automation or Identity broker due to an expired password, the script directs you to KB 424323 to reset the vmware-system-user password before proceeding.
Submits the add component request to Fleet lifecycle for each component that passes all checks.

NOTE:** See Additional Information section if these steps do not resolve the issue for Log management and the task still fails

Remediation Steps:

Note: All steps in this KB must be performed from the SDDC Manager appliance as root.

Download the remediate_deploy_mgmt_components.py script attached to this KB.
Copy the script to the /home/vcf/ directory on the SDDC Manager appliance:
SSH into the SDDC manager appliance with vcf user and then elevate to root with su.
Run the script

python /home/vcf/remediate_deploy_mgmt_components.py

(The script writes a log file to: /var/log/vmware/vcf/vcf_remediation_backups/remediation_YYYYMMDD_HHMMSS.log)
Follow the prompts of the script to identify and remediate any detected issues.

See What the Script Does above for a full description of each option's behavior.

Additional Information

** If after performing Option 2 and the upgrade task is still failing for the Log Management component and there currently is a VCF Operations for Logs 9.0.2 cluster appliance installed, the appliance is likely in a down state due to a certificate mismatch between appliance nodes. A recent upgrade to 9.0.2 might have caused this.

See VCF Operations for Logs UI is not available after upgrade to 9.0.2

Attachments

remediate_deploy_mgmt_components.py get_app