VCF Automation upgrade pre-check fails when SSH passwords differ between the vRA 8.x source nodes

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When upgrading from vRA 8.18.1 to VCF Automation (VCFA) 9.1.x, the upgrade pre-check phase runs a set of source-system validation checks.
If the source system is a multi-node (HA) vRA 8.18.1 deployment and the cluster appliances do not all share the same root password, the script sync fails on the appliances whose password differs from the node that was imported into Fleet LCM.
The source validation pre-check then aborts and reports a misleading product-identification failure, even though the source system is in fact a supported product/version.
This is commonly seen when SDDC Manager manages the source appliances and intentionally keeps a different root password on each appliance.

Symptoms

The upgrade fails during the source validation pre-check (vcfa-precheck-upgrade-80-source). The reported pre-check error:

The source system is not a VMware Aria Automation (vRA) 8.x product.
[com.broadcom.vcfa.upgrade.80.source.check.test-source-01-8x-product-identification]

In the pre-check pod logs on the VCF services runtime control-plane node (kubectl -n prelude logs <vcfa-precheck-...-source-...>), the underlying failure is

an SSH script-sync error against one or more source cluster nodes, for example:

[INFO] Syncing scripts (checksum: <hash>) to 3 nodes: <ip-1> <ip-2> <ip-3>
[ERROR] Failed to create directory on <ip-2>: ... Permission denied, please try again.
[ERROR] Failed to create directory on <ip-3>: ... Permission denied, please try again.
[ERROR] Failed to sync scripts to 2 node(s). Successfully synced to 1 node(s).
[ERROR] SYSTEM ERROR: Failed to execute ... on source system (SSH/execution failure)
Only the node that was imported into Fleet LCM (whose credentials are in the upgrade payload) is synced successfully; the other appliances reject the SSH credentials with Permission denied.

Environment

Target system: VCF Automation 9.1.0, VCF Automation 9.1.0 EP1, VCF Automation 9.1.0 EP2,
Source system: VMware Aria Automation (vRA) 8.18.x clustered deployments or
Source cluster appliances configured with different root passwords per node (e.g. managed by SDDC Manager)

Cause

The upgrade pre-check library discovers all source cluster node IP addresses (via kubectl get nodes) and attempts to SSH-sync the helper scripts to every node in parallel, using the single set of credentials supplied for the node that was imported into Fleet LCM.
The same ssh password is a requirement for the 9.x source systems, as they need to have all scripts on each node.
However, this is not the case for 8.x source systems.
When the appliances have different root passwords, only the imported node accepts those credentials. The sync to the remaining appliances fails with Permission denied, which causes the overall sync — and therefore the source product-identification pre-check — to fail.
Because the pre-check could not run its source script, it returns a generic "not a supported product/version" result, which does not reflect the true root cause (an SSH authentication failure to the other cluster nodes).

Resolution

Resolution
A code fix will be included in VCF Automation 9.1.0 EP3. With the fix, the upgrade pre-check syncs the helper scripts only to the single source node that was imported into Fleet LCM, instead of to all discovered cluster nodes. This removes the dependency on every source appliance sharing the same root password.

Workaround

Identify the root password of the source appliance that was imported into Fleet LCM (the node that passes the pre-check).
Set the root password on all other source cluster appliances to match that password, so that every appliance in the source cluster shares the same root password.
Re-run the upgrade pre-check / restart the upgrade workflow from Fleet LCM.
Confirm the source validation pre-check (vcfa-precheck-upgrade-80-source or the 9.x equivalent) now completes successfully.