Unable to sync management domain inventory after 9.0.2 upgrade

Products

• VCF Operations
• VMware SDDC Manager / VCF Installer

Issue/Introduction

The management domain inventory sync fails after upgrading VCF Operations to version 9.0.2.

The Fleet Management > Tasks UI shows the "Sync domain" task failing with the following details:

• Task Status: Failed

• Subtask: Synchronize domain inventory

• Progress Messages: Import operation for sync with id <UUID> failed internally...

• Error Message: Message: Import operation for sync with id <UUID> failed internally...Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/<UUID>/ on the SDDC Manager appliance

Review of the SDDC Manager logs confirms the failure:

/var/log/vmware/vcf/domainmanager/brownfield/<UUID>/vcf_brownfield.log

Aborting operation as SSH keys have been rejected by the user
Failed to sync domain

/var/log/vmware/vcf/fixHostKeys.log

Error for <host_fqdn> with algo: ssh-rsa : Incompatible ssh peer (no acceptable host key)

An attempt was made to run the script from KB:  How to update the SSH host keys on the SDDC Manager

Environment

VCF Operations 9.0.x
SDDC Manager 9.0.x

Cause

SDDC is trying to establish a secure connection to all the nodes in the management domain, but it’s failing due to one or more SSH keys not validating / updating with the node's FQDN.

In an attempt to run the fix_known_hosts.sh remediation script from the KB, if it was executed using a lowercase FQDN for the management domain vCenter/Hosts, the script likely silently failed. The node name in the vCenter inventory is case-sensitive and listed with an UPPERCASE shortname (e.g., HOST.domain.com). This mismatch prevents the script from running properly and SSH keys from being correctly updated for the specific entity required by the sync process.

Resolution

To resolve this issue, execute the fix_known_hosts.sh script using the exact casing displayed in the vCenter inventory.
Reference:  How to update the SSH host keys on the SDDC Manager

1. Verify the casing of the host/node name in the vCenter inventory (e.g., HOSTNAME.domain.com vs hostname.domain.com).
   
   NOTE: You can also check all the nodes SDDC Manager is trying to reach during the sync task from the output file. The <UUID> for the directory and file changes after each task retry attempt
   
   /var/log/vmware/vcf/domainmanager/brownfield/<UUD>/output/progress_report_<UUID>.json    
   
   
   

2. Run the fix_known_hosts.sh script using the exact casing found in step 1.

3. Once the script completes successfully, retry the inventory sync task.

Additional Information

How to update the SSH host keys on the SDDC Manager