Sync Inventory from VCF operation fails - Import operation for sync with id ##### failed internally - SSH keys have been rejected by the user
search cancel

Sync Inventory from VCF operation fails - Import operation for sync with id ##### failed internally - SSH keys have been rejected by the user

book

Article ID: 437355

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

Message: Import operation for sync with id ###-###-###-### failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name <domain_name> --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/###-###-###-###/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/###-###-###-###/ on the SDDC Manager appliance
Remediation Message:
Reference Token: ####
Cause:

 

  • Navigate to the log location present in the error message. You will find the sync failed with rejected SSH Keys
    /var/log/vmware/vcf/domainmanager/brownfield/###-###-###-###/
[yyyy-mm-dd hh:mm:ss]  [CRITICAL] discover_domain:1442: Aborting operation as SSH keys have been rejected by the user. Operation can be retried with the same command line arguments.
[yyyy-mm-dd hh:mm:ss]  [CRITICAL] vcf_brownfield:1007: ESC[91mFailed to sync domain DOMAIN-NAME ESC[00m
Traceback (most recent call last):
  File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py", line 984, in sync_domain
    sync_domain_result: list[str] = sync.sync_domain()
                                    ^^^^^^^^^^^^^^^^^^
  File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/domain_sync/sync_domain.py", line 68, in sync_domain
    actual_data: DomainInventory = self.__load_domain_inventory_from_discovery_model(discovery_model)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/domain_sync/sync_domain.py", line 160, in __load_domain_inventory_from_discovery_model
    discovery_model.get_inventory_payload()[SyncConstants.DomainInventoryFieldsConstants.DOMAIN_INVENTORY_KEY])
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get_inventory_payload'
[yyyy-mm-dd hh:mm:ss]  [INFO] vcf_brownfield:633: Operation ESC[93msyncESC[00m completed on target: ESC[93mDOMAIN-NAME ESC[00m with status: ESC[93mFAILESC[00m in ESC[93m22.53ESC[00ms
(END)

 

  • In the domain manager logs: /var/log/vmware/vcf/domainmanager/domainmanager.log

[yyyy-mm-dd hh:mm:ss]  DEBUG Deserialized progress report of the task with id ####-####-####-####: {"process_id":"","status":"completed with error","progress":{},"total_steps":4,"errors":[{"error_code":"OPERATION_ISSUE","message":"Result st
atus code from set datasource: 400","severity":"ERROR"},{"error_code":"OPERATION_ISSUE","message":"ESXI-HOST has SSH disabled. Enabling the service temporarily in order
to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXI-HOST has SSH disabled. Enabling the service temporarily in order to d
iscover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXI-HOST has SSH disabled. Enabling the service temporarily in order to disco
ver the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXI-HOST has SSH disabled. Enabling the service temporarily in order to discover
the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXI-HOST","severity":"WARNING"},{"error_code":"OPERATION_ISSU
E","message":"Stopping SSH service on ESXI-HOST","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXI-HOST
ager.cmspod.net","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXI-HOST","severity":"WARNING"},{"error_code":"OPERATION_ISSUE
","message":"Aborting operation as SSH keys have been rejected by the user. Operation can be retried with the same command line arguments.","severity":"ERROR"},{"error_code":"OPERATION_ISSUE","message":"Fai
led to sync domain WLD01","severity":"ERROR"}]}
[yyyy-mm-dd hh:mm:ss]  ERROR [vcf_dm,] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-10]  [5N1DVI] IMPORT_OPERATION_FAILED Import operation for sync with id ####-####-####-##### failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name <DOMAIN-NAME> --skip-ssh-thumbprint-validation --intern
al-vcf-auth --import-edge-clusters --accept-edge-password-reset --output-dir /var/log/vmware/vcf/domainmanager/brownfield/######-#####-#####-######/ --non-interactive. Please check the log files
located in /var/log/vmware/vcf/domainmanager/brownfield/######-#####-#####-######/ on the SDDC Manager appliance
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Import operation for sync with id ######-#####-#####-###### failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scr
ipts/vcf-import-tool/vcf_brownfield.py sync --domain-name <DOMAIN-NAME> --skip-ssh-thumbprint-validation --internal-vcf-auth --import-edge-clusters --accept-edge-password-reset --output-dir /var/log/vmware/vcf/doma
inmanager/brownfield/######-#####-#####-######/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/######-#####-#####-######/ on the
SDDC Manager appliance

Environment

VCF 9.x

Cause

 This issue occurs due to stale or untrusted SSH host keys. There is a discrepancy between the SSH known hosts maintained by the SDDC Manager and the actual keys present on the ESXi hosts.

Resolution

 

To resolve this issue, update the SSH host keys on the SDDC Manager using the fixHostKeys.py script.

1. Take a snapshot of the SDDC Manager VM.
2. Log in to the SDDC Manager via SSH using the vcf user account and switch to the root user.
3. Update the SSH host keys by following the instructions and leveraging the fixHostKeys.py script provided in the KB article How to update the SSH host keys on the SDDC Manager
4. After successfully updating the SSH host keys, retry the Inventory Sync task from VCFOps UI.

 

NOTE: If the error persists even after executing the fixHostKeys.py script, use the fix_known_hosts.sh bash script as an alternative remediation method.

Powered by Wolken Software