Inventory Sync fails with "TypeError: 'NoneType' object is not iterable" during Brownfield Import in VMware Cloud Foundation
Article ID: 426154
Updated On:
Products
VMware SDDC Manager
VCF Operations
VMware vCenter Server 8.0
Issue/Introduction
- While starting the inventory sync, the task fails with below mentioned error
Message: Import operation for sync with id 17##0697-###-4d95-###-bddb####060 failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vct_brownfield.py sync -- domain-name EXAMPLE.COM -- skip-ssh-thumbprint-validation -- internal-vcf-auth -- output-dir /var/log/vmware/vcf/domainmanager/brownfield/17##0697-###-4d95-###-bddb####060/ -- non-interactive. Please check the log files located /var/log/vmware/vcf/domainmanager/brownfield/17##0697-###-4d95-###-bddb####060/ on the SDDC Manager appliance Remediation Message:Reference Token: #####
- Following log snippets are observed at
/var/log/vmware/vcf/domainmanager/domainmanager.log
YYYY-MM-DDThh:mm:ss.817+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] [YYYY-MM-DD hh:mm:ss,816] [WARNING] local_command_utils: Discovered only ## out of ### SSH keys, retrying. YYYY-MM-DDThh:mm:ss.817+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] [YYYY-MM-DD hh:mm:ss,817] [WARNING] discover_domain: Stopping SSH service on ESXi <FQDN_OF_ESX_HOST> YYYY-MM-DDThh:mm:ss.597+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] [YYYY-MM-DD hh:mm:ss,597] [INFO] sddc_manager_helper: Retrieving SDDC Manager known hosts YYYY-MM-DDThh:mm:ss.607+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] [YYYY-MM-DD hh:mm:ss,607] [INFO] request_helper: Response status from retrieving known hosts: 200 YYYY-MM-DDThh:mm:ss.610+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] [YYYY-MM-DD hh:mm:ss,607] [CRITICAL] vcf_brownfield: Failed to sync domain <example.com> YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] Traceback (most recent call last): YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py", line 984, in sync_domain YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] sync_domain_result: list[str] = sync.sync_domain() YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] ^^^^^^^^^^^^^^^^^^ YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/domain_sync/sync_domain.py", line 60, in sync_domain YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] discovery_model, discovery_errors = self.__run_discovery(sso_remote_endpoint, domain_type, self.interactive, YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/domain_sync/sync_domain.py", line DD2, in __run_discovery YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] discovery_output: DiscoveryModel = discovery.discover_domain(domain_type=domain_type, activating_status=False, YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/domain_discovery/discover_domain.py", line ss39, in discover_domain YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] ssh_keys_are_trusted: bool = TrustSshKeys().establish_trust_for_ssh_keys(product_ssh_keys, SddcManagerHelper( YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/domain_trust/trust_ssh_keys.py", line 26, in establish_trust_for_ssh_keys YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] hosts_to_trust: list[SshKnownHost] = SecurityUtils.compare_known_hosts( YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] File "/opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/common/utils/security_utils.py", line 64, in compare_known_hosts YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] for candidate in known_host_candidates: YYYY-MM-DDThh:mm:ss.611+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] TypeError: 'NoneType' object is not iterable YYYY-MM-DDThh:mm:ss.6ss+0000 INFO [vcf_dm,69695###2f61e4c###298a##46e4b,f9de] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-27] [YYYY-MM-DD hh:mm:ss,6ss] [INFO] vcf_brownfield: Operation sync= completed on target:<example.com> with status: FAIL 00m in 64.55ms YYYY-MM-DDThh:mm:ss.699+0000 ERROR [vcf_dm,69695###2f61e4c###298a##46e4b,6483] [c.v.v.v.s.b.r.BrownfieldRunnerService,dm-exec-35] Brownfield process failed with exit value 1 for operation sync with id 178####97-c39e-####-9c7e-bd####2MM060 YYYY-MM-DDThh:mm:ss.703+0000 DEBUG [vcf_dm,69695###2f61e4c###298a##46e4b,6483] [c.v.v.v.s.b.u.BrownfieldImportProgressReportService,dm-exec-35] Deserialized progress report of the task with id 178####97-c39e-####-9c7e-bd####2MM060: {"process_id":"178####97-c39e-####-9c7e-bd####2MM060","status":"completed with error","progress":{},"total_steps":4,"errors":[{"error_code":"OPERATION_ISSUE","message":"Result status code from set datasource: 400","severity":"ERROR"},{"error_code":"OPERATION_ISSUE","message":"<FQDN_OF_ESX_HOST> SSH disabled. Enabling the service temporarily in order to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXi <FQDN_OF_ESX_HOST> has SSH disabled. Enabling the service temporarily in order to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXi <FQDN_OF_ESX_HOST> has SSH disabled. Enabling the service temporarily in order to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXi <FQDN_OF_ESX_HOST> has SSH disabled. Enabling the service temporarily in order to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXi <FQDN_OF_ESX_HOST> has SSH disabled. Enabling the service temporarily in order to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXi <FQDN_OF_ESX_HOST> has SSH disabled. Enabling the service temporarily in order to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXi <FQDN_OF_ESX_HOST> has SSH disabled. Enabling the service temporarily in order to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXi <FQDN_OF_ESX_HOST> has SSH disabled. Enabling the service temporarily in order to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"ESXi <FQDN_OF_ESX_HOST> has SSH disabled. Enabling the service temporarily in order to discover the host SSH key","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Discovered only 17 out of 18 SSH keys, retrying...","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Discovered only ## out of ## SSH keys, retrying...","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Discovered only ## out of ## SSH keys, retrying...","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXi <FQDN_OF_ESX_HOST>","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXi <FQDN_OF_ESX_HOST>","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXi <FQDN_OF_ESX_HOST>","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXi <FQDN_OF_ESX_HOST>","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXi <FQDN_OF_ESX_HOST>","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXi <FQDN_OF_ESX_HOST>","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXi <FQDN_OF_ESX_HOST>","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXi <FQDN_OF_ESX_HOST>","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Stopping SSH service on ESXi <FQDN_OF_ESX_HOST>","severity":"WARNING"},{"error_code":"OPERATION_ISSUE","message":"Failed to sync domain <EXAMPLE.COM>","severity":"ERROR"}]} YYYY-MM-DDThh:mm:ss.710+0000 ERROR [vcf_dm69695###2f61e4c###298a##46e4b,6483,6483] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-35] [REFERENCE TOKEN] IMPORT_OPERATION_FAILED Import operation for sync with id 178####97-c39e-####-9c7e-bd####2MM060 failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name <Example.com> --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/178####97-c39e-####-9c7e-bd####2MM060/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/17860697-####-4d95-####-bddb####060/ on the SDDC Manager appliance com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Import operation for sync with id 17860697-####-4d95-####-bddb####060 failed internally. Command was python3 /opt/vmware/vcf/domainmanager/scripts/vcf-import-tool/vcf_brownfield.py sync --domain-name <Example.com> --skip-ssh-thumbprint-validation --internal-vcf-auth --output-dir /var/log/vmware/vcf/domainmanager/brownfield/17860697-####-4d95-####-bddb####060/ --non-interactive. Please check the log files located in /var/log/vmware/vcf/domainmanager/brownfield/17860697-####-4d95-####-bddb####060/ on the SDDC Manager appliance at com.vmware.vcf.vimanager.services.brownfield.runner.BrownfieldRunnerService.waitForProcess(BrownfieldRunnerService.java:106) at com.vmware.vcf.vimanager.services.brownfield.runner.BrownfieldRunnerService.runBrownfieldProcess(BrownfieldRunnerService.java:61) - From
/var/log/vmware/vcf/domainmanager/brownfield/41###be-####-426b-####-ffc####36449/output/progress_report_178####97-c39e-####-9c7e-bd####2MM060.json
{ "process_id": "17####0697-####-4d95-####-bdd####MM060", "status": "completed with error", "progress": {}, "total_steps": 4, "errors": [ { "error_code": "OPERATION_ISSUE", "message": "Result status code from set datasource: 400", "remediation_message": null, "severity": "ERROR" }, { "error_code": "OPERATION_ISSUE", "message": "<FQDN_OF_ESX_HOST> has SSH disabled. Enabling the service temporarily in order to discover the host SSH key", "remediation_message": null, "severity": "WARNING" }, { "error_code": "OPERATION_ISSUE", "message": "Stopping SSH service on ESXi<FQDN_OF_ESX_HOST>", "remediation_message": null, "severity": "WARNING" }, { "error_code": "OPERATION_ISSUE", "message": "Failed to sync domain <Example.com>", "remediation_message": null, "severity": "ERROR" } ] }
Environment
VCF 9.X
Cause
The primary cause of this failure is that the SSH service is disabled on one or more ESXi hosts managed by the vCenter Server being imported.
During inventory sync, the VCF Import Tool attempts to discover and synchronize SSH trusted keys across the environment to establish security guardrails. While the script attempts to temporarily enable SSH on hosts where it is disabled, if the discovery of these keys fails or returns a null value for a host, the Python script encounters a NoneType error when attempting to iterate through the list of "known hosts." This causes the entire import synchronization task to crash.
Resolution
To resolve this issue and ensure a successful Inventory sync, the SSH service must be manually enabled and verified on all ESX hosts within the target vCenter before starting the import process
-
Identify Affected Hosts: Review the logs or the progress report JSON mentioned in the Issue section to identify which specific ESX hosts reported "SSH disabled."
-
Enable SSH Service:
-
Log in to the vSphere Client.
-
Navigate to each ESXi host > Configure tab.
-
Under System, select Services.
-
Locate the SSH service and click Start.
-
(Optional) Set the Startup Policy to "Start and stop with host" temporarily to ensure it remains active during the migration window.
-
-
Verify Connectivity: Ensure the SDDC Manager can reach the ESXi hosts over port 22.
-
Restart the Domain inventory sync task : Once SSH is confirmed as "Running" on all hosts within the cluster/domain being imported, restart the inventory process from the SDDC Manager UI.
Feedback
Yes
No
Powered by
