VCF 9.x Brownfield Domain Import fails with error: vCenter with the name <vCenter-fqdn> not found in the inventory
search cancel

VCF 9.x Brownfield Domain Import fails with error: vCenter with the name <vCenter-fqdn> not found in the inventory

book

Article ID: 446522

calendar_today

Updated On:

Products

VCF Operations VMware vCenter Server VMware SDDC Manager / VCF Installer

Issue/Introduction

  • During a brownfield environment import into VMware Cloud Foundation (VCF) 9.x, the domain import workflow fails at a post-import orchestration stage.
  • Specifically, the process fails at the "Generate inputs for configuring NSX Manager after Import" task, even though the vCenter Server reflects an active status within the SDDC Manager and VCF inventory.
  • The Tasks under VCF Ops  Manager user interface displays a failure on the following task:

    Task Name: Generate inputs for configuring NSX Manager after Import
    Status: Failed
    Error Message: Message: A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.
    Reference Token: ######
    Underlying Cause Listed in UI: Cause: vCenter with the name <vCenter-fqdn> not found in the inventory



  • Under the VCF Ops Inventory, the vCenter and other components are visible.

  • Checking the status of the vCenter and Domains in the SDDC DB are all in ACTIVE state, which is as follows:

  • Under SDDC Manager's /var/log/vmware/domainmanager/domainmanager.log, below log snips are found (which is failing to check the inventory with lowercase of the vCenter):

    YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,############,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-##]  [TOKEN] VCF_ERROR_INTERNAL_SERVER_ERROR Invocation of prefix '' part of task GeneratePostNsxImportConfigurationsInputAction in plugin ViDomainServicePlugin failed with exception.
    com.vmware.evo.sddc.common.core.error.InternalServerErrorException: Invocation of prefix '' part of task GeneratePostNsxImportConfigurationsInputAction
     in plugin ViDomainServicePlugin failed with exception.
            at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:450)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at java.base/java.lang.Thread.run(Thread.java:840)
    Caused by: java.lang.RuntimeException: vCenter with the name <vcenter-fqdn.example.com> not found in the inventory
            at com.vmware.vcf.vimanager.fsm.actions.GeneratePostNsxImportConfigurationsInputAction.lambda$getVcenterInventoryData$9(GeneratePostNsxImportConfigurationsInputAction.java:274)

  • Under SDDC Manager's /var/log/vmware/vcf/domainmanager/brownfield/<UUID>/output/guardrails_report_vcenter-fqdn.example.com_all.csv, below logs are found (which are all successful, but with lowercase of the vCenter):

    "NsxtCluster","<nsx-fqdn.example.com>","NSX Import","INFO","VALIDATION_SUCCESSFUL","Multi-NSX compute managers","Checks if the NSX has multiNsx compute managers related to the vCenter","The NSX does not have multiNsx compute managers for vCenter vcenter-fqdn.example.com",""
    "NsxtCluster","<nsx-fqdn.example.com>","NSX Import","INFO","VALIDATION_SUCCESSFUL","Import NSX-vCenter compute manager","Check if NSX has a compute manager registered for the vCenter for import","NSX has a compute manager for vCenter vcenter-fqdn.example.com",""
    "VcManager","vcenter-fqdn.example.com","Import","INFO","VALIDATION_SUCCESSFUL","vCenter Server Connection Latency","Check if connection latency to vCenter Server is within acceptable limits","vCenter Server latency is within acceptable limits",""

  • The vCenter, that needs to be imported is configured with UPPERCASE Hostname and PNID.

    Example output:

    /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost && hostname -f

    VCENTER-FQDN.EXAMPLE.COM
    VCENTER-FQDN.EXAMPLE.COM

Environment

  • VCF 9.x.
  • vCenter 8.x / 9.x.
  • SDDC Manager.

Cause

  • The failure is caused by a case-sensitivity mismatch.
  • While the vCenter Server's hostname and Primary Network Identifier (PNID) are natively set with specific character casing, importing it into VMware Cloud Foundation (VCF) Operations often results in a mismatch if the provided hostname uses conflicting UPPERCASE or lowercase.
  • The SDDC Manager database lookup enforces strict case-sensitivity, preventing it from finding the lowercase string in uppercase inventory tables.

Resolution

To resolve this issue, follow below steps:

  1. Take snapshot of the SDDC Manager.
  2. Note the workflow ID from the failed task's URL in the SDDC Manager UI.
  3. SSH to SDDC Manager as vcf user and then su root.
  4. Update the applications.properties file to all access, using following command:

    echo "vcf.vault.http-access=true" >> /etc/vmware/vcf/domainmanager/application.properties

  5. Restart the domainmanager service, using following command:

    systemctl restart domainmanager

  6. Retrieve Workflow Spec, using following command:

    curl -k -s http://localhost/domainmanager/internal/vault/<workflow_id> -X GET | json_pp > /tmp/workflow.json

  7. Correct all instances of the lowercase vCenter FQDN to UPPERCASE Hostname in /tmp/workflow.json file.
  8. Update the spec file for the workflow, using following command:

    curl -k -s http://localhost/domainmanager/internal/vault/<workflow_id> -X PUT -H "Content-type: text/plain" -d @/tmp/workflow.json

  9. From the SDDC Manager UI, Click on the RESTART TASK to restart the failed task.

Additional Information

For more information, refer: Re-try an existing workflow by modifying the workflow spec file

Powered by Wolken Software