Deploying new vCenter Server Appliance fails with error "vCenter installation failed. Check logs under /var/log/vmware/vcf/domainmanager/ci-installer-<date and log ID> for more details. Reference Token: <ID>"

Products

• VMware Cloud Foundation

Issue/Introduction

• VMware Cloud Foundation deployment fails to deploy new vCenter Server Appliance with error "Deploy vCenter Server" stage with error "vCenter installation failed. Check logs under /var/log/vmware/vcf/domainmanager/ci-installer-<date and log ID> for more details. Reference Token: <ID>"
  
  
• The /var/log/vmware/vcf/domainmanager/ci-installer.log of the VCF Installer Appliance show log entries below.
  
  YYYY-MM-DD <TIME> - vCSACliInstallLogger - INFO - ==========VCSA Deployment Progress Report==========
        Task: Install required RPMs for the appliance.(SUCCEEDED 100/100)       - Task has completed successfully.
        Task: Run firstboot scripts.(FAILED 0/100)      - Starting ...
                Error: The supplied System Name <VC FQDN/short name> is not valid.
                Resolution: If the supplied system name is a FQDN, then make sure the DNS forward lookup results in at least one valid IP address in the system. If the supplied system name is an IP address, then it should be one of the valid IP address(es) in the system.
  
  
• There are no issues with the DNS resolution of the vCenter Server FQDN. Both forward and reverse lookup towards the FQDN and IP Address works fine.
  
  
• The DNS server itself is reachable from both the installer and the ESXi Host where the VCSA virtual machine is getting deployed.
  
  
• On further investigating the /var/log/vmware/vcf/domainmanager/ci-installer-<folder ID>/workflow_<folder ID>/ci-conf-<folder ID>/vcsa_deployment.json file, it confirms that when running the first boot scripts post RPM installations, the installer attempted to resolve and validate the PNID of the vCenter Appliance. Under the "args" section of the log is where it confirms the PNID value it expects. The same can either be the vCenter server FQDN or the short name. 
  
        "key": "firstboot",
      "value": {
        "cancelable": false,
        "description": {
          "args": [],
          "default_message": "Run firstboot scripts.",
          "id": "com.vmware.vcenter.deploy.task.description.firstboot"
        },
        "end_time": "YYYY-MM-DDT<time>Z<timezone>",
        "operation": "",
        "progress": {
          "completed": 0,
          "message": {
            "args": [
              ""
            ],
            "default_message": "Starting ...",
            "id": "install.ciscommon.component.starting"
          },
          "total": 100
        },
        "result": {
          "errors": [
            {
              "id": "install.ciscommon.validatePNID",
              "message": {
                "args": [
                  "<vCenter server FQDN/short name>"
                ],
                "default_message": "The supplied System Name <vCenter Server FQDN/short name> is not valid.",
                "id": "install.ciscommon.validatePNID.error"
              },
              "resolution": {
                "args": [],
                "default_message": "If the supplied system name is a FQDN, then make sure the DNS forward lookup results in at least one valid IP address in the system. If the supplied system name is an IP address, then it should be one of the valid IP address(es) in the system.",
                "id": "install.ciscommon.validatePNID.resolution"
              }
            }
          ]
        },
        "service": "",
        "start_time": "YYYY-MM-DDT<time>Z<timezone>",
        "status": "FAILED"
      }
    }
  ]
}

Environment

VMware Cloud Foundation 9.x

Cause

By design, the first boot script gets the PNID which can either be the FQDN or the short name. It then performs a forward lookup on this PNID. In case the PNID returned is the short name which is not resolvable by the DNS server considering the DNS forward and reverse lookup was only setup for the complete FQDN, this validation error is seen.

To confirm the value of appliance PNID

• SSH to the newly deployed vCenter Server Appliance.
  
  
• Run the following command which should return the PNID of the appliance.
  root@vi-vcenter-1 [ ~ ]# install-parameter appliance.net.pnid

Resolution

Ensure that forward and reverse lookup resolution works for both the FQDN and short name of the newly deployed vCenter Server Appliance.