VCF 9.1 Deployment Fails at 'Deploy and configure VCF Automation' due to in-use IP address
search cancel

VCF 9.1 Deployment Fails at 'Deploy and configure VCF Automation' due to in-use IP address

book

Article ID: 450292

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer VCF Automation

Issue/Introduction

  • VCF Installer shows below error,



  • Below error reported on VCF Installer 

    A The Fleet lifecycle 'VSP component installation' task with ID #####-####-####-####-############ failed. Internal errors: Reference Token: O2BP9J
    A The Fleet lifecycle 'VSP component installation' task with ID #####-####-####-####-############ failed. Internal errors: A pre-validation prevented the bootstrap: From the IPv4 pool, found the following in-use IP address(es): <in-use_ip_list>, Failed to bootstrap VCF
  • /var/log/vmware/vcf/domainmanager/domainmanager.log on VCF installer shows below error,

    YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,################################,####] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-####]  Contract task InstallVspComponentFleetLcmAction input: {"_executionUuid":"<execution-UUID>","componentSpec":{"deploymentType":"VspComponentSpec","sddcLcmId":"<LCM OP ID>","componentType":"VCFA","version":"9.1.0.0200.25556825","repository":{"downloadUrl":"https://VCF_Automation_FQDN/vmware/vcf/bundle/<bundle ID>/<bundle ID>/vcfa-bundle-9.1.0.0200.25556825.tar"},"fqdn":"VCF_Automation_FQDN","componentNamespace":"prelude","configSpec":{"size":"small","adminSystemPassword":"*****","VCD_MIGRATOR":{"deploymentType":"VspComponentSpec","sddcLcmId":"<LCM OP ID>","componentType":"VCD_MIGRATOR","version":"9.1.0.0200.25556825","repository":{"downloadUrl":"https://VCF_Automation_FQDN/vmware/vcf/bundle/<bundle ID>/<bundle ID>/vcd-migrator-9.1.0.0200.25556825.tgz"},"componentNamespace":"vcd-migrator","configSpec":{"size":"small"}}},"vspClusterSpec":{"deploymentType":"VspClusterSpec","sddcLcmId":"<LCM OP ID>","version":"9.1.0.0200.25555874","platformFqdn":"FQDN","systemUserPassword":"*****","size":"small","ipAddresses":[<in_use-IP_range>]}}}

 

Environment

VCF 9.1

Cause

VCF Automation deployment requires unique available IP address. This error occurs if the IP address provided was already in-use or resolves to an IP address that is already active on the network, triggering a validation failure at the time VCF Automation deployment.

Resolution

 The existing configuration has to be re-initiated post updating new IP range to VCF Automation on runtime workflow json spec. Follow below steps in order to resolve the issue.

  1. Take a snapshot of VCF Installer

  2. Login to Management vCenter UI and delete VMs under folder "vcf-automation" 

  3. Log in to SDDC Manager machine using vcf user and switch to root user

  4. Execute the below command to export the json for the current configuration using execution_UUID retrieved from /var/log/vmware/vcf/domainmanager/domainmanager.log, (or you can refer KB article 314620)

    echo vcf.vault.http-access=true >> /etc/vmware/vcf/domainmanager/application.properties && systemctl restart domainmanager

    curl -X GET http://localhost/domainmanager/internal/vault/<execution_UUID> | json_pp > /tmp/workflow.json

  5. Edit /tmp/workflow.json and replace with new IP address which is not in use under vcfAutomationSpec within java.util.ArrayList as below,

    "vcfAutomationSpec" :[
    "com. vmware. vcf.rest. api.model. v1. sddcs.VcfAutomationSpec",
    "adminUserPassword" : "####################",
    "hostname" : "<VCF_Automation_FQDN>",
    "internalClusterCidr" : "<CIDR>",
    "ipPool" : [
    "java.util.ArrayList",
    "<New_IP_address_1>",
    "<New_IP_address_2>",
    "<New_IP_address_3>",
    "<New_IP_address_4>",
    "<New_IP_address_5>"
    ]

  6. Update the vault with the modified JSON

    curl -H 'Content-Type:text/plain' -X PUT http://localhost/domainmanager/internal/vault/<execution_uuid> -d @/tmp/workflow.json

  7. Restart the "Deploy VCF Management Components" Task

    1. In the VCF Operations UI browse to Build > Lifecycle > Tasks

    2. Locate the most recent Failed status Task for "Deploy and configure VCF Automation"

    3. Click on the failed task and copy the Task ID (UUID format) from the task details

    4. SSH to the SDDC Manager appliance as the vcf user and switch to root user

    5. Open the application-prod.properties file for editing with the following command

      vi /etc/vmware/vcf/domainmanager/application-prod.properties

    6. Add or update the following entry, replacing <Task_ID from step c> taken from Step 3 above and save the file

      fleetlcm.task.id.to.recreate=<Task_ID from step c>

    7. Restart domain manager service on the SDDC manager appliance using the following command

      systemctl restart domainmanager

    8. Return to the VCF OPs UI, navigate to Build > Tasks, locate the "Deploy VCF Management Components" task and click Restart Task

  8. Once the task is completed, revert the changes made to the application.properties file and restart the service on SDDC manager,

    sed -i '/vcf.vault.http-access=true/d' /etc/vmware/vcf/domainmanager/application.properties && systemctl restart domainmanager
Powered by Wolken Software