NSXT Manager deployment using the cloud builder fails with error "Duplicate Network Address Detected for Requested Virtual IPv4 Address"

Products

• VMware NSX

Issue/Introduction

• The error, "Duplicate Network Address Detected for Requested Virtual IPv4 Address" can be seen while attempting to deploy the NSX Manager VIP.
• It is confirmed and verified that the IP is not duplicate. 

Cause

This issue is specific to the SoftLayer data center fabric. If ARP is sent for an IP address, a MAC will be returned back (from the fabric) even though the IP is not in use.  This appears to have been the behavior with Softlayer fabric for a long time and is not easily changeable.  

In NSX 4.x, NSBU added a check to ensure the LB VIP is not already used. However, in the SoftLayer environment, it sees the IP has an associated MAC and thinks it is in use. NSBU has provided a 'force=true' parameter which can be added to the API call to force setting the IP on the LB.   

Resolution

Workaround:

1. Once the Workflow fails at setting VIP, then set the VIP using NSXM manager API, Set VIP using NSXM API

   #curl -k -u 'admin:<nsx-manager-password>' -X POST "https://<nsx-manager-ip-or-fqdn>/api/v1/cluster/api-virtual-ip?action=set_virtual_ip&ip_address=<virtual-ip-address>&force=true"

2. Get the failed workflow execution id
   • Get Bringup Exec Id
     • open the logs /var/log/vmware/vcf/bringup/vcf-bringup-debug.log Get the execution Id of the Failed workflow. or,

       curl -X GET -u'admin:######' https://<cloud-builder-ip-address>/v1/sddcs -k |json_pp > bringup.json

       From the above json, get the id field
3. Skip the Failed Task by executing the following commands in SQL
   1. Take a ssh to the cloudbilder VM 

      ssh admin@<cloud-builder-ip>

   2. Elevate to root user

      #su

   3. Take a backup of bring database

      # pg_dump -U postgres -h localhost bringup > bringup-db.dump

   4. Login to the postgres database

      #psql -U postgres -h localhost

   5. Execute the following commands in the SQL prompt

      #create table processing_task_bak as select * from processing_task;
      
      #create table processing_context_bak as select * from processing_context;
      
      #update processing_task set status='POSTVALIDATION_COMPLETED_WITH_SUCCESS' where task_name='SetVipForNsxtManagersAction' and execution_id='87bee553-e332-4282-#####-############';
      
      #update processing_context set status='PENDING' where processing_state_name='ConfigureNsxtSecurityCertificates' and execution_id='87bee553-e332-4282-#####-############';
      
      #update processing_task set status='COMPLETED_WITH_FAILURE' where task_name='IssueVMCACertsForNsxtManagerAction' and execution_id='87bee553-e332-4282-#####-############';

      Note: In the above commands, the execution Id '87bee553-e332-4282-#####-#############' is an example. Please get the Execution Id from Step-2
4. Retry the Failed Workflow which proceeds to the Next action IssueVMCACertsForNsxtManagerAction

 

 

Additional Information

Impact/Risks:

NSXT Manager deployment gets stuck.