Creating an NSX-T VI workload domain in VMware Cloud Foundation 3.5.x fails

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Creating an NSX-T VI workload domain in VMware Cloud Foundation 3.5.x fails on the "Attach pnics and vmknics to nvds" task or on the "Join vSphere hosts to NSX-T Fabric" task.
• One or more hosts in the NSX-T VI workload domain does not have a physical NIC named vmnic0.
• You see a message similar to the following in the /var/log/vmare/vcf/domainmanager/domainmanager.log file on the SDDC Manager VM:

2019-02-13 22:11:02.292 [vcf_dm,075e83c31e63551f,bc7accecc6341c84] [2-thread-7] ERROR [c.v.e.sddc.orchestrator.model.error.ErrorFactory]   [B44V60] UNABLE_TO_CONFIGURE_HOST_FABRIC Unable to configure fabric for  on nsxmanager-2.vcf.corp.local
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Unable to configure fabric for  on nsxmanager-2.vcf.corp.local

Caused by: com.vmware.vapi.std.errors.InvalidRequest: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) => {
    messages = [],
    data = struct => {error_message=Physical NIC vmnic0 was not found on the host., httpStatus=BAD_REQUEST, error_code=9508, module_name=NsxSwitching service}
}

2019-02-13 23:08:39.603 [vcf_dm,1a037c291fa07497,535a14f875faca8a] [2-thread-9] ERROR [c.v.v.c.f.p.n.a.AttachPnicsAndVmknicsToNvdsAction]   UNABLE_TO_UPDATE_TRANSPORT_NODE
java.lang.RuntimeException: Partial physical nic configuration found present. Pnic array: [Pnic (com.vmware.nsx.model.pnic) => {
    deviceName = vmnic4,
    uplinkName = uplink-1
}], Input Vmnics: [vmnic0, vmnic5], Input Uplinks: [uplink-1, uplink-2]

java.lang.IllegalArgumentException: Number of vmnics and nvds uplinks do not match
 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 3.5.x

Resolution

This is a known issue affecting VMware Cloud Foundation 3.5.x. This issue is resolved in VMware Cloud Foundation 3.7.

Workaround:
Use the following steps to workaround this issue:

1. In vsphere client, go to the Networking view.
2. Expand the new vCenter Server, expand the Datacenter, right-click the Distibuted Switch, select Add and manage hosts.
3. Select Manage host network, click NEXT.
4. Click the Attached hosts button, select all affected hosts, click OK.
5. Click NEXT.
6. For each affected host, select one uplink (the lowest numbered one that is not named vmnic1) and then click the Unassign adapter link.
7. Click NEXT.
8. Click NEXT.
9. Click NEXT.
10. Click FINISH.
11. In the NSX Manager UI, go to Dashboard, click on Hosts.
12. If the failed task was "Join vSphere hosts to NSX-T Fabric", stay on the Hosts tab and check the box next to one of the affected hosts. Click the Actions dropdown and choose Configure as Transport Node.

Note: If the failed task was "Attach pnics and vmknics to nvds", skip to Step 24.

13. Enter the host's FQDN in the Name field.
14. Check the box next to the overlay transport zone in the list of Available Transport Zones. Click the right arrow button to move the overly transport zone into the Selected list.
15. Click on the N-VDS tab.
16. Click the dropdown next to N-VDS Name and select the available N-VDS.
17. Click the dropdown next to NIOC Profile and select nsx-default-nioc-hostswitch-profile.
18. Click the dropdown next to Uplink Profile and select nsx-default-uplink-hostswitch-profile.
19. Click the dropdown next to IP Assignment and select Use DHCP.
20. Click the dropdown next to Physical NICs and choose the NIC that was unassigned in Step 6.
21. Click the ADD button.
22. Repeat Steps 12 through 21 for any other affected hosts.
23. Click the RESTART TASK button next to the failed workflow in SDDC Manager.

Note: It is very likely that the "Attach pnics and vmknics to nvds" will fail as well. Refer to the following section to resolve that issue.

24. If the failed task was "Attach pnics and vmknics to nvds", go to the Transport Nodes tab. Check the box next to one of the affected hosts. Click the EDIT button.
25. Click on the N-VDS tab.
26. Click the dropdown next to Physical NICs and choose the NIC that was unassigned in Step 6.
27. Click the ADD button.
28. Repeat Steps 24 through 27 for any other affected hosts
29. Expand the Tasks pane in the SDDC Manager UI. Click on the Creating domain <domain name> task.

Note: A URL similar to the following is displayed in the browser:

https://sddc-manager.vcf.corp.local/ui/sddc-manager/administration/details/vrealize/vrealize-operations(monitoring-panel:monitoring/tasks/subtasks/a914be10-2fd1-11e9-979a-49712d01d7cc)

Note: In this example, a914be10-2fd1-11e9-979a-49712d01d7cc is the workflow id value. 

30. Log in to the SDDC Manager VM as the vcf user and then issue the su - command to switch to the root user.
31. Issue a command similar to the following to create a json file with the workflow input:

curl -s http://localhost/domainmanager/internal/vault/a914be10-2fd1-11e9-979a-49712d01d7cc -XGET | json_pp > /tmp/add_vi_with_nsxt_input.json

Note: Replace a914be10-2fd1-11e9-979a-49712d01d7cc with the workflow id value obtained in Step 29.

32. Open /tmp/add_vi_with_nsxt_input.json with a text editor.
33. Fine the stanza that begins with MigrateFromVdsToNvds____10__listOfHosts. In this stanza, there will be a com.vmware.evo.sddc.common.plugin.model.contract.nsxt.HostInfo sub-stanza for each host. Update the vmnic# values in each host stanza as appropriate.
34. Save and close the file.
35. Issue a command similar to the following to update the input parameters to the workflow:

curl -s http://localhost/domainmanager/internal/vault/a914be10-2fd1-11e9-979a-49712d01d7cc -XPUT -H "Content-type: text/plain" -d @/tmp/add_vi_with_nsxt_input.json

Note: Replace a914be10-2fd1-11e9-979a-49712d01d7cc with the workflow id value obtained in Step 29.

36. Click the RESTART TASK button next to the failed workflow in SDDC Manager.

Additional Information

To be alerted when this article is updated, click the Subscribe to Article link in the Actions box.