Domain creation or expansion workflows will fail when provided license key(s) do not exist in the SDDC Manager inventory

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• If an API user triggers Add Domain with an ESXi license that is not added to the SDDC manager inventory or is invalid/mistyped, the VI workflow will fail at FetchLicensesForDomainAction. This workflow will be unrecoverable, and IPs from the networkpool will be taken as 'used'. No changes to inventory for the hosts/VI will occur, but these networkpool IPs will no longer be usable for future workflows.
• Workflow will fail and will not be able to complete. Network Pool IPs will be reserved but unused (orphaned).
•  Workflow must be started again with valid licenses.

Cause

The license Key(s) are not present in the VMware SDDC Manager inventory prior to the start of the workflow

Resolution

This is expected behaviour as the license key is a required parameter for the workflow. 
Please use the steps below to resolve this problem.

1. Add the ESXi license to the VMware SDDC Manager. See the Manage License Keys for the Software in Your Cloud Foundation System documentation for more information. 
2. Verify the Network Pool has sufficient IPs to attempt a new Add VI workflow.
3. If the Network Pool has sufficient IPs and the there are not concern about the orphaned IPs, proceed with creating the new Add VI workflow with the same domainSpec as used before.
4. The FetchLicensesForDomainAction will now succeed since the license has been added .

If the Network Pool does NOT have enough IPs for a new Add VI workflow, please see the options below and the corresponding entries in the Workaround section:

• Option 1: Add a new IP range for each of the networks in the Network pool which is sufficient for a new Add VI workflow
• Option 2: Release the orphaned IPs. 

Workaround:

For Option 1, please see Create a Network Pool documention for more information. Once the additional Network Pool(s) is create, please proceed with ensuring the license(s) are added and starting a new VI workflow. 

For Option 2, see the workaround steps below:

1. Open an SSH session to the SDDC Manager and change to the root user:

su - 

2. Use the following command to obtain the task id to be used in later step as the execution id:

curl localhost/domainmanager/inventory/tasks | json_pp

3. Get the ID for each network in the network pool

curl localhost/networkpools | json_pp

Example:
[
{
"networks" : [
"51fd518d-2ac9-41fd-a92e-55694022c9e2",
"dcff1e1f-d6a0-4d08-96e9-d4e574557712"
],
"name" : "sfo-m01-np01",
"id" : "0e06eff5-9fe7-4299-940b-5c8beb3f3ac0"
},
{
"networks" : [
"4f6584a4-6b7d-439e-a7ba-ee43b870a60a",
"1a53e9a3-af24-4f6f-b24b-2da841be709e"
],
"name" : "sfo-w01-np01",
"id" : "8b721a9b-67d6-4125-b09b-07fceb8e5040"
}
]

4. Use the Network Pool id to verify which networkId corresponds to which network:

curl localhost/networkpools/{network_id}/networks | json_pp

Example:
[
{
"type" : "VSAN",
"usedlist" : [
"172.28.223.101",
"172.28.223.102",
"172.28.223.103",
"172.28.223.104"
],
"subnet" : "172.28.223.0",
"mask" : "255.255.255.0",
"ippools" : [
{
"end" : "172.28.223.104",
"start" : "172.28.223.101"
}
],
"vlanid" : 3723,
"gateway" : "172.28.223.253",
"id" : "4f6584a4-6b7d-439e-a7ba-ee43b870a60a",
"mtu" : 9000
},
{
"gateway" : "172.28.222.253",
"id" : "1a53e9a3-af24-4f6f-b24b-2da841be709e",
"mtu" : 9000,
"type" : "VMOTION",
"usedlist" : [
"172.28.222.101",
"172.28.222.102",
"172.28.222.103",
"172.28.222.104"
],
"subnet" : "172.28.222.0",
"mask" : "255.255.255.0",
"ippools" : [
{
"end" : "172.28.222.104",
"start" : "172.28.222.101"
}
],
"vlanid" : 3722
}
]

5. For a Network Pool that is only being used for this VI Workload Domain, then simply take all of the 'usedList' as the IPs for the request body and skip step 5.
6. If the Network Pool is shared across Workload Domains / Management Domain, run the following command to retrieve the exact IPs that match for the ESXis in this specific domainSpec using the execution ID.

curl localhost/domainmanager/internal/vault/{execution-id} | grep vsanIpAddress
curl localhost/domainmanager/internal/vault/{execution-id} | grep vmotionIpAddress

7. Create a request body json for the the IPs required. Run the following commands:

vi /tmp/vsanIps.json

example:
["172.28.223.101","172.28.223.102","172.28.223.103","172.28.223.104"]

vi /tmp/vmotionIps.json
example:

["172.28.222.101","172.28.222.102","172.28.222.103","172.28.222.104"]

8. Run the following API commands to release the IPs for use:

curl -X DELETE -H "Content-type:application/json"
localhost/networkpools/{networkPoolId}/networks/{networkId}/ip

-d @/tmp/vsanIps.json

 

 

curl -X DELETE -H "Content-type:application/json"
localhost/networkpools/{networkPoolId}/networks/{networkId}/ip -d

@vmotionIps.json

After the above steps have been completed, start a new Add VI workflow with the same domainSpec as original. Important: Do not retry the failed operation.