Ingress http/https Virtual Servers are not getting created when creating PKS cluster using DNS Lookup Network Profile

Products

• VMware Cloud PKS

Issue/Introduction

Deploying PKS cluster using DNS Lookup Network-Profile. The PKS cluster gets deployed successfully. The cluster LB Ingress http/https Virtual Servers are not getting created when creating the cluster.

Note: This can be verified by:

1. Logging in to the NSX-T manager UI >> Advanced Networking & Security >> Load Balancing
2. Click on the LB for the cluster UUID >> Virtual Servers

Similar errors messages as seen below can be found in the /var/vcap/sys/log/ncp//ncp.stdout.log on the Cluster's Master node.

2020-05-08T17:56:04.923Z 50ffdb89-c0b1-4aca-8009-ff13fc64c870 NSX 11263 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] vmware_nsxlib.v3.client The HTTP request returned error code 409, whereas 201/200 response codes were expected. Response body {u'error_code': 5141, u'error_message': u'Requested IP Address ##.###.##.### is already allocated.', u'httpStatus': u'CONFLICT', u'module_name': u'id-allocation service'}

2020-05-08T17:56:04.923Z 50ffdb89-c0b1-4aca-8009-ff13fc64c870 NSX 11263 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="ERROR" errorCode="NCP00164"] nsx_ujo.ncp.nsx.manager.base_k8s_nsxapi Failed to allocate ip from ip pool: NSX IP ##.###.##.### is already allocated in ip pool 93a4fb62-fd30-447e-b4ff-0d78f4697c41

Similar errors messages as seen below  are found in the /var/log/syslog on the NSX-t Manager

<179>1 2020-05-08T17:56:04.921Z cdppnnsxtm0001 NSX 6084 POOL-MGMT [nsx@6876 comp="nsx-manager" errorCode="MP5141" level="ERROR" reqId="ecc7a0ff-dc21-4a04-96fb-2073973703a1" subcomp="manager" username="pks-87f55023-7545-4298-aebd-a18012a9d456"] Requested ip ##.###.##.### already allocated

The Ingress IP Address is allocated to the floating IP pool. To verify:

1. ssh to the NSX-T manager as root 
2. Run the following command to get the allocated IP address
   
   # curl -k -u <dmin:,Password> get https://localhost/api/v1/pools/ip-pools/</floating_IP_Pool>/allocations
   
   EX:
   curl -k -u 'admin:VMware1!VMware1!' get https://localhost/api/v1/pools/ip-pools/493a4fb62-fd30-447e-b4ff-0d78f4697c41/allocations
{
"allocation_id": "##.###.##.###",
"_protection": "NOT_PROTECTED"
},
{
"allocation_id": "##.###.##.###",
"_protection": "NOT_PROTECTED"
},​​​​ 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

TKGi (PKS) versions 1.6, 1.7.

Cause

The cluster creation task will first trigger the NSX-T to allocate the ingress IP Address and later NCP will try to allocate the IP again but it won't be able to since the IP Address is already allocated and this will cause the creation of Virtual Servers to fail.

Resolution

Fixed in TKGi 1.8. Review the release notes for future TKGi (PKS) version passed 1.6.2 and 1.7.0 as they may contain the fix.

Workaround:

After the PKS cluster creation task completes successfully without creating the cluster LB Ingress http/https Virtual Servers, apply the following steps:

1. SSH to NSX-T manager as Admin
2. type st eng
3. Enter the root account password
4. Run the following curl command to verify if the Ingress IP Address was allocated to the /floating IP Pool or not.
   
   # curl -k -u 'admin:<Password>' get https://localhost/api/v1/pools/ip-pools/<FIP_POOL_ID>/allocations

5. If the Ingress IP Address was already allocated then run the following command to release it.
   
   # curl X POST -H Content-Type:application/json -H X-Allow-Overwrite:True -ku 'admin:<NSXT_Admin_Password>' https://localhost/api/v1/pools/ip-pools/FIP_Pool_ID>?action=RELEASE -d '{"allocation_id":"<Ingress_IP>"}'
   
   Ex:
   
   curl X POST -H Content-Type:application/json -H X-Allow-Overwrite:True -ku 'admin:VMware1!VMware1!' https://localhost/api/v1/pools/ip-pools/93a4fb62-fd30-447e-b4ff-0d78f4697c41?action=RELEASE -d '{"allocation_id":"##.##.##.##"}'

6. Wait five minutes, then run the curl command from step 4 to confirm that the ingress IP Address got released.
7. Once confirmed that the ingress IP address was released, ssh to the NCP master node:
   1. Run the following command to get all the cluster deployment Instances.
      
      # bosh vms --column=Instance
      
      
   2. Run the following command to identify the NCP master node in case there are more than one master nodes
      
      # bosh ssh -d <service-instance_UUID> master -c "sudo /var/vcap/jobs/ncp/bin/nsxcli -c get ncp-master status" | grep "This instance is the NCP master"

Ex:
      
      root@opsman:~# bosh ssh -d service-instance_d470208a-6733-465d-a583-46175b772d55 master -c "sudo /var/vcap/jobs/ncp/bin/nsxcli -c get ncp-master status" | grep "This instance is the NCP master"

master/15ed1d27-e7bc-43ec-ac0f-181aa1c4bfc6: stdout | This instance is the NCP master
   3. ssh to the NCP master node 
      
      # bosh ssh -d <service-instance_UUID> master/<UUID>

Ex:
      
      # bosh ssh -d service-instance_d470208a-6733-465d-a583-46175b772d55 master/15ed1d27-e7bc-43ec-ac0f-181aa1c4bfc6
      
      
8. Check the "ncp" services status.
   
   # sudo -I
# monit summary

9. Stop and start the NCP service.
   
   # monit stop ncp
# monit summary
# monit start ncp
# monit summary

10. Restarting the NCP service will trigger it to check that all the PKS NSX-T objects do exist, and since the cluster Ingress Virtual Servers ware not created, the NCP will allocate the ingress IP Address again to the floating IP pool and will start to create the cluster Ingress http/https Virtual Servers.

Notes:

• You can run the curl command in step 4 to verify that the Ingress IP Address was allocated.
• Verify that the ingress Virtual Servers were created:
  1. Login to the NSX-T manager UI >> Advanced Networking & Security >> Load Balancing,
  2. Click on the LB for the cluster UUID >> Virtual Servers.