The ingress http/https Virtual Servers are not getting created when creating PKS cluster using DNS Lookup Network Profile
search cancel

The ingress http/https Virtual Servers are not getting created when creating PKS cluster using DNS Lookup Network Profile

book

Article ID: 345589

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:
Note: This can be verified by Login to the NSX-T manager UI >> Advanced Networking & Security >> Load Balancing >> Click on the LB for the cluster UUID >> Virtual Servers
  • Similar errors messages as seen below can be found in the /var/vcap/sys/log/ncp//ncp.stdout.log  on the Cluster's Master node.
2020-05-08T17:56:04.923Z 50ffdb89-c0b1-4aca-8009-ff13fc64c870 NSX 11263 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] vmware_nsxlib.v3.client The HTTP request returned error code 409, whereas 201/200 response codes were expected. Response body {u'error_code': 5141, u'error_message': u'Requested IP Address 23.234.23.201 is already allocated.', u'httpStatus': u'CONFLICT', u'module_name': u'id-allocation service'}

2020-05-08T17:56:04.923Z 50ffdb89-c0b1-4aca-8009-ff13fc64c870 NSX 11263 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="ERROR" errorCode="NCP00164"] nsx_ujo.ncp.nsx.manager.base_k8s_nsxapi Failed to allocate ip from ip pool: NSX IP 23.234.23.201 is already allocated in ip pool 93a4fb62-fd30-447e-b4ff-0d78f4697c41
  • Similar errors messages as seen below  are found in the /var/log/syslog on the NSX-t Manager
<179>1 2020-05-08T17:56:04.921Z cdppnnsxtm0001 NSX 6084 POOL-MGMT [nsx@6876 comp="nsx-manager" errorCode="MP5141" level="ERROR" reqId="ecc7a0ff-dc21-4a04-96fb-2073973703a1" subcomp="manager" username="pks-87f55023-7545-4298-aebd-a18012a9d456"] Requested ip 23.234.23.201 already allocated
  • The Ingress IP Address is allocated to the floating IP pool.
To Verify :- 
  1. ssh to the NSX-T manager as root .
  2. Run the following command to get the allocated IP address in the  
# curl -k -u <dmin:,Password> get https://localhost/api/v1/pools/ip-pools/</floating_IP_Pool>/allocations

EX:
curl -k -u 'admin:VMware1!VMware1!' get https://localhost/api/v1/pools/ip-pools/493a4fb62-fd30-447e-b4ff-0d78f4697c41/allocations
{
"allocation_id": "23.234.23.200",
"_protection": "NOT_PROTECTED"
},
{
"allocation_id": "23.234.23.201",
"_protection": "NOT_PROTECTED"
},​​​​​​
 
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware PKS 1.x

Cause

The cluster creation task will first trigger the NSX-T to allocate the ingress IP Address and later NCP will try to allocate the IP again but it won't be able to since the IP Address is already allocated and this will cause the creation of Virtual Servers to fail.

Resolution

  • This is a known issue affecting TKGi (PKS) versions 1.6, 1.7 and  the fix  was released in  TKGi 1.8.
  • Please review the release notes for future TKGi (PKS) version pass 1.6.2 and 1.7.0 as they may contain the fix.


Workaround:
After the PKS cluster creation task completed successfully without creating the cluster LB Ingress http/https  Virtual Servers please apply the following steps:-
  1. SSH to NSX-T manager as Admin
  2. type st eng
  3. Enter the root account password
  4. Run the following curl command to verify if the Ingress IP Address was allocated to the /floating IP Pool or not.
# curl -k -u 'admin:<Password>' get https://localhost/api/v1/pools/ip-pools/<FIP_POOL_ID>/allocations
  1. If the Ingress IP Address was already  allocated then run the following command to release it.
 # curl X POST -H Content-Type:application/json -H X-Allow-Overwrite:True -ku 'admin:<NSXT_Admin_Password>' https://localhost/api/v1/pools/ip-pools/FIP_Pool_ID>?action=RELEASE -d '{"allocation_id":"<Ingress_IP>"}'
 
Ex:
curl X POST -H Content-Type:application/json -H X-Allow-Overwrite:True -ku 'admin:VMware1!VMware1!' https://localhost/api/v1/pools/ip-pools/93a4fb62-fd30-447e-b4ff-0d78f4697c41?action=RELEASE -d '{"allocation_id":"10.40.14.51"}'
 
  1. Wait five minutes then run the curl command from step 4 to confirm that the ingress IP Address got released.
  2. Once confirm that the ingress IP address was released, please ssh to the NCP master node using the following steps :
  • Run the following command to get all the cluster deployment Instances
# bosh vms --column=Instance
  • Run the following command to identify the NCP master node in case there are more than one master nodes
# bosh ssh -d <service-instance_UUID> master -c "sudo /var/vcap/jobs/ncp/bin/nsxcli -c get ncp-master status" | grep "This instance is the NCP master"
 
Ex"
root@opsman:~# bosh ssh -d service-instance_d470208a-6733-465d-a583-46175b772d55 master -c "sudo /var/vcap/jobs/ncp/bin/nsxcli -c get ncp-master status" | grep "This instance is the NCP master"

master/15ed1d27-e7bc-43ec-ac0f-181aa1c4bfc6: stdout | This instance is the NCP master
  • ssh to the NCP master node
 # bosh ssh -d <service-instance_UUID> master/<UUID>
 
Ex:
# bosh ssh -d service-instance_d470208a-6733-465d-a583-46175b772d55 master/15ed1d27-e7bc-43ec-ac0f-181aa1c4bfc6
 
  1. Check the "ncp" services status
# sudo -I
# monit summary
  1. Stop and start the NCP service.
# monit stop ncp
# monit summary
# monit start ncp
# monit summary
  1. Restarting NCP service will trigger it to check that all the PKS NSX-T objects do exist, and since the cluster Ingress Virtual Servers ware not created, the NCP will allocate the ingress IP Address again to the floating IP pool and will start to create the cluster Ingress http/https Virtual Servers.
Notes:
  • You can run the curl command in step 4 to verify that the Ingress IP Address was allocated.
  • Verify that the ingress Virtual Servers were created by: Login to the NSX-T manager UI >> Advanced Networking & Security >> Load Balancing >> Click on the LB for the cluster UUID >> Virtual Servers.


Powered by Wolken Software