Installing Isolation Segment failed with CPI error There are more than one pool member which have the same IP address

Products

• VMware Tanzu Application Service

Issue/Introduction

When deploying Prisma Cloud for  TAS and Isolation Segment, got error as below at "Installing Isolation Segment" step.  TAS and Isolation segment are 6.0.13:

Task 17xxxxxxx | 01:59:33 | Creating missing vms: isolated_router/xxxxxxxxxxxxxx (6) (00:01:28)
Updating deployment:

Expected task '175984536' to succeed but state is 'error'
Exit code 1
                          L Error: Invalid CPI response - SchemaValidationError: { error => Object {"type"=>"Unknown", "message"=>[{"httpStatus"=>"BAD_REQUEST", "error_code"=>23613, "module_name"=>"LOAD-BALANCER", "error_message"=>"There are more than one pool member which have the same IP address 10.xxx.xxx.211 and port 80."}], "ok_to_retry"=>false} doesn't validate against any of #<Membrane::Schemas::Value:0x0000xxxxxxxxxx>, #<Membrane::Schemas::Record:0x0000xxxxxxxxx> }

Environment

TAS 6.0.13

NSX 4.2.x

Isolation segment 6.0.13

Cause

Previous iteration of the isolation segment isolated_router resulted in unexpected removal of the VM from vCenter, resulting in any next attempt to recreate the VM in failure due to the Assigned IP address is already part of the member pool in NSX of the existing LoadBalancer modules.

Resolution

Pause reconciliation in bosh

bosh update-resurrection off

Get the IP from the error 10.xxx.xxx.211 and search in NSX if this IP is part of any Loadbalancer pool members. Please note there are usually at least 2 LB created for port 80 and port 443, however it is possible there could be more than two LB set for the isolation segment. the IP have to manually removed. 

Find the pool member select the member with IP from above and Delete and then Save the configuration.

After all the pools are cleaned up 

Apply Change or enable resurrection for bosh to recreate the missing VM.

Unpause reconciliation in bosh

bosh update-resurrection on

Additional Information

There could be multiple reasons why a VM can be deleted, possible reasons could be deleted VM by mistake, some connectivity problem and VM being recreated and could not complete the CPI operations.