Cannot active service engine group in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• Cannot active service engine group as Load Balancer cannot be activated on this Edge Gateway in VMware Cloud Director, you will see error:

Record not found.

You will see the below from the VMware Cloud Director logs, 

Segment created in NSX
| DEBUG    | networking-web-activity-pool-1 | NsxTEdgeLoadBalancer           | Successfully created Load Balancer segment with id 80f1####-####-####-####-####7e18 for Edge Gateway <Edge Gateway name> in NSX | requestId=c503####-####-####-####-####59a4

Attempting to update cloud with new segment and tier1
| DEBUG    | networking-web-activity-pool-1 | NsxAlbLoadBalancerCloud        | Registering Tier-1 router 471c####-####-####-####-####2ce4 and Segment /infra/segments/80f1####-####-####-####-####7e18 on Load Balancer Cloud ###-###-###-###-###with backing cloud-31f1####-####-####-####-####3818 in NSX-ALB. Total Tier 1 Routers: ## | requestId=c503####-####-####-####-####59a4

Cloud update call to NSX-ALB
| TRACE    | networking-web-activity-pool-1 | LoggingRestTemplateProxy       | PUT <NSX-ALB url>/api/cloud/cloud-31f1###-###-###-###-###3818      
>"tier1_segment_config": {
>// Omitted other details to show interested entry
>  {
>      "segment_id": "/infra/segments/80f1####-####-####-####-####7e18",
>      "tier1_lr_id": "/infra/tier-1s/471c####-####-####-####-####2ce4"
>  }

Cloud update fails in NSX-ALB
| INFO     | networking-web-activity-pool-1 | NsxTEdgeLoadBalancer           | Failed to enable Load Balancer configuration on Edge Gateway <Edge Gateway name>  | requestId=c503####-####-####-####-####59a4
>com.vmware.vcloud.common.network.NsxAlbApiException: record not found
>    at com.vmware.vcloud.fabric.net.nsxalb.NsxAlbRestTemplateErrorHandler.handleError(NsxAlbRestTemplateErrorHandler.java:##)
>    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:##)

Able to fetch the newly created segment just after cloud update fails
| DEBUG    | networking-web-activity-pool-1 | LoggingRestTemplateProxy       | Request::URI:<NSX-ALB url>/policy/api/v1/infra/segments/80f1####-####-####-####-####7e18 method:GET response:200 execution time:64 ms | requestId=c503####-####-####-####-####59a4

Segment is delete by vCD to avoid stale objects
| DEBUG    | networking-web-activity-pool-1 | NsxTEdgeLoadBalancer           | Deleting Load Balancer segment with id 80f1####-####-####-####-####7e18 for Edge Gateway <Edge Gateway name>  in NSX | requestId=c503####-####-####-####-####59a4

Cause

This issue occurs when the update is happening on cloud, AVI was searching for Service Engine group which is stale causing the configuration error.

Resolution

1. Take identify the SE group login to AVI UI and then all event check the error and make a note of the se_group_template_ref id. 
2. SSH to AVI and take a backup.
3. Run the below query in AVI Database to verify if this SE group is present.
   show serviceenginegroup
4. Remove the stale Service Engine group template by running below commands :
   configure cloud
no se_group_template_ref
save