GSLB Pool Members Remain in Oper_Disabled State Following Virtual Service Recreation in AMKO Setups

Products

• VMware Avi Load Balancer

Issue/Introduction

• In environments using AMKO with GSLB, service pool members may remain in an Oper_Disabled state after a Virtual Service(Pool member) is deleted and recreated

• In this scenario, the Virtual Services at individual sites may appear locally UP, but the GSLB service does not become active.
• When reviewing the GSLB service runtime output from the controller's CLI (show gslb service <gslb service name> runtime), Error: "Configured and Operation VIPs are out of sync" will be seen.

• This situation can occur if the original Virtual Services associated with the GSLB service are deleted and new Virtual Services are created. Because recreated Virtual Services receive new UUIDs, the existing GSLB configuration may continue referencing the old UUIDs. As a result, the GSLB service members may remain disabled. Please refer the below document for more information on this
• In some cases, logs from AMKO may show that the operator attempts to create a new GSLB service instead of updating the existing one, which may lead to repeated HTTP 409 conflict errors during configuration attempts. Additionally, expected events for removing or updating GSLB members may not appear in the controller.

Environment

• VMware Avi Load Balancer
• Avi Multi-Cluster Kubernetes Operator

Cause

• This issue can occur when the AMKO UUID (amko-uuid) changes between deployments.
• Each GSLB service created by AMKO contains a created_by field that identifies the AMKO instance UUID responsible for managing that service. AMKO only synchronizes and updates GSLB services that were created using its own UUID. If AMKO is reinstalled or the GSLBConfig CRD is deleted and recreated without preserving the original amko-uuid, the new AMKO instance may run with a different UUID.
• When this happens:
  • The existing GSLB services appear as objects created by another AMKO instance.
  • The current AMKO instance does not update the existing GSLB services.Instead, AMKO may attempt to create a new GSLB service with its own UUID, which can result in 409 errors because the GSLB service already exists.
  • This mismatch can also lead to situations where Virtual Service updates are not properly reflected in the GSLB configuration, leaving pool members in an Oper Disabled state.

Resolution

• To resolve the issue, verify that the AMKO UUID configuration remains consistent across deployments.

Recommended steps:

• Check the amko-uuid configured in the AMKO deployment. This can be extracted from the AMKO logs.

• Verify the created_by field of the existing GSLB service on the Avi Controller to identify the UUID that originally created the service. This can be obtained from the below CLI command

show gslbservice <GSLB service_name>

• Configure AMKO with the same amko-uuid used when the GSLB services were originally created.
• If Virtual Services were recreated, ensure that the GSLB service members are updated with the correct Virtual Service mappings
• If required, update the VS mapping for the GSLB service through the Avi Controller UI or API to restore the service temporarily

For guidance on configuring AMKO and setting the amko-uuid, refer to the following documentation:

AMKO Installation and Configuration Guide

AMKO Routing Rules and CRD Configuration

These documents provide additional information on maintaining consistent AMKO configuration during reinstallations or upgrades.