GSLB Health Monitor Deletion Attempts Failing with 403 Error

Products

• VMware Avi Load Balancer

Issue/Introduction

In AMKO version 1.12.1 with GSLB configuration, frequent delete requests are made by AMKO to remove GSLB Health Monitors. However, these deletion attempts fail with a 403 error because the Health Monitor is still in use by the Pool in the Controller.

Environment

• AMKO Version:1.12.1
• AVI Load balancer with GSLB configuration

Cause

The issue stems from the naming convention used for GSLB objects. For example, a GSLB might be created with a name such as test-think-xxxx-ffff-0001253.

AMKO automatically creates a Health Monitor (HM), which includes the GSLB name as a substring in its description. This leads AMKO to misidentify the Health Monitor as stale, triggering repeated deletion attempts despite the monitor being actively used.

{
"name": "amko--xxxxxxxxxxxxxxxxxxx",            "description": "created by: amko, gsname: xxxxxxxxxxxxxxxxxxxxx , path: /, protocol: https, created from:  xxx-xxxx-amko-template"
}

 

Log verification:

Logs from AMKO show frequent DELETE calls for the Health Monitor.

2024-10-03T05:52:13.641Z        INFO    rest/dq_nodes.go:1195   name of HM to be deleted from the cache: amko--xxxxxxxxxxxxxxxx
2024-10-03T05:52:14.143Z        WARN    utils/avi_rest_utils.go:135     RestOp method DELETE path /api/healthmonitor/healthmonitor-xxxxxxxxxxxx tenant admin Obj null returned err {"code":0,"message":"map[error:Cannot delete, object is referred by: ['GslbService xxxxxxxxxxxxxxxx'] obj_name:amko--xxxxxxxxxxxxxxxxxxx]","Verb":"DELETE","Url":"https://x.x.x.x//api/healthmonitor/healthmonitor-xxxxxxxxxxxx","HttpStatusCode":403} with response null
2024-10-03T05:52:14.143Z        WARN    rest/dq_nodes.go:1240   key: admin/xxxxxxxxxxxxxxxxxxxxxxx, HealthMonitor: amko--xxxxxxxxxxx, msg: failed to delete, will retry
2024-10-03T05:53:23.097Z        INFO    rest/dq_nodes.go:1195   name of HM to be deleted from the cache: amko--xxxxxxxxxxxxxxxxxx

Resolution

• As of now there is no workaround for this issue.
• Please upgrade to AMKO version 1.13.1 which has fix for this issue.