Gorouter repeats "endpoint-not-unregistered" with routing-api in stdout log

Products

• VMware Tanzu Application Service for VMs
• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

"endpoint-not-unregistered" message is being recorded on all gorouter stdout logs identically as below. IP of backend is cloud controller instance IP. The message repeats every 20 seconds for each backend. 

{"log_level":1,"timestamp":"2024-05-27T16:28:31.245310805Z","message":"endpoint-not-unregistered","source":"vcap.gorouter.registry","data":{"uri":"<CAPI_FQDN>/routing","route_service_url":"","backend":"***.***.***.***:3001","application_id":"","instance_id":"a402045f-b702-4c7c-6fe9-7c04e01d1294","server_cert_domain_san":"routing-api.service.cf.internal","protocol":"http1","modification_tag":{"guid":"","index":0},"isolation_segment":"-","isTLS":true}}

Environment

TPCF ~v4.0

Cause

1. Routing-api is deployed across all cloud controller instances, only one which acquired the lock is active, others are at standby mode for high availability purpose.  
2. Route-registrar on each cloud controller instance runs health check against local cloud controller(at port 9024) and routing-api(at port 3031) at 20 seconds interval. 
3. For the active routing-api, route-registrar sends "register route" message to gorouters, for those standby routing-api(s), route-registrar sends "unregister route" message. 
4. Standby routing-api(s) have no routes on gorouters, thus one "unregister route" message will result in one "endpoint-not-unregistered" log in gorouter.stdout.log.

Resolution

This issue only exits in early releases of TPCF v2.0~4.0, it has been fixed in all releases of TPCF 6.0 or above.