High memory consumption observed in medium NSX Manager appliance due to unhealthy notification watcher

Products

• VMware NSX

Issue/Introduction

• High memory alarms are triggered for the NSX Manager nodes similar to the below:
  
  
  
• A notification watcher such as Aria Operations has been configured for NSX.
• Log similar to the below are encountered in var/log/phonehome-coordinator/phonehome-coordinator.log
  

  WARN NfwCollectorExternalSendServiceImpl-3-1 NfwCollectorExternalRestSendServiceImpl 74506 - [nsx@6876 comp="nsx-manager" level="WARNING" s2comp="notification" subcomp="monitoring"] Fail to send request. Exception: org.springframework.web.client.ResourceAccessException with org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://##.##.##.##:443/webhooks/nsxt/notification-watcher": Connect to ##.##.##.##:443 [/##.##.##.##] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to ##.##.##.##:443 [/##.##.##.##] failed: connect timed out
  
  .
  .
  .
  
  WARN NfwCollectorExternalSendServiceImpl-2-1 NfwCollectorExternalSendServiceImpl$NfwCollectorPeriodicNotificationSender 74506 MONITORING [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="monitoring"] Fail to send request. Exception: java.util.concurrent.TimeoutException with java.util.concurrent.TimeoutException

Environment

VMware NSX

Cause

This issue occurs due to an unhealthy notification watcher being configured in the environment, a memory leak is encountered as the task is not cleaned up successfully. 

Resolution

This issue is resolved in VMware NSX 4.2.2, and VMware NSX 9.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

1. Address the unhealthy notification watcher
   • Investigate any potential connectivity issue with the unhealthy notification watcher.
     
     or
   • Remove the unhealthy notification watcher, in Aria Operations this can be done in Aria Operation by deregistering NSX.
     
     or
     
     
   • Remove the unhealthy notification watcher using the NSX API  
     
     

     Returns a list of registered notification watchers. Returns a list of registered notification watchers.	GET /api/v1/notification-watchers
     Add a new notification watcher. Add a new notification watcher.	POST /api/v1/notification-watchers
     Delete an existing Notification-Watcher. Delete notification watcher.	DELETE /api/v1/notification-watchers/{watcher-id}
     Returns notification watcher by watcher id. Returns notification watcher by watcher id.	GET /api/v1/notification-watchers/{watcher-id}
     Update notification watcher. Update notification watcher.	PUT /api/v1/notification-watchers/{watcher-id}
     Get notifications for the specified watcher ID. Get notifications for the specified watcher ID.	GET /api/v1/notification-watchers/{watcher-id}/notifications
     Add uri filters for the specified watcher ID. Add uri filters for the specified watcher ID.	POST /api/v1/notification-watchers/{watcher-id}/notifications?action=add_uri_filters
     Delete uri filters for the specified watcher ID. Delete uri filters for the specified watcher ID.	POST /api/v1/notification-watchers/{watcher-id}/notifications?action=delete_uri_filters
     Update notifications for the specified watcher ID. Update notifications for the specified watcher ID.	PUT /api/v1/notification-watchers/{watcher-id}/notifications

2. Restart the phonehome-coordinator service on all NSX Manager nodes using the below command in root:
   /etc/init.d/phonehome-coordinator restart

Additional Information

Registering Notification Watchers

VMware NSX 4.2.2 Release Notes, see "Fixed Issue 3497315"