Stale Alarm: Connectivity to LDAP server is lost in NSX-T

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

• A similar alarm is observed in the NSX UI

• You may observe that the LDAP Server being referenced in the Alarm is not present under section System -> Identity Firewall AD in the NSX-T UI.
  • The problematic LDAP Server is also not present when the API command is ran against the NSX Manager 

    GET https://<NSX-Manager>/policy/api/v1/infra/firewall-identity-stores

• You will observe similar logging on the NSX Manager in /var/log/phonehome-coordinator/phonehome-coordinator.log

  <Year>-<Month>-<Day><Time> FATAL pool-88-thread-2 MonitoringServiceImpl 4061 MONITORING [nsx@6876 alarmId="f08650fd-1383-####-####-c1e783499ed7" alarmState="OPEN" comp="nsx-manager" entId="5d9b98d0-####-###-b476-7315bf1d2948" errorCode="MP701099" eventFeatureName="identity_firewall" eventSev="CRITICAL" eventState="On" eventType="connectivity_to_ldap_server_lost" level="FATAL" nodeId="2e8b091e-####-####-ab1b-7cf2635a5cf4" subcomp="monitoring"] The connectivity to LDAP server <LDAPServerName> is lost.

• All LDAP servers show as UP indicating this is a stale alarm
  
  

Environment

VMware NSX

VMware vDefend Firewall

Cause

In some cases the the alarm client library fails to clear and the alarm needs to be manually cleared. See resolution section. 

Resolution

To clear the stale alarm follow the below steps:

1. Use API  to find which NSX-T Manager sends the alarm. GET https://<nsx-ip>/api/v1/alarms?status=OPEN
2. Restart proton on that NSX-T Manager to clear the stale alarm in alarm client library. As root user run the command /etc/init.d/proton restart
3. Resolve the alarm on UI