Security Services Health Degraded Alarm Observed in NSX UI Following Upgrade to Version 9.1.0.x

Products

• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vDefend Firewall

Issue/Introduction

After upgrading vDefend/NSX to version 9.1.0.x,  "Security Services Health Degraded" alarm is actively reported on the NSX Manager User Interface.

This condition occurs because the security services container (SCRX) on the affected ESXi host node fails to start. Consequently:

• Traffic will not be subjected to Intrusion Detection and Prevention System (IDPS) inspection, allowing potentially malicious traffic to pass undetected.
• Layer 7 (L7) traffic on the affected host may experience disruption, and rules based on App ID or FQDN may fail to match.

Symptoms:

• "Security Services Health Degraded" alarm is raised for one or more ESXi hosts.

  The security services container (SCRX) on the host node is down. Traffic will not be subject to IDPS and malicious traffic could go undetected.

 

• Executing the status query on the host via the NSX CLI fails to return the container uptime:

  /opt/vmware/nsx-cli/bin/nsx-appctl -t /var/run/vmware/scx/sdp.ctl sdp/get/status

 

• /var/log/proton/nsxapi.log on the NSX Manager, messages similar to the following appear:

  ####-##-##T##:##:##.855Z WARNING NSX #### [nsx@4413 comp="nsx-manager" level="WARNING" logger="ScxFlowDiscoveredNodeStreamListener" msgID="FABRIC" subcomp="manager" threadName="ScxFlowDiscoveredNodeStreamListener-1-1"] Scx flow StreamListener: skipping as DeploymentProgressState is INSTALL_FAILED (not INSTALL_SUCCESSFUL) for DiscoveredNode: #######-####-####-####-#####:host-##

 

• In the ESXi host /var/run/log/nsx-syslog.log, nsx-opsagent reports critical alerts regarding container state and uptime failure:

  ####-##-##T##:##:##.###Z -CRITICAL nsx-opsagent #### [nsx@4413 comp="nsx-esx" entId="########-####-####-########" eventType="security_services_health_degraded" eventSev="critical" s2comp="nsx-monitoring" eventFeatureName="scrx" eventState="On" threadID="#####"] The security services container on host node ########-####-####-########  is down. Traffic will not be subject to IDPS, and malicious traffic could go undetected. For L7 traffic, App ID or FQDN based rules may not match.

  
  
  

• In /var/run/log/nsx-syslog.log on the ESXi host, following error logs repeat every 30 seconds:

  ####-##-##T##:##:##:###Z -ERROR nsx-opsagent ##### [nsx@4413 comp="nsx-esx" subcomp="opsagent" s2comp="ctxteng" errorCode="CTX774" threadID="#####"] IdpsAlarm:CollectScrxDownSample scrx container is UP but failed to get uptime, ret: 1

 

Environment

VMware vDefend/NSX 9.1.0.x

Cause

The nsx-scx service startup configuration (chkconfig) on the ESXi host may be unexpectedly set to off. As a result, when the host reboots, the nsx-scx daemon fails to launch automatically, triggering the SCRX alarm in the NSX UI.

Resolution

This issue will be resolved in an upcoming release; if you suspect your environment is impacted, please open a Service Request (SR) with Broadcom Support under the vDefend or Distributed Firewall component.