Detecting & Preventing CVE-2021-44228 (Log4Shell) with the NSX Distributed IDS/IPS

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

This KB article covers how the NSX Distributed IDS/IPS can be used to detect and protect against attempts at exploiting the CVE-2021-44228 vulnerability, also known as Log4shell. This Remote Code Execution (RCE) vulnerability in Apache Log4j2 allows malicious actors to load and execute arbitrary code from LDAP servers when the Log4j message lookup substitution feature has been enabled. VMware's Threat Analysis unit has overserved active exploitation attempts and is continuously monitoring adversary activities. The NSX IDS/IPS signatures provided are continuously updated based on exploit and obfuscation techniques observed. More information about the analysis by the VMware Threat Analysis Unit is available at Log in the Shell: An Analysis of Log4Shell Exploitation.

Note: The IDS/IPS feature is available through a subscription license for both NSX on-premises (Intrusion Detection in NSX-T 3.0 and Intrusion Prevention in NSX-T 3.1) and VMware Cloud on AWS (VMC) using SDDC 1.15+ and above.

While this article specifically covers the NSX IDS/IPS, other controls and solutions can/should also be leveraged to limit the attack surface and detect/prevent post-exploit activity such as lateral movement or deployment of ransomware or crypto miners. In addition, software vendors are providing workarounds and patches to remediate the issue, as well broad recommendations for any software leveraging vulnerable versions of open-source Apache Log4j library.

The NSX Distributed and Gateway Firewall should be leveraged to restrict outbound communication associated with the exploit. Specifically, LDAP and RMI communication should only be allowed to trusted servers based on an allow-list approach.  In addition, the distributed firewall should also be used to prevent post-exploit activity and lateral movement beyond the initial target. NSX Network Detection & Response (NDR) and Network Traffic Analysis (NTA) can further identify traffic anomalies associated with unusual post-exploit activity and correlate this activity with the initial exploit. More information regarding how other VMware products including the NSX Advanced Load Balancer (AVI) and Carbon Black Endpoint & NDR can help is available at Investigating CVE-2021-44228 Log4Shell Vulnerability

Environment

VMware NSX-T Data Center
VMware NSX
VMware vDefend Firewall

Resolution

As the distributed IDS/IPS is applied to the network interface (vNIC) of a workload, intrusion detection and prevention is provided for every workload regardless of the physical location (which hypervisor), the origination (external/internal attacker) or the tactic (i.e., initial access, discovery, lateral movement, exfiltration) being leveraged. 

Several NSX IDS/IPS Signatures have been released to detect and prevent network activity associated with attempts of exploiting this vulnerability. This activity could be indicative of scanning  (which accounts for the majority of observed associated events at the time of writing) or real exploitation attempts. The signatures we provide detect the "JNDI ${}" string which triggers the exploit, as well as related inbound and outbound network activity including LDAP and RMI and obfuscation techniques known to be used with this exploit. At the time of writing of this KB article, 28 IDS/IPS signatures associated with CVE-2021-44228 are available NSX customers. This includes signatures released by Emerging Threat Labs (ET) as well as SpiderLabs Research (SLR).

Below are the available signatures at the time of writing. Additional signatures may be provided through signature set updates delivered from the NSX Threat Intelligence cloud. 
 

SpiderLabs Research:

• 4103226 SLR Alert - Apache Log4j Log4Shell RCE HTTP LDAP Attempt (CVE-2021-44228) rev:1
• 4103227 SLR Alert - Apache Log4j Log4Shell RCE TCP LDAP Attempt (CVE-2021-44228) rev:1
• 4103228 SLR Alert - Apache Log4j Log4Shell RCE HTTP LDAPS Attempt (CVE-2021-44228) rev:1
• 4103229 SLR Alert - Apache Log4j Log4Shell RCE TCP LDAPS Attempt (CVE-2021-44228) rev:1
• 4103230 SLR Alert - Apache Log4j Log4Shell RCE HTTP RMI Attempt (CVE-2021-44228) rev:1
• 4103231 SLR Alert - Apache Log4j Log4Shell RCE TCP RMI Attempt (CVE-2021-44228) rev:1
• 4103232 SLR Alert - Apache Log4j Log4Shell RCE HTTP DNS Attempt (CVE-2021-44228) rev:1
• 4103233 SLR Alert - Apache Log4j Log4Shell RCE TCP DNS Attempt (CVE-2021-44228) rev:1
   

Emerging Threat:

• 2034647 ET EXPLOIT Apache log4j RCE Attempt (http ldap) (CVE-2021-44228) rev:1
• 2034648 ET EXPLOIT Apache log4j RCE Attempt (http rmi) (CVE-2021-44228) rev:1
• 2034649 ET EXPLOIT Apache log4j RCE Attempt (tcp ldap) (CVE-2021-44228) rev:1
• 2034650 ET EXPLOIT Apache log4j RCE Attempt (tcp rmi) (CVE-2021-44228) rev:1
• 2034651 ET EXPLOIT Apache log4j RCE Attempt (udp ldap) (CVE-2021-44228) rev:2
• 2034652 ET EXPLOIT Apache log4j RCE Attempt (udp rmi) (CVE-2021-44228) rev:2
• 2034653 ET EXPLOIT Apache log4j RCE Attempt (udp dns) (CVE-2021-44228) rev:2
• 2034654 ET EXPLOIT Apache log4j RCE Attempt (tcp dns) (CVE-2021-44228) rev:2
• 2034655 ET EXPLOIT Apache log4j RCE Attempt (http dns) (CVE-2021-44228) rev:2
• 2034656 ET EXPLOIT Apache log4j RCE Attempt (udp ldaps) (CVE-2021-44228) rev:2
• 2034657 ET EXPLOIT Apache log4j RCE Attempt (tcp ldaps) (CVE-2021-44228) rev:2
• 2034658 ET EXPLOIT Apache log4j RCE Attempt (http ldaps) (CVE-2021-44228) rev:2
• 2034659 ET EXPLOIT Apache log4j RCE Attempt - lower/upper TCP Bypass (CVE-2021-44228) rev:1
• 2034660 ET EXPLOIT Apache log4j RCE Attempt - lower/upper UDP Bypass (CVE-2021-44228) rev:2
• 2034667 ET EXPLOIT Apache log4j RCE Attempt (udp iiop) (CVE-2021-44228) rev:2
• 2034668 ET EXPLOIT Apache log4j RCE Attempt (tcp iiop) (CVE-2021-44228) rev:2
• 2034671 ET EXPLOIT Apache log4j RCE Attempt - 2021/12/12 Obfuscation Observed M1 (CVE-2021-44228) rev:1
• 2034672 ET EXPLOIT Apache log4j RCE Attempt - 2021/12/12 Obfuscation Observed M1 (CVE-2021-44228) rev:1
• 2034673 ET EXPLOIT Possible Apache log4j RCE Attempt - 2021/12/12 Obfuscation Observed M2 (CVE-2021-44228) rev:1
• 2034674 ET EXPLOIT Possible Apache log4j RCE Attempt - 2021/12/12 Obfuscation Observed M2 (CVE-2021-44228) rev:1
   

Downloading the latest signature set:

NSX Manager can be configured to automatically download the latest IDPS signature set from the NSX Threat Intelligence Cloud and propagate it to the individual transport nodes (hypervisors) on which the distributed IDPS has been enabled on. Alternatively, for air-gapped NSX installations, the signatures can be downloaded manually from the threat intelligence cloud and upload the bundle to NSX manager. The offline download mechanism is described at Offline Downloading and Uploading NSX Intrusion Detection Signatures.



Image: select the latest signature set as the active set (not required when the "auto update new versions" feature has been enabled. 

Viewing the relevant signatures

Once the latest signature set has been downloaded and selected as the active set, the signatures listed above can be confirmed as being available via Global Signature Management. 

Click "View and manage global signature set>>" and then apply a filter based on CVE 2021-44228. The following list of signatures should be seen:

Image: Apply a CVE-based filter in Global Signature Management to confirm signatures for CVE 2021-44228 are available. 

Note: The number of signatures available to cover this vulnerability depends on the signature set version. A higher number than the number shown at time of writing this article may be seen.

Creating an IDPS profile that includes these signatures:

The next step is to create an IDPS profile that includes these signatures. Since all these signatures have a severity of "HIGH", and attack target of "Server", one option is to create a profile that includes all signatures with severity of "HIGH" and "CRITICAL" and Attack Target of "Server".



Image: Create an IDPS profile to include the signatures related to this vulnerability. 

Click “Manage signatures for this profile >>" and filter based on CVE 2021-44228 to confirm these signatures are included in the profile. Note that these signatures come with a default action set to "Alert". For the NSX Distributed IDS/IPS to drop the offending traffic in addition to alerting, the action must be changed to either "Drop" or "Reject".  Note that the distributed IDS/IPS will only take a preventative action when it's deployed in "detect & prevent" mode.


Image: Verifying signatures are included in the IDPS profile and changing the action to "Drop" 

Note: If an IDPS profile is already created, based on the inclusion criteria these signatures may already be included. The severity for all these signatures is set to "HIGH". Click "Manage signatures for this profile >>" within a profile to confirm the signatures are included. 
 

Creating workloads groups and an IDPS policy:

The next and final configuration step is to create an IDPS policy and rule(s). These define where the policy that was configured will be applied to. 
Optionally, if is known (i.e., as a result of vulnerability assessment) what workloads in the environment are running a service that has implemented a vulnerable version of Apache log4j logging service, the IDPS profile can be applied specifically to those workloads. To do so, apply a Tag to each vulnerable workload, and create a group that includes the tagged workloads. Alternatively, if the scope of vulnerable services has not been established, the IDPS profile can be applied more broadly to the workloads in the environment. 


Image: Creating a new security tag 


Image: Tagging potentially vulnerable virtual machines


Image: Creating a Group based on the tag

Once groups have been created, the IDPS policy and rule(s) can be created. Set the "Applied To" for these rules to the security group that was just created. In order take preventative action in addition to alerting when an exploit is attempted, set the mode to "Detect & Prevent". In addition, as mentioned earlier, the signature action also needs to be changed to "drop" or "reject" for the NSX distributed IDS/IPS to take a preventative action. To get alerted when a signature is triggered, set the action to "Detect only".

.


Image: Creating an IDPS policy and rule "applied-to" set to the group you created earlier, applying the IDPS profile in "Detect & Prevent" mode.

Note: If IDPS policy and rules already exist and cover the potentially vulnerable servers, there is no need for an additional rule. In this case, ensure that the appropriate signatures have been enabled in the profile applied in the rule. 
 

Monitoring Log4Shell activity:

When an IDPS signature is triggered on a distributed IDS/IPS instance running on a hypervisor transport node, the event is sent up to the NSX manager where it can be consumed from the NSX UI. Alternatively, syslog can be configured to send IDPS events directly from the hypervisor transport node to your SIEM/syslog collector. 

To monitor 2021-44228 related activity using the NSX Manager UI, navigate to Distributed IDS/IPS → Events. A filter can be created set to "2021-44228" to display all signature events related to this vulnerability. The timeline can be changed to look at events across all distributed IDS/IPS instances for the last 24 hours - last 14 days. Additional details such as the Source, Destination IP address, affected VMs, whether the attempt was prevented and details about the vulnerability are available from each individual event. 


Image: Monitoring IDS/IPS events related to CVE 2021-44228 from the NSX UI.