Impact analysis of CVE-2021-44228 on VMware NSX Lastline Defender On-Premises

Products

• VMware

Issue/Introduction

VMware NSX Lastline Defender On-Premises is not impacted by CVE-2021-44228 or CVE-2021-45046. While the product captures inputs from untrusted sources (e.g., network traffic captured on a customer network) and stores parts of this data in Elasticsearch, our analysis shows that attackers cannot exploit the vulnerability to their advantage, as data is stored in Elasticsearch but not logged using the vulnerable framework. Furthermore, while authenticated Lastline customers may indirectly interact with Elasticsearch by performing queries against the product search API or via Kibana, no user-provided data is used in the vulnerable component: Elastic issued a statement saying that Elasticsearch is not vulnerable to the Remote Code Execution (RCE) vulnerability , and our analysis determined that no user data is sent directly to Elasticsearch, which also prevents the related Server Side Request Forgery (SSRF) attack that Elasticsearch would otherwise be vulnerable to.

To avoid any confusion or risk, we are releasing a workaround and patch. The vulnerable component is installed on Lastline Defender Datanode and Pinbox appliances with NTA support. Other Lastline Defender appliances are not affected by the vulnerability.

This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:
•    CVE-2021-44228 – VMSA-2021-0028

See the Change log at the end of this article for all changes and subscribe to the article for updates.

Resolution

Version 9.5.1 (appliance version 1120.1) was released on December 15, 2021. This version includes the fix which disables log4j's message lookup substitution in Elasticsearch as an additional mitigation step.

The workarounds described in this document are meant to be a temporary solution only.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available.

Workaround:

While we have not identified any scenario where user data may be used by the vulnerable component, we have released version 9.5.1 (appliance version 1120.1) on December 15, 2021. This version includes the fix which disables log4j's message lookup substitution in Elasticsearch as an additional mitigation step. 

If you cannot upgrade to 9.5.1 only: This workaround is for Datanode and Pinbox appliances only (other appliance types are not affected), if you cannot upgrade to 9.5.1 perform the following steps:

1. Log into the affected Pinbox or Datanode appliance using the monitoring or lastline user (refer to the Lastline appliance installation manual for details)
2. Make a copy or backup of the configuration files located in step 3 (in case the patch needs to be reverted). Example backup commands:

   • cp -p /usr/share/appliance-config/modules/elasticsearch/templates/jvm.options.5.5.erb /usr/share/appliance-config/modules/elasticsearch/templates/jvm.options.5.5.erb.bkp
     
     cp -p /usr/share/appliance-config/modules/elasticsearch/templates/jvm.options.erb /usr/share/appliance-config/modules/elasticsearch/templates/jvm.options.erb.bkp

3. Edit the Elasticsearch configuration files in (e.g., using the vim editor) using the user root (e.g., using sudo):

   • sudo vim /usr/share/appliance-config/modules/elasticsearch/templates/jvm.options.5.5.erb
     
     sudo vim /usr/share/appliance-config/modules/elasticsearch/templates/jvm.options.erb

4. Add a new java parameter in both files in the corresponding section (add the bold line):

   • # log4j 2
     -Dlog4j.shutdownHookEnabled=false
     -Dlog4j2.disable.jmx=true
     -Dlog4j.skipJansi=true
     -Dlog4j2.formatMsgNoLookups=true

5. Save and exit the editor (e.g. save using command :wq)
6. Apply the configuration to all services as user root (e.g., using sudo):

   • sudo lastline_apply_config -o

7. To verify the workaround for CVE-2021-44228 has been correctly applied to VMware NSX Lastline Defender On-Premises perform the following steps:
   • Run the below commands as the user root (e.g., using sudo). This command should list at least one file with the updated java parameter:

     rgrep Dlog4j2.formatMsgNoLookups /etc/elasticsearch

     Example output:
     root@customer-pinbox:~# rgrep Dlog4j2.formatMsgNoLookups /etc/elasticsearch
/etc/elasticsearch/lldns/jvm.options:-Dlog4j2.formatMsgNoLookups=true

• This command should show the running java process using the updated configuration parameter:

  ps aux | grep -v grep | grep java.*log4j2.formatMsgNoLookups=true

  Example output:
  root@customer-pinbox:~# ps aux | grep -v grep | grep java.*log4j2.formatMsgNoLookups=true
elastic+ 3419 0.8 15.0 31104360 18642684 ? SLsl Dec10 3:00 /usr/bin/java -Xms16g -Xmx16g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -Dlog4j2.formatMsgNoLookups=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch/lldns -Des.distribution.flavor=oss -Des.distribution.type=deb -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet

To revert the workaround for CVE-2021-44228 to VMware NSX Lastline Defender On-Premises perform the following steps:

1. Restore the .bkp files created in step 2 back to the original file name
2. Execute sudo lastline_apply_config -o to propagate changes.

Additional Information

Change log:

• December 15th 2021 - 04:50 EST: Updated Knowledge Base article advising GA of version 9.5.1. Added notice at the top concerning the recent updates on CVE-2021-44228 from Apache.

Impact/Risks:

None are expected due to the nature of log4j.