Information regarding CVE-2021-44228 & CVE-2021-45046 in NSX-T Data Center (2.5.0-3.1.3)
search cancel

Information regarding CVE-2021-44228 & CVE-2021-45046 in NSX-T Data Center (2.5.0-3.1.3)

book

Article ID: 317795

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

CVE-2021-44228 & CVE-2021-45046 has been determined to potentially impact VMware NSX-T Data Center via the Apache Log4js open-source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:



Environment

VMware NSX-T Data Center

Resolution

December 22nd, 2021 Note: The previous version of the workaround distributed on December 17th, 2021, omitted some JndiLookup class files embedded in internal services within NSX-T. Due to an abundance of caution, we have updated the Debian package to remove additional occurrences of java class files. This updated package is comprehensive and includes the previous workarounds. VMware recommends applying the new Debian package 'unified-appliance-log4j2-patch_3.1.3.6.0.19099859_all.deb' if you have not already completed the previous workaround. There are no known exploits of the aforementioned internal services for customers who have already used the earlier Debian package, 'unified-appliance-log4j2-patch_3.1.3.6.0.19078151_all.deb

 
The workarounds described in this document are meant to be a temporary solution only. 
 
NSX-T Data Center 2.5.3.4, 3.0.3.1 and 3.1.3.5 provide resolution for CVE-2021-44228 & CVE-2021-45046. 
VMware recommends the following upgrade paths: 
  • NSX-T 2.5.x should upgrade to 2.5.3.4   
  • NSX-T 3.0.x should upgrade to 3.0.3.1 
  • NSX-T 3.1.x should upgrade to 3.1.3.5 
Please refer to the releases notes for further details:  
 
NSX-T 2.5.3.4 Release Notes: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.5.3/rn/VMware-NSX-T-Data-Center-2534-Release-Notes.html
 


Workaround:
To apply the workaround for CVE-2021-44228 & CVE-2021-45046, connect to the VMware NSX-T Data Center Manager or NSX-T Cloud Service Manager and perform the following steps.

Important: Apply these steps to one manager at a time and allow time for your management cluster to stabilize before moving onto the next manager.  
 

1. Take an SFTP Backup of your NSX-T Manager.  

Reference the following document for guidance, if needed: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-E6181BF1-2CB7-4870-B508-BFAF5B47D702.html

2. Download the Debian package 'unified-appliance-log4j2-patch_3.1.3.6.0.19099859_all.deb' from the attachment section on the right side of this KB article.

3. Copy the Debian package provided on this KB article to all three of the NSX-T Manager appliances. The use of WinSCP, Linux based SCP, or any like method is recommended

4. Login to the NSX-T Unified Appliance via SSH and using the admin account.

If SSH is not enabled, log in to vCenter Server and open the VM console of the NSX-T Manager. To start the SSH Service, run command: start service ssh

5. Switch to root user:

st en

Enter the root password when prompted.

6. Install the Debian package with the following command, run in the directory in which you've copied the Debian package to on the NSX-T Unified Appliance.

dpkg -i unified-appliance-log4j2-patch_3.1.3.6.0.19099859_all.deb

7. Reboot the system:

/sbin/reboot

Warning: Do not reboot several NSX-T Manager appliances at the same time. Ensure the cluster services stabilize, prior to proceeding to the next NSX-T Manager appliance.

To monitor NSX-T Manager cluster services, at the 'admin' level of SSH, use the following command:

get cluster status  
 

Verification Steps
To verify the workaround for CVE-2021-44228 & CVE-2021-45046 has been correctly applied to each NSX-T Manager Appliance, perform the following steps with root access via SSH on the appliance: 

Step 1: Verify the configuration files have been updated with the “formatMsgNoLookup=true” flag.
grep "wrapper.java.additional.100=" /usr/tanuki/conf/*-wrapper.conf
Example of expected output:

Note:  The number of files below will vary for NSX-T Manager and NSX-T Cloud Service
Manager. The presence of any number of files confirms that the workaround was
successfully applied.

/usr/tanuki/conf/async-replicator-service-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/cluster-boot-manager-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/cm-inventory-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/corfu-log-replication-server-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/corfu-nonconfig-server-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/corfu-server-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/cross-cloud-upgrade-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/idps-reporting-service-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/intelligence-upgrade-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/migration-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/nsx-ccp-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/phonehome-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/policy-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/proton-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/proxy-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/search-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
/usr/tanuki/conf/upgrade-coordinator-tomcat-wrapper.conf:wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true

Step 2: Verify the JndiLookup class was removed from all jar files on the appliance.
find /usr /opt -xdev -iname "*.jar" | while read l; do  unzip -l $l|grep "JndiLookup.class"; if [[ ${PIPESTATUS[1]} -eq 0 ]]; then echo $l; fi; done 
Expected Return: None

Note: If any files are found with the above command, please contact VMware Support.
 

To revert the workaround for CVE-2021-44228 & CVE-2021-45046 to VMware NSX-T Data Center perform the following steps on each NSX-T Manager or NSX-T Cloud Service Manager.

Restore to the NSX-T backup, taken prior to proceeding with the workaround process above.
 
Reference the following document for guidance, if needed: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-9749F041-15E5-4662-85E7-756D4B071C17.html


Additional Information

NOTE: Customer who have applied the workaround earlier only for CVE-2021-44228 need to re-apply the workaround again as mentioned in this KB. (New CVE-2021-45046 details was added on December 17th 17.00 PST)
Change Log:
  • December 13th 2021 - 11:00 PST: Removed validation test steps as it was only applicable to one attack vector, specifically for NSX-T 3.0.1 and earlier. 
  • December 13th 2021 - 11:30 PST: Clarified that NSX-T Edge Nodes, VM or Bare Metal, are not affected by this issue. 
  • December 13th 2021 - 12:30 PST: Removed erroneous word 'exit' from workaround. 
  • December 13th 2021 - 13:00 PST:  Added note regarding re-applying the workaround if an environment is restored to backup. 
  • December 13th 2021 - 16:00 PST: Minor edits to include NSX-T Cloud Service Manager to this KB as well.
  • December 14th 2021 - 05:00 PST: Minor edit to add -p on cp commands.
  • December 14th 2021 - 12:30 PST: Added clarity regarding upgrading to a newer, vulnerable version of NSX-T after applying the workaround.
  • December 15th 2021 - 11:30 PST: Added notice acknowledging CVE-2021-45046 and an impending release containing log4j version 2.16. 
  • December 17th 2021 - 15:00 PST: Added detail regarding NSX T 3.2.0 release pertinent to CVE-2021-44228 & CVE-2021-45046.
  • December 17th 2021 - 17:00 PST: Changed the workaround provided to the new VMware recommended NSX-T workaround.  
  • December 21th 2021 - 02:30 PST: Added a note related to workaround which need to be carried by all those customers who followed the previous version of workaround.
  • December 22nd 2021 - 10:00 PST: Changed the provided Debian package to an updated version, which provides further mitigation against CVE-2021-44228 & CVE-2021-45046. 
  • December 22nd 2021 - 12:00 PST: Minor changes to formatting and verbiage within the article to highlight important of prior update.  No procedural changes. 
  • December 22nd 2021 - 14:00 PST: Clarified recommendations around updated Debian package. 
  • December 23rd 2021 - 11:00 PST: Added additional versions that provide resolution to CVE-2021-44228 & CVE-2021-45046. Clarified upgrade path guidance. Provided more accurate command in verification step 2.
  • December 24th 2021 - 02:10 PST Improve validation Step 2 command to avoid false positives detected from backup files.


Impact/Risks:

A malicious actor with network access to an impacted VMware product may exploit this issue to invoke remote code execution.  However, there are multiple layers of protection in NSX-T that will make exploiting CVE-2021-44228 & CVE-2021-45046 difficult:

  • Internal NSX-T processes/JVM’s are prevented from making external connections via the iptables configuration in the NSX-T Unified Appliance.
  • The version of Java JDK in NSX-T Datacenter version 3.0.2 and above prevents the most common remote code execution publicized (ie. LDAP attack vector).
All versions of NSX-T Data Center contain the log4js.  Further exploit of the log message lookup feature may be possible.
 

Note: NSX-T Edge VM's and Bare Metal Edge Nodes are not affected by this issue. 

Note: If the below workaround is applied to an NSX-T Manager, and that NSX-T Manager is subsequently upgraded to a newer vulnerable version of NSX-T, the workaround must be re-applied post upgrade. 

Note: This workaround will have to be re-applied on the post-restore NSX-T Managers or NSX-T Cloud Service Managers if an environment is restored from backup. 
 


Attachments

unified-appliance-log4j2-patch_3.1.3.6.0.19099859_all get_app
Powered by Wolken Software