Information regarding CVE-2021-44228 & CVE-2021-45046 in NSX Data Center for vSphere

Products

VMware NSX

Issue/Introduction

CVE-2021-44228 & CVE-2021-45046 has been determined to potentially impact VMware NSX Data Center for vSphere via the Apache Log4js open-source component it ships. This vulnerability and its impact on VMware products is documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

CVE-2021-44228 & CVE-2021-45046 – VMSA-2021-0028

Environment

VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.4.x

Resolution

The workarounds described in this document are meant to be a temporary solution only.

NSX Data Center for vSphere 6.4.12 contains improved protection to vulnerabilities addressed in CVE-2021-44228 & CVE-2021-45046. Please refer to the releases notes for further details: https://docs.vmware.com/en/VMware-NSX-Data-Center-for-vSphere/6.4/rn/VMware-NSX-Data-Center-for-vSphere-6412-Release-Notes.html

Workaround:

Pre-check
Make sure the script is not already executed on set up. Please check folder '/home/secureall/secureall/' and check presence of files '.log4j2-patched'. If file is present , then script is already executed on set up. Please do following steps in that case :

Delete /home/secureall/secureall/.log4j2-patched
Delete /home/secureall/secureall/log4j_patch.sh

To apply the workaround for CVE-2021-44228 & CVE-2021-45046 perform the following steps in your NSX-V environment:

1. Take a backup of your NSX-V Manager.

Follow the Backup NSX manager admin guide for steps
OR
Alternatively, take a "cold clone" of your NSX-V Manager, prior to applying the workaround.

Steps:
1. Shutdown your existing NSX-V Manager.
2. Clone the NSX-V Manager to a VM with a different name, indicating that it is a clone of the original NSX-V Manager.
3. Power on your original NSX-V Manager.
4. Do NOT power on the clone NSX-V Manager, unless needed for rollback.

2. Download the following file from the attachment section of this KB article: signed_bsh_fix_log4j.encoded

3. Run the following REST API POST call using either Option A or Option B as described below.

Option A: POSTMAN

URL : POST  https://<NSX-Manager-IP>/api/1.0/services/debug/script
(Replace the <NSX-Manager-IP> in the above URL with the actual IP address of NSX-V Manager)

Authentication: Basic Auth (Username: admin)

Headers: Content-Type - application/xml

Body: Select 'Binary' as body type and attach the file signed_bsh_fix_log4j.encoded

Sample screenshots:

Option B: Curl command for Linux

curl --verbose --noproxy '*' -u admin -H "Content-Type: application/xml" -k -X POST "https://<NSX-Manager-IP>/api/1.0/services/debug/script" --data @/tmp/signed_bsh_fix_log4j.encoded

NOTES:

- Replace '<NSX-Manager-IP>' with the IP of your NSX Manager.
- curl is a linux based API utility. This must be run on a linux based OS.
- The source OS must have the 'signed_bsh_fix_log4j.encoded' file local to the OS. The /tmp/ directory is an example. It may be local to any directory.
- The source OS must have IP reachability to the NSX-V Manager over TCP port 443.

Expected Response: 200

Please ensure above POST call returns HTTP Status 200 OK before proceeding.

4. Reboot the NSX-V Manager.

Validation:

To verify the workaround for CVE-2021-44228 has been correctly applied to VMware NSX-V Manager perform the following steps:

Steps to login at the root level are:

SSH into the NSX V Manager as 'admin'
Type 'en' and provide the 'privileged mode' password.(HINT: Often, this is the same as your admin password, yet it is an end user configured value)
yet it is customer set.Type 'st en' once in the privileged mode, and provide the root password.
Type 'y' when prompted
Provide the password: IAmOnThePhoneWithTechSupport

Once the system has rebooted, SSH to the NSX-V Manager and elevate to root prompt (see KB 321408 ) and invoke the tcpdump command:

#tcpdump -i lo -s 1500 -XX port 389

Expected Response:

[root@nsx-v-manager ~]# tcpdump -i lo -s 1500 -XX port 389
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on lo, link-type EN10MB (Ethernet), snapshot length 1500 bytes

While the tcpdump packet capture is running, send following REST API :

URL : POST https://<NSX-Manager-IP>/api/2.0/services/securitygroup/globalroot-0
Authentication: Basic Auth (Username: admin)
Headers: Content-Type - application/xml

Body:
<securitygroup>
<name>${jndi:ldap://127.0.0.1/e}</name>
</securitygroup>

Example Screenshot:

If the patch has been applied correctly, there will be no tcpdump output.

If the patch has not been applied correctly, you will see the following output in the tcpdump packet capture:

10:31:50.349103 IP localhost.43162 > localhost.ldap: Flags [S], seq 4118243761, win 43690, options [mss 65495,nop,nop,sackOK,nop,wscale 7], length 0
0x0000: 0##0 0##0 0##0 0##0 0##0 0##0 0##0 4##0 ..............E.
0x0010: 0##4 3##f 4##0 4##6 0##3 7##0 0##1 7##0 .41?@.@.........
0x0020: 0##1 a##a 0##5 f##7 6##1 0##0 0##0 8##2 .......wi.......
0x0030: a##a f##8 0##0 0##4 f##7 0##1 0##2 0##3 ...(............
0x0040: 0##7 ..
10:31:50.349118 IP localhost.ldap > localhost.43162: Flags [R.], seq 0, ack 4118243762, win 0, length 0
0x0000: 0##0 0##0 0##0 0##0 0##0 0##0 0##0 4##0 ..............E.
0x0010: 0##8 0##0 4##0 4##6 3##e 7##0 0##1 7##0 .(..@.@.<.......
0x0020: 0##1 0##5 a##a 0##0 0##0 f##7 6##2 5##4 ...........wi.P.
0x0030: 0##0 a##4 0##0 ......
10:31:50.353257 IP localhost.43164 > localhost.ldap: Flags [S], seq 2371466404, win 43690, options [mss 65495,nop,nop,sackOK,nop,wscale 7], length 0

To revert the workaround for CVE-2021-44228 to VMware NSX Data Center perform the following steps on each NSX-V Manager:

Restore to the backup of your NSX-V Manager that you took, prior to applying the workaround detailed above.
Steps to follow Restore NSX Manager Admin guide
OR
Alternatively, if a "cold clone" was taken, follow the steps below:

1. Shut down the original NSX-V Manager, which you wish to roll back from.
2. Power on the cloned NSX-V Manager, which you wish to roll back to.

Additional Information

NOTE: Customer who have applied the workaround earlier only for CVE-2021-44228 need to re-apply the workaround again as mentioned in this KB. (New CVE-2021-45046 details was added on December 17th 11.15 PST)
Change log:

December 14th 2021 - 12:30 PST: Added clarity regarding upgrading to a newer, vulnerable version of NSX-V after applying the workaround.
December 15th 2021 - 11:30 PST: Added notice acknowledging CVE-2021-45046 and an impending release containing log4j version 2.16. Updated article title for consistency.
December 17th 2021 - 11:15 PST: Changed the workaround provided to the new VMware recommended NSX-V workaround.
December 18th 2021 - 02:30 PST: Removed hyperlink from step 2 in workaround section. File is attached to this KB.
December 21th 2021 - 02:30 PST: Added a note related to workaround which need to be carried by all those customers who followed the previous version of workaround.
December 21st 2021 - 14:00 PST: Clarified requirements for curl API utility.
December 22nd 2021 - 12:00 PST: Added the release of NSX Data Center for vSphere 6.4.12 to the resolution section of this article.
January 24th 2022 - 10:00 PST: Replaced signed_bsh_fix_log4j file with an updated version of the file, suited for additional versions of NSX V.

Impact/Risks:

A malicious actor with network access to an impacted VMware product may exploit this issue to invoke remote code execution.

All versions of NSX-V Data Center for vSphere contain the log4js and require this workaround.

Note: NSX-V Edge Service Gateways, NSX-V Controllers, and NSX-V Guest Introspection VM's are not affected by this issue.

Note: If the below workaround is applied to an NSX-V Manager, and that NSX-V Manager is subsequently upgraded to a newer vulnerable version of NSX-V, the workaround must be re-applied post upgrade.

Note: This workaround will have to be re-applied on the post-restore NSX-V Manager if an environment is restored from backup.

Attachments

signed_bsh_fix_log4j get_app