Workaround instructions to address CVE-2021-44228 and CVE-2021-45046 in VMware Aria Operations (SaaS)

Products

• VMware Aria Suite

Issue/Introduction

CVE-2021-44228 has been determined to impact VMware Aria Operations (SaaS) (formerly known as vRealize Operations Cloud) via the Apache Log4j open source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

• CVE-2021-44228 - VMSA-2021-0028

We have taken the necessary actions to protect your environment from exploitation due to CVE-2021-44228.  The vRealize Operations Cloud services have already been patched.  Any Cloud Proxy appliances deployed on your local site(s) must have the workaround implemented manually by following the steps in the article.

Symptoms:
Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors. 

We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of VMware Aria Operations Cloud Proxy, as outlined by our software support policies.  VMSA-2021-0028 will be updated when these releases are available. In the interim, we have updated this Knowledge Base article with revised guidance to remove all JndiLookup classes per Apache Software Foundation guidance.

Please subscribe to this article to be informed when updates are published.

Resolution

The workarounds described in this document are meant to be a temporary solution only.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 and CVE-2021-45046 when available.

Workaround:
Notice: The below content has been updated as of 12/15/2021 to add workaround steps for the related CVE-2021-45046 as noted above.  Please re-run all of the below steps even if you have already implemented the original CVE-2021-44228 workaround steps by running the cp-log4j-fix.sh script.

To apply the workaround for CVE-2021-44228 and CVE-2021-45046 to VMware Aria Operations (SaaS) Cloud Proxies, perform the following steps:

1. Copy the attached cp-log4j-fix.sh and vrops-log4j-fix.sh files to the /tmp directory on all Cloud Proxies using an SCP utility.
2. Log into each Cloud Proxy as root via SSH or Console, pressing ALT+F1 in a Console to log in.
3. Change to the /tmp directory on all Cloud Proxies:

cd /tmp

4. Run the following command on all Cloud Proxies to make the cp-log4j-fix.sh script executable:

chmod +x cp-log4j-fix.sh

5. Run the following command on all Cloud Proxies to make the vrops-log4j-fix.sh script executable:

chmod +x vrops-log4j-fix.sh

6. Run the following command on all Cloud Proxies to execute the cp-log4j-fix.sh script:

./cp-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

7. Run the following command on all Cloud Proxies to execute the vrops-log4j-fix.sh script:

./vrops-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

8. Run the following command on all Cloud Proxy nodes to restart the CaSA and Collector services:

service vmware-casa restart; service collector restart

To verify the workaround for CVE-2021-44228 has been correctly applied to VMware Aria Operations (SaaS) Cloud Proxies, perform the following steps:

1. Log into each node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Run the following command to verify if the data-rc-witness-log4j-fix.sh script was successful:

ps axf | grep --color log4j2.formatMsgNoLookups | grep -v grep

Note: There should be output from the above command.  If there was no output on any particular node(s), that node(s) was not successfully modified.  Re-run the script on that node(s) following the instructions above.

3. Run the following command to verify if the vrops-log4j-fix.sh script was successful:

./tmp/vrops-log4j-fix.sh

Note: You should receive output reading:

Searching for impacted .jar files. Please wait...
No impacted .jar files found

Additional Information

To revert the workaround for CVE-2021-44228 and CVE-2021-45046 to VMware Aria Operations (SaaS) Cloud Proxies, perform the following steps:

1. Revert to the snapshot taken of each node prior to implementing the workaround.

Change Log:

• December 10th 2021 - 17:58 MST:    Drafted initial document with initial workaround.
• December 15th 2021 - 13:30 MST:    Added the vrops-log4j-fix.sh script and instructions to remove vulnerable JndiLookup classes from all .jar files.
• December 16th 2021 - 11:35 MST:    Added "CVE-2021-45046" to the title.

Impact/Risks:
It is highly recommended to take snapshots of the VMware Aria Operations (SaaS) Cloud Proxies.
Note: These snapshots are required if you should have to revert the workaround for any reason.

The mitigation will be undone if a new Cloud Proxy was deployed after applying the workaround.  The below workaround steps must be reapplied after installing any Management PAKs.