Workaround instructions to address CVE-2021-44228 and CVE-2021-45046 in vRealize Operations 8.x

Products

• VMware Aria Suite

Issue/Introduction

CVE-2021-44228 has been determined to impact vRealize Operations 8.0.x - 8.6 via the Apache Log4j open source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

• CVE-2021-44228 - VMSA-2021-0028

Notes:

• For Application Remote Collector (ARC) versions 8.1.x - 8.3.x, where the standalone Application Remote Collector appliance is available, use the Analytic (Primary, Replica, Data), Remote Collector and Witness node workaround instructions in this article.  There is no need to run the scripts against the Application Remote Collector appliance, as it does not have any Log4j dependencies or jars.
• For Application Remote Collector (ARC) versions 8.4 and above, where a Cloud proxy is in use, follow the Cloud Proxy workaround instructions in this article.
• For those still using End Point Operations agents, End Point Operations is based on Log4j 1.2.x which may be impacted by CVE-2021-45046, however we have no components using the JMSAppender configuration.  End Point Operations agents are not impacted.

Symptoms:
Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors. 

We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating Log4j to version 2.16 in forthcoming releases of vRealize Operations, as outlined by our software support policies.  VMSA-2021-0028 will be updated when these releases are available. In the interim, we have updated this Knowledge Base article with revised guidance to remove all JndiLookup classes per Apache Software Foundation guidance.

Please subscribe to this article to be informed when updates are published.

Environment

VMware vRealize Operations 8.3.x
VMware vRealize Operations 8.x
VMware vRealize Operations 8.0.x
VMware vRealize Operations 8.4.x
VMware vRealize Operations 8.1.x
VMware vRealize Operations 8.6.x
VMware vRealize Operations 8.2.x
VMware vRealize Operations 8.5.x

Resolution

This issue is resolved in the following releases:

• vRealize Operations 8.1.1 Hot Fix 10
• vRealize Operations 8.2 Hot Fix 10 
• vRealize Operations 8.3 Hot Fix 9 
• vRealize Operations 8.4 Hot Fix 6 
• vRealize Operations 8.5 Hot Fix 5
• vRealize Operations 8.6.2 

Note: These Hot Fixes are not applicable for non-native and Third Party Management Packs.  Non-native and Third Party Management Packs should be upgraded to safe versions.  Failure to do so could leave vRealize Operations vulnerable due to these management packs introducing the security risk.

The workarounds described in this document are meant to be a temporary solution only.
Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 and CVE-2021-45046 when available.

Workaround:
Notice: The below content has been updated as of 12/15/2021 to add workaround steps for the related CVE-2021-45046 as noted above.  Please re-run all of the below steps even if you have already implemented the original CVE-2021-44228 workaround steps by running the data-rc-witness-log4j-fix.sh and cp-log4j-fix.sh scripts.

To apply the workaround for CVE-2021-44228 and CVE-2021-45046 to vRealize Operations, perform the following steps:

For Analytic (Primary, Replica, Data), Remote Collector and Witness nodes:

1. Log into the vRealize Operations Manager Admin UI as the local admin user.
2. Click Take Offline under Cluster Status.

Note: Wait for Cluster Status to show as Offline.

3. Copy the attached data-rc-witness-log4j-fix.sh and vrops-log4j-fix.sh files to the /tmp directory on all Analytic, Remote Collector and Witness nodes in the cluster using an SCP utility.
4. Log into each Analytic, Remote Collector and Witness node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
5. Change to the /tmp directory on all nodes:

cd /tmp

6. Run the following command on all nodes to make the data-rc-witness-log4j-fix.sh script executable:

chmod +x data-rc-witness-log4j-fix.sh

7. Run the following command on all nodes to make the vrops-log4j-fix.sh script executable:

chmod +x vrops-log4j-fix.sh

8. Run te following command on all nodes to execute the data-rc-witness-log4j-fix.sh script:

./data-rc-witness-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

9. Run the following command on all nodes to execute the vrops-log4j-fix.sh script:

./vrops-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

10. Run the following command on all nodes to restart the CaSA service:

service vmware-casa restart

11. Log into the vRealize Operations Manager Admin UI as the local admin user.
12. Click Bring Online under Cluster Status.

Note: Wait for Cluster Status to show as Online.

For Cloud Proxies  

1. Copy the attached cp-log4j-fix.sh and vrops-log4j-fix.sh files to the /tmp directory on all Cloud Proxies using an SCP utility.
2. Log into each Cloud Proxy as root via SSH or Console, pressing ALT+F1 in a Console to log in.
3. Change to the /tmp directory on all Cloud Proxies:

cd /tmp

4. Run the following command on all Cloud Proxies to make the cp-log4j-fix.sh script executable:

chmod +x cp-log4j-fix.sh

5. Run the following command on all Cloud Proxies to make the vrops-log4j-fix.sh script executable:

chmod +x vrops-log4j-fix.sh

6. Run the following command on all Cloud Proxies to execute the cp-log4j-fix.sh script:

./cp-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

7. Run the following command on all Cloud Proxies to execute the vrops-log4j-fix.sh script:

./vrops-log4j-fix.sh

Note: Ensure there are no ERROR messages in the script output.

8. Run the following command on all Cloud Proxy nodes to restart the CaSA and Collector services:

service vmware-casa restart; service collector restart

To verify the workaround for CVE-2021-44228 has been correctly applied to vRealize Operations, perform the following steps:

1. Log into each node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Run the following command to verify if the data-rc-witness-log4j-fix.sh script was successful:

ps axf | grep --color log4j2.formatMsgNoLookups | grep -v grep

Note: There should be output from the above command.  If there was no output on any particular node(s), that node(s) was not successfully modified.  Re-run the script on that node(s) following the instructions above.

3. Run the following command to verify if the vrops-log4j-fix.sh script was successful:

./tmp/vrops-log4j-fix.sh

Note: You should receive output reading:

Searching for impacted .jar files. Please wait...
No impacted .jar files found

Additional Information

To revert the workaround for CVE-2021-44228 and CVE-2021-45046 to vRealize Operations perform the following steps:

1. Revert to the snapshot taken of each node prior to implementing the workaround.

Change Log:

• December 10th 2021 - 14:22 MST:    Drafted initial document with initial workaround.
• December 15th 2021 - 13:30 MST:    Added the vrops-log4j-fix.sh script and instructions to remove vulnerable JndiLookup classes from all .jar files.
• December 16th 2021 - 10:56 MST:    Added notes pertaining to Application Remote Collector (ARC) in the Purpose section.
• December 16th 2021 - 11:30 MST:    Added "CVE-2021-45046" to the title.  Added note about End Point Operations in the Purpose section.
• December 22nd 2021 - 1:43 MST:    Added note regarding WAR files in the Impact/Risks section.  Added note and download link for vRealize Operations 8.6.2.
• January 7th 2022 - 14:31 MST:   Added links for vRealize Operations 8.1.1 Hot Fix 10, vRealize Operations 8.2 Hot Fix 10, vRealize Operations 8.3 Hot Fix 9, vRealize Operations 8.4 Hot Fix 6, and vRealize Operations 8.5 Hot Fix 5.
• February 4th 2022 - 11:15 MST:   Added security risk note about non-native and third party management packs to the resolution section.
• February 8th 2022 - 2:23 MST:   Clarified security risk note about non-native and third party management packs in the resolution section.

For workaround steps for vRealize Operations Cloud see Workaround instructions to address CVE-2021-44228 in vRealize Operations Cloud.

Impact/Risks:
It is highly recommended to take snapshots of the vRealize Operations nodes following How to take a Snapshot of vRealize Operations.
Note: These snapshots are required if you should have to revert the workaround for any reason.

The mitigation will be undone if Management PAK’s are installed or Activated after applying the workaround, which bring Log4j-core components below version 2.16.  The below workaround steps must be reapplied after installing any Management PAKs.

The mitigation will be undone if the any vRealize Operations nodes, or Cloud Proxies are freshly deployed or updated to a version which doesn’t contain the fix.  The below workaround steps must be reapplied after deploying new vRealize Operations nodes or Cloud Proxies, or updating to a release that does not contain the fix.

The mitigation will be undone of any WAR files included with vRealize Operations are redeployed.  This only happens during upgrades, fresh deployments, or a WAR generated folder is accidentally deleted by the end user.
Note: There is no valid work flow to trigger any vulnerabilities from the included WAR files, so the WAR files themselves do not need to have any workaround applied to them.