Workaround instructions to address CVE-2021-44228 in vSphere Replication

Products

VMware Live Recovery VMware Cloud on AWS

Issue/Introduction

Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.
We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of VMware vSphere Replication, as outlined by our software support policies. This Knowledge Base article and VMSA-2021-0028 will be updated when these releases are available. Please subscribe to this article to be informed when updates are published.

You can validate exposure on a replication/site recovery appliance by running the following command from a shell as the root user:
grep -R 'JndiLookup.class' /opt/vmware/
grep -R 'JndiLookup.class' /var/opt/apache-tomcat/
If the mitigation has been successfully applied this command will not return results.
Verify the environment variables have been properly set (all in one line):
for pid in $( ps ax | grep java | grep -v grep | awk '{print $1}' ); do cat /proc/$pid/environ |tr '\0' '\n' | grep "LOG4J_FORMAT_MSG_NO_LOOKUPS"; done
Note: expected output multiple lines of "LOG4J_FORMAT_MSG_NO_LOOKUPS=true"

Highlighted sections indicate the most recent updates. See the Change log at the end of this article for all changes.

Environment

VMware vSphere Replication 8.3.x
VMware vSphere Replication 8.5.x
VMware vSphere Replication 8.4.x

Cause

CVE-2021-44228 has been determined to impact Site Recovery Manager and vSphere Replication via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

CVE-2021-44228 - VMSA-2021-0028

Resolution

This issue is resolved in the following release; Site Recovery Manager 8.3.1.5, 8.4.0.4 and 8.5.0.2 and vSphere Replication 8.3.1.5, 8.4.0.4 and 8.5.0.2. VMware recommends patching to these versions over applying the workarounds detailed in this KB.

Workaround:

vSphere Replication Management Server Appliance

Confirm no workflows are currently running; virtual machines should be in a stable state; not failing over or in a reprotect/initial sync.

Connect via SSH to the vSphere Replication Appliance , the process to enable ssh is the same for all versions
After Connecting you will need to stop the Java based services This is version specific:
Version 8.5
You will need to gain root access using the su command in order to stop the following services:
systemctl stop hms.service
systemctl stop drconfigui.service
systemctl stop dr-configurator.service
systemctl stop dr-client.service

Version 8.4
You will need to gain root access using the su command in order to stop the following services:
systemctl stop dr-configurator.service
systemctl stop drconfigui.service
systemctl stop tomcat.service
systemctl stop hms.service

Version 8.3 - Mitigation requires the appliance to have internet connectivity.
Stop the following services:
systemctl stop tomcat.service
systemctl stop hms.service
tdnf install zip

Version 8.3 - Mitigation without internet connectivity.
Download the attached file on the right of this page "zip-3.0-2.ph2.x86_64.rpm" and SCP it to the /root/ folder on the appliance
Stop the following services:
systemctl stop tomcat.service
systemctl stop hms.service
rpm --install /root/zip-3.0-2.ph2.x86_64.rpm
Apply the workaround.
This has been automated with the "disable_log4j_vr.bash" file downloadable on the right of this page.
SCP the file to the /root/ folder then via the ssh session execute:
chmod +x /root/disable_log4j_vr.bash
/root/disable_log4j_vr.bash

Note: You might see the following message for each log4j-core-*-sources.jar "zip error: Nothing to do! (/path/to/log4j-core-2.13.3-sources.jar)" These can be ignored safely.
Restart the services based on the VR version installed:
Version 8.5
You may need to gain root access using the su command.
systemctl start hms.service
systemctl start drconfigui.service
systemctl start dr-configurator.service
systemctl start dr-client.service

Version 8.4
You may need to gain root access using the su command.

systemctl start hms.service
systemctl start drconfigui.service
systemctl start dr-configurator.service
systemctl start tomcat.service

Version 8.3

systemctl start hms.service
systemctl start tomcat.service

Validate for vulnerable jars are patched successfully (All versions) (the next command should return no results).

grep -R 'JndiLookup.class' /opt/vmware/
grep -R 'JndiLookup.class' /var/opt/apache-tomcat/
Verify the environment variables have been properly set (all in one line):
for pid in $( ps ax | grep java | grep -v grep | awk '{print $1}' ); do cat /proc/$pid/environ |tr '\0' '\n' | grep "LOG4J_FORMAT_MSG_NO_LOOKUPS"; done
Note: expected output multiple lines of "LOG4J_FORMAT_MSG_NO_LOOKUPS=true"

vSphere Replication Server (Add-on VR server) Appliance

Confirm no workflows are currently running; virtual machines should be in a stable state; not failing over or in a reprotect/initial sync.

Connect via SSH to the vSphere Replication Appliance , the process to enable ssh is the same for all versions
Run the following version specific commands stop the services:

Version 8.3 - Mitigation requires the appliance to have internet connectivity.
tdnf install zip

Version 8.3 - Mitigation without internet connectivity.
Download the attached file on the right of this page "zip-3.0-2.ph2.x86_64.rpm" and SCP it to the /root/ folder on the appliance
rpm --install /root/zip-3.0-2.ph2.x86_64.rpm

Version 8.4 or above
You may need to gain root access using the su command
systemctl stop dr-configurator.service
systemctl stop drconfigui.service
Apply the workaround.
This has been automated with the "disable_log4j_vr.bash" file downloadable on the right of this page.
SCP the file to the /root/ folder then via the ssh session execute:
chmod +x /root/disable_log4j_vr.bash
/root/disable_log4j_vr.bash

Note: You might see the following message for each log4j-core-*-sources.jar "zip error: Nothing to do! (/path/to/log4j-core-2.13.3-sources.jar)" These can be ignored safely.
Restart all Java services on version 8.4 or above.
You may need to gain root access using the su command.
systemctl start dr-configurator.service
systemctl start drconfigui.service
Validate for vulnerable jars are patched successfully (the next command should return no results).
grep -R 'JndiLookup.class' /opt/vmware/
Verify the environment variables have been properly set (all in one line):
for pid in $( ps ax | grep java | grep -v grep | awk '{print $1}' ); do cat /proc/$pid/environ |tr '\0' '\n' | grep "LOG4J_FORMAT_MSG_NO_LOOKUPS"; done
Note: expected output multiple lines of "LOG4J_FORMAT_MSG_NO_LOOKUPS=true"

Additional Information

Note: Products older than vSphere Replication 8.3 are beyond End of General Support date, so patches will not be made available for these older versions. As there may be other non-validated exposure, VMware recommends updating EOGS versions to a supported release as a priority.
VR 8.1.x and 8.2.x Appliances were shipped with log4j 1.x. Based on https://logging.apache.org/log4j/2.x/security.html we believe these versions are not exposed to CVE-2021-44228 because the product uses log4j 1.x and does not have JMSAppender configured in its log4j configuration. Customers are urged to monitor https://logging.apache.org/log4j/2.x/security.html as guidance from the Apache Logging team may change. It is important to note that Log4j 1.x has reached end of life and is no longer supported by the Apache Logging team. As per https://logging.apache.org/log4j/2.x/security.html "Vulnerabilities reported after August 2015 against Log4j 1.x were not checked and will not be fixed". VMware strongly recommends upgrading to a supported version of vSphere Replication to minimize exposure.

Change Log:
December 17th 2021 - 11:29PST: Updated guidance to indicate patched version availability
December 16th 2021 - 15:13 PST: Script refreshed as a character was missing from the start of line 29
December 16th 2021 - 14:00 PST: Edited for clarity
December 16th 2021 - 11:20 PST: Updated script and added steps required to add environment flag for appliance.
December 15th 2021 - 09:26 PST: Added notice of intent.
December 15th 2021 - 06:43 PST: Added additional verification step for tomcat path; and clarification on older versions.

Impact/Risks:

This issue impacts all releases prior to Site Recovery Manager 8.3.1.5, 8.4.0.4 and 8.5.0.2 and vSphere Replication 8.3.1.5, 8.4.0.4 and 8.5.0.2.

Note: VMware automatically manages VMware Site Recovery (VSR) components which are deployed in a customer's SDDC in VMware Cloud on AWS. Customers are responsible for managing their on-prem SRM and vSphere Replication appliances and should review these KB articles to determine if their on-prem appliances are affected.

The workarounds described in this document are meant to be a temporary solution only. Issue mitigation is currently achieved by removal of the vulnerable code.

There is no functional impact to this workaround as these products do not leverage the vulnerable "JndiLookup.class".

Attachments

disable_log4j_vr get_app

zip-3.0-2.ph2.x86_64 get_app