Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.
We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of VMware Site Recovery Manager, as outlined by our software support policies. This Knowledge Base article and VMSA-2021-0028 will be updated when these releases are available. Subscribe to this article to be informed when updates are published.
To expand the specific technical details of the issue in Knowledge Base article: Workaround instructions to address CVE-2021-44228 in Site Recovery Manager and vSphere Replication.
You can validate exposure on a replication/site recovery appliance by running the following command from a shell as the root user:
grep -R 'JndiLookup.class' /opt/vmware/
grep -R 'JndiLookup.class' /var/opt/apache-tomcat/
If the mitigation has been successfully applied this command will not return results.
Verify the environment variables have been properly set (all in one line):
for pid in $( ps ax | grep java | grep -v grep | awk '{print $1}' ); do cat /proc/$pid/environ |tr '\0' '\n' | grep "LOG4J_FORMAT_MSG_NO_LOOKUPS"; done
Note: expected output multiple lines of "LOG4J_FORMAT_MSG_NO_LOOKUPS=true"
Highlighted sections indicate the most recent updates. See the Change log at the end of this article for all changes.
This issue is resolved in the following releases:
VMware recommends patching to these versions over applying the workarounds detailed in this KB.
Verify that there are no pending cleanup operations on recovery plans and that there are no configuration issues for the virtual machines that Site Recovery Manager protects.
In version of the appliance at 8.3 or newer you will need to ensure the ssh service is enabled for the admin user.
You can enable or disable an SSH access to the appliance only for the admin account.
You will need to elevate the admin account to the root user to facilitate the stopping of services:
Once you are connected to the appliance you will need to implement the workaround:
This has been automated with the "disable_log4j_srm.bash" file in the attachments section at the bottom of this article.
SCP the file to the /root/ folder then via the ssh session execute:
Note: You might see the following message for each log4j-core-*-sources.jar zip "JndiLookup.class not found in /path/to/log4j-core-2.13.3-sources.jar" This is expected behavior and safe to ignore, as the workaround is not needed.
If you see a warning message, manual verification of the file and its contents are needed.
Verify that there are no pending cleanup operations on recovery plans and that there are no configuration issues for the virtual machines that Site Recovery Manager protects.
Once you are connected to the Windows session as an administrative user, you will need to implement the workaround:
Note: If you ever need to re-enable the "Windows Installer" In the future use the Startup type value from step 4 usually it is set to 3. From Command Prompt:
Note: Products older than SRM 8.3 are beyond End of General Support date, so patches will not be made available for these older versions. As there may be other non-validated exposure, VMware recommends updating EOGS versions to a supported release as a priority.
SRM 8.1.x and 8.2.x on Windows or Photon Appliances were shipped with log4j 1.x. Based on https://logging.apache.org/log4j/2.x/security.html we believe these versions are not exposed to CVE-2021-44228 because the product uses log4j 1.x and does not have JMSAppender configured in its log4j configuration. Customers are urged to monitor https://logging.apache.org/log4j/2.x/security.html as guidance from the Apache Logging team may change. It is important to note that Log4j 1.x has reached end of life and is no longer supported by the Apache Logging team. As per https://logging.apache.org/log4j/2.x/security.html "Vulnerabilities reported after August 2015 against Log4j 1.x were not checked and will not be fixed". VMware strongly recommends upgrading to a supported version of SRM to minimize exposure.
Change log:
December 17th 2021 - 11:29PST: Updated guidance to indicate patched version availability
December 16th 2021 - 14:52PST: Added VMC on AWS DRaaS guidance and improve readability of KB
December 16th 2021 - 13:16 PST: Corrected missed item
December 16th 2021 - 08:42 PST: Updated script and added steps required to add environmental flag for appliance; in-progress steps to remove JndiLookup.class for windows.
December 15th 2021 - 09:26 PST: Added notice of intent.
December 15th 2021 - 06:43 PST: Added additional verification step for tomcat path; and clarification on older versions.
December 14th 2021 - 11:03 PST: Added screenshot of the registry key that needs modification for clarity.
Impact/Risks:
This issue impacts all releases prior to Site Recovery Manager 8.3.1.5, 8.4.0.4 and 8.5.0.2 and vSphere Replication 8.3.1.5, 8.4.0.4 and 8.5.0.2.
Note: This includes VMware Site Recovery on VMware Cloud on AWS Customers' on-prem appliances.
The workarounds described in this document are meant to be a temporary solution only. Issue mitigation is currently achieved by removal of the vulnerable code.
There is no functional impact to this workaround as these products do not leverage the vulnerable "JndiLookup.class".
Note: Disabling the "Windows Installer" will prevent MSI applications from being Installed/Uninstalled/Modified or Repaired; The Auto-repair feature will also be interrupted. This is to prevent the SRM MSI from reverting the patch. You will need to follow the re-enabling steps (see the note in Part 3 Disable "Windows Installer") prior to updating your SRM install to the patched version for Windows (8.3.1.5)