Workaround instructions to address CVE-2021-44228 in Site Recovery Manager
search cancel

Workaround instructions to address CVE-2021-44228 in Site Recovery Manager

book

Article ID: 318874

calendar_today

Updated On:

Products

VMware Live Recovery VMware Cloud on AWS

Issue/Introduction

Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors. 
We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of VMware Site Recovery Manager, as outlined by our software support policies. This Knowledge Base article and VMSA-2021-0028 will be updated when these releases are available. Subscribe to this article to be informed when updates are published. 

To expand the specific technical details of the issue in Knowledge Base article: Workaround instructions to address CVE-2021-44228 in Site Recovery Manager and vSphere Replication.

You can validate exposure on a replication/site recovery appliance by running the following command from a shell as the root user:
grep -R 'JndiLookup.class' /opt/vmware/
grep -R 'JndiLookup.class' /var/opt/apache-tomcat/
If the mitigation has been successfully applied this command will not return results.
Verify the environment variables have been properly set (all in one line):
for pid in $( ps ax  | grep java | grep -v grep | awk '{print $1}' ); do cat /proc/$pid/environ |tr '\0' '\n' | grep "LOG4J_FORMAT_MSG_NO_LOOKUPS"; done
Note: expected output multiple lines of  "LOG4J_FORMAT_MSG_NO_LOOKUPS=true"

Highlighted sections indicate the most recent updates. See the Change log at the end of this article for all changes.

Environment

  • VMware Site Recovery Manager 8.5.x
  • VMware Site Recovery Manager 8.3.x
  • VMware Site Recovery Manager 8.4.x

Cause

CVE-2021-44228 has been determined to impact Site Recovery Manager and vSphere Replication via the Apache Log4j open source component it ships.  This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

Resolution

This issue is resolved in the following releases:


VMware recommends patching to these versions over applying the workarounds detailed in this KB.

Workaround:

Site Recovery Manager Appliance

Verify that there are no pending cleanup operations on recovery plans and that there are no configuration issues for the virtual machines that Site Recovery Manager protects.

  • All recovery plans are in the Ready state.
  • The protection status of all the protection groups is OK.
  • The protection status of all the individual virtual machines in the protection groups is OK.
  • The recovery status of all the protection groups is Ready.
     

Connect to the appliance via SSH

In version of the appliance at 8.3 or newer you will need to ensure the ssh service is enabled for the admin user.
You can enable or disable an SSH access to the appliance only for the admin account.

Procedure

  1. Log in to the Site Recovery Manager Appliance Management Interface as admin.
  2. Click Access.
  3. In the SSH pane, click Enable or Disable.

You will need to elevate the admin account to the root user to facilitate the stopping of services:

Procedure

  1. Log in to the Site Recovery Manager Virtual Appliance via ssh as the admin user.
  2. Run the following command. su
  3. Enter the root password.


Once you are connected to the appliance you will need to implement the workaround:

Procedure

  1. In order to mitigate the issue we need to stop all Java services first.
    This is achieved with the following commands:
    systemctl stop dr-client.service
    systemctl stop drconfigui.service
    systemctl stop dr-configurator.service
  2. Mitigate all log4j core jar files.

    This has been automated with the "disable_log4j_srm.bash" file in the attachments section at the bottom of this article. 
    SCP the file to the /root/ folder then via the ssh session execute:

    chmod +x /root/disable_log4j_srm.bash
    /root/disable_log4j_srm.bash

    Note: You might see the following message for each log4j-core-*-sources.jar zip "JndiLookup.class not found in /path/to/log4j-core-2.13.3-sources.jar" This is expected behavior and safe to ignore, as the workaround is not needed.
    If you see a warning message, manual verification of the file and its contents are needed.

  3. Restart all Java services.
    systemctl start dr-client.service
    systemctl start drconfigui.service
    systemctl start dr-configurator.service     
     
  4. Verify that the vulnerable jars are mitigated successfully.
    grep -R 'JndiLookup.class' /opt/vmware/
    grep -R 'JndiLookup.class' /var/opt/apache-tomcat/
    Note: We expect this command not to return any results. If it does please return to step 1 and re-run the workaround.
  5. Verify the environmental variables have been properly set (all in one line):
    for pid in $( ps ax  | grep java | grep -v grep | awk '{print $1}' ); do cat /proc/$pid/environ |tr '\0' '\n' | grep "LOG4J_FORMAT_MSG_NO_LOOKUPS"; done
    Note: expected output multiple lines of  "LOG4J_FORMAT_MSG_NO_LOOKUPS=true"


Site Recovery Manager Windows install (version 8.3.x)


Verify that there are no pending cleanup operations on recovery plans and that there are no configuration issues for the virtual machines that Site Recovery Manager protects.

  • All recovery plans are in the Ready state.
  • The protection status of all the protection groups is OK.
  • The protection status of all the individual virtual machines in the protection groups is OK.
  • The recovery status of all the protection groups is Ready.

Once you are connected to the Windows session as an administrative user, you will need to implement the workaround:

Procedure 

Part 1 Update the registry

Option 1a  Using regedit

  1. Make sure you have a full backup of the Registry before moving forward
  2. Update the registry by launching regedit and change the values :
    Regedit HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\vmware-dr-client\Parameters\Java
    Key to modify:
    ------------------


    key to be modified  -  Options
    value: In edit box go to end and after
    -Djava.util.logging.config.file=C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client\conf\logging.properties Shift+Enter then append
    -Dlog4j2.formatMsgNoLookups=true

    Reference: complete value of the registry after the edit should look as below
    -Dcatalina.home=C:\Program Files\VMware\VMware vCenter Site Recovery Manager\external\tomcat
    -Dcatalina.base=C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client
    -Djava.io.tmpdir=C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client\temp
    -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
    -Djava.util.logging.config.file=C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client\conf\logging.properties
    -Dlog4j2.formatMsgNoLookups=true
  3. Create Windows System Variable:
    JAVA_TOOL_OPTIONS=-Dlog4j2.formatMsgNoLookups=true
  4. Restart "VMware vCenter Site Recovery Manager Client" service from Control Panel or from command line:
    net stop vmware-dr-client
    net start vmware-dr-client
    NOTE: If you run     Repair a Site Recovery Manager Server Installation the -Dlog4j2.formatMsgNoLookups=true will be removed from the registry and you must re-implement the workaround.
  5. Next patch the impacted Java files

Option 1b Using Powershell

  1. Use the following Powershell script to update the registry:
    $opts = $((Get-ItemProperty 'HKLM:\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\vmware-dr-client\Parameters\Java').Options)
    $opts1 = $opts + "-Dlog4j2.formatMsgNoLookups=true"
    Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\vmware-dr-client\Parameters\Java' -Name 'Options' -Value $opts1
  2. Create Windows System Variable:
    JAVA_TOOL_OPTIONS=-Dlog4j2.formatMsgNoLookups=true
  3. Restart "VMware vCenter Site Recovery Manager Client" service
    net stop vmware-dr-client
    net start vmware-dr-client
    NOTE: If you run Repair a Site Recovery Manager Server Installation the -Dlog4j2.formatMsgNoLookups=true will be removed from the registry and you must re-implement the workaround.
  4. Next patch the impacted Java files

Part 2 Patching the impacted Java files

  1. Next we need to remove the "Jndilookup.class" from the impacted java files If you have installed zip utility in your SRM Windows Server, use this zip utility in the steps below instead of "7z".
    Download and install 7zip from https://www.7-zip.org/ as using 64-bit Windows x64 EXE version. Do not install MSI 64-bit version, because Windows Installer service might be disabled 
    Disable "Windows Installer" service following the steps below "How to disable "Windows Installer" service via Command Prompt" in order to disable Windows® installer self-repair or resiliency feature, because it may revert the patched files. 
    NOTE: This will disable install, uninstall, repair, modify for all MSI applications and Windows Installer, including SRM installer. If you enable "Windows installer" feature due any reason, you will need to re-run the mitigation steps and disable the "Windows installer" service again (see the section below)
  2. Stop "VMware vCenter Site Recovery Manager Client" service from Command Prompt
    net stop vmware-dr-client
  3. From Command Prompt patch file C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client\webapps\dr\WEB-INF\lib\log4j-core-2.13.0.jar
    "C:\Program Files\7-Zip\7z" d -tzip "C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client\webapps\dr\WEB-INF\lib\log4j-core-2.13.0.jar" org/apache/logging/log4j/core/lookup/JndiLookup.class
  4. From Command Prompt patch file C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client\webapps\dr.war
    1. Create a temp dir c:\temp
      mkdir c:\temp
    2. Unzip C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client\webapps\dr.war in c:\temp\dr folder
      "C:\Program Files\7-Zip\7z" x "C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client\webapps\dr.war" -o"c:\temp\dr"
    3. Patch file C:\temp\dr\WEB-INF\lib\log4j-core-*.jar
      "C:\Program Files\7-Zip\7z" d -tzip "C:\temp\dr\WEB-INF\lib\log4j-core-2.13.0.jar" org/apache/logging/log4j/core/lookup/JndiLookup.class
    4. Create a patched dr.war
      cd temp
      "C:\Program Files\7-Zip\7z" a -tzip dr.war ./dr/*
    5. Overwrite original file dr.war
      copy "C:\temp\dr.war" "C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\runtime\srm-client\webapps"
    6. If appropriate, delete temp folder c:\temp
  5. From Command Prompt patch file C:\Program Files\VMware\VMware vCenter Site Recovery Manager\tools\ls-verify\ls-verify-tool.jar (if it exists)
    "C:\Program Files\7-Zip\7z" d -tzip "C:\Program Files\VMware\VMware vCenter Site Recovery Manager\tools\ls-verify\ls-verify-tool.jar" org/apache/logging/log4j/core/lookup/JndiLookup.class
  6. Start "VMware vCenter Site Recovery Manager Client" service from Command Prompt:
    net start vmware-dr-client

Part 3 Disable "Windows Installer"

  1. Next we need to disable the "Windows Installer" service
    From the command prompt
    net stop MSIServer
  2. Run REG query "HKLM\SYSTEM\CurrentControlSet\services\MSIServer" /v Start to see current Startup type. Usually it is 3.
    REG query "HKLM\SYSTEM\CurrentControlSet\services\MSIServer" /v Start
    Note: Possible values are: 2 - Automatic; 3 - Manual; 4 - Disabled; 2 - Automatic (Delayed Start) but with DelayedAutoStart value to 1
  3. Run REG add "HKLM\SYSTEM\CurrentControlSet\services\MSIServer" /v Start /t REG_DWORD /d 4 /f  to disable "Windows Installer" service
    REG add "HKLM\SYSTEM\CurrentControlSet\services\MSIServer" /v Start /t REG_DWORD /d 4 /f
  4. Check new value with reg query "HKLM\SYSTEM\CurrentControlSet\services\MSIServer" /v Start. It should be 4
    REG query "HKLM\SYSTEM\CurrentControlSet\services\MSIServer" /v Start
     

Note: If you ever need to re-enable the "Windows Installer" In the future use the Startup type value from step 4 usually it is set to 3. From Command Prompt:

REG add "HKLM\SYSTEM\CurrentControlSet\services\MSIServer" /v Start /t REG_DWORD /d 3 /f

Additional Information

Note: Products older than SRM 8.3 are beyond End of General Support date, so patches will not be made available for these older versions. As there may be other non-validated exposure, VMware recommends updating EOGS versions to a supported release as a priority.
SRM 8.1.x and 8.2.x on Windows or Photon Appliances were shipped with log4j 1.x. Based on https://logging.apache.org/log4j/2.x/security.html we believe these versions are not exposed to CVE-2021-44228 because the product uses log4j 1.x and does not have JMSAppender configured in its log4j configuration. Customers are urged to monitor https://logging.apache.org/log4j/2.x/security.html as guidance from the Apache Logging team may change.  It is important to note that Log4j 1.x has reached end of life and is no longer supported by the Apache Logging team. As per https://logging.apache.org/log4j/2.x/security.html "Vulnerabilities reported after August 2015 against Log4j 1.x were not checked and will not be fixed". VMware strongly recommends upgrading to a supported version of SRM to minimize exposure.


Change log:
December 17th 2021 - 11:29PST: Updated guidance to indicate patched version availability
December 16th 2021 - 14:52PST: Added VMC on AWS DRaaS guidance and improve readability of KB
December 16th 2021 - 13:16 PST: Corrected missed item
December 16th 2021 - 08:42 PST: Updated script and added steps required to add environmental flag for appliance; in-progress steps to remove JndiLookup.class for windows.
December 15th 2021 - 09:26 PST: Added notice of intent.
December 15th 2021 - 06:43 PST: Added additional verification step for tomcat path; and clarification on older versions.
December 14th 2021 - 11:03 PST: Added screenshot of the registry key that needs modification for clarity.

Impact/Risks:

This issue impacts all releases prior to Site Recovery Manager 8.3.1.5, 8.4.0.4 and 8.5.0.2 and vSphere Replication 8.3.1.5, 8.4.0.4 and 8.5.0.2.
Note: This includes VMware Site Recovery on VMware Cloud on AWS Customers' on-prem appliances.

The workarounds described in this document are meant to be a temporary solution only. Issue mitigation is currently achieved by removal of the vulnerable code.
There is no functional impact to this workaround as these products do not leverage the vulnerable  "JndiLookup.class".
Note: Disabling the "Windows Installer" will prevent MSI applications from being Installed/Uninstalled/Modified or Repaired; The Auto-repair feature will also be interrupted. This is to prevent the SRM MSI from reverting the patch. You will need to follow the re-enabling steps (see the note in Part 3 Disable "Windows Installer") prior to updating your SRM install to the patched version for Windows (8.3.1.5)

Attachments

disable_log4j_srm get_app
disable_log4j_srm get_app
disable_log4j_srm get_app
disable_log4j_srm get_app
disable_log4j_srm get_app
Powered by Wolken Software