Workaround instructions to address CVE-2021-44228 & CVE-2021-45046 in vCenter Server Windows

search cancel

Workaround instructions to address CVE-2021-44228 & CVE-2021-45046 in vCenter Server Windows

book

Article ID: 318830

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

CVE-2021-44228 & CVE-2021-45046 has been determined to impact Windows-based vCenter 6.7.x & vCenter 6.5.x via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

CVE-2021-44228 & CVE-2021-45046 - VMSA-2021-0028

IMPORTANT: This article pertains only to Windows-based vCenter Server 6.5 and 6.7. For information about the vCenter Server Appliance, see Workaround instructions to address CVE-2021-44228 in vCenter Server and vCenter Cloud Gateway (87081)

See the Change log at the end of this article for all changes and subscribe to the article for updates.

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.0.x
VMware vCenter Server 6.x
VMware vCenter Server 6.7.x

Resolution

This issue is resolved in:

vCenter Server 7.0 Update 3c, build 19234570
vCenter Server 6.7 Update 3q, build 19300125
vCenter Server 6.5 Update 3s, build 19261680

Please note that it is not necessary to revert the workaround steps in this article before upgrading to a fixed release of vCenter Server.
Do not use the vc_log4j_mitigator.py script on vCenter Servers that have already been upgraded to a fixed version.

Workaround:

The workarounds described in this document are meant to be a temporary solution only.

IMPORTANT: vc_log4j_mitigator.py will now mitigate CVE-2021-44228 & CVE-2021-45046 on vCenter Server end-to-end without extra steps. This script replaces the need to run any of the manual steps or use remove_log4j_class.py. However, it is not necessary to run if you've already used those in your environment.

Please use the below 2 scenarios to verify if you're previous steps were sufficient:

Completed remediation scenarios:
1. Used vc_log4j_mitigator.py
or
2. Used the manual workaround steps in this KB and remove_log4j_class.py.

If you are unsure if the previous steps were sufficient, run the vc_log4j_mitigator.py script with the "--dryrun" option to verify if the environment still has vulnerable files. If any are found, run it without the dryrun flag to correct them.

Automated Workaround (Recommended)

To apply the workaround for CVE-2021-44228 & CVE-2021-45046with the unified script:

1. Download the script attached to this KB (vc_log4j_mitigator.py)

2. Open an administrator cmd prompt and navigate to the location of the script.

3. Execute the script copied in step 1:

"%VMWARE_PYTHON_BIN%" vc_log4j_mitigator.py

This will stop all vCenter services, updates all necessary files with the formatMsgNoLookups flag, removes the JndiLookup.class from all jar/war files on the appliance, and finally starts all vCenter services. The files that the script modifies will be reported as the script runs.

5. To verify that no more vulnerable files exist, execute the script again with the "dry run" flag:

"%VMWARE_PYTHON_BIN%" vc_log4j_mitigator.py -r

The list of vulnerable files found should be zero. The vCenter Server is now fully mitigated.

Note: If you have issues with running the script in Windows, run the following:

echo "%VMWARE_PYTHON_BIN%"

Use the resulting value for the path to python.exe. For example:

"C:\Program Files\VMware\vCenter Server\python\python.exe" vc_log4j_mitigator.py

Note: Certain 3rd party vSphere Client plugins may be detected as vulnerable even after remediation is done. This is because the jar files are recreated each time the client is started. These must be addressed by either patching the plugin to an unaffected version or removing the plugin via the vCenter MOB.

This has been found with older versions of the Pure Storage HTML client plugin 4.3.1 and below (Pure login required) . The file continually marked as vulnerable will be: ProgramData\VMware\vCenterServer\runtime\vsphere-ui\server\work\Catalina\localhost\ROOT\eclipse\configuration\org.eclipse.osgi\330\0\.cp\log4j-core-2.11.2.jar

Optional Arguments
-h, --help            show this help message and exit
-d dirnames [dirnames ...], --directories dirnames [dirnames ...]
                        space separated list of directories to check
                        recursively for CVE-2021-44228 vulnerable java archive
                        files.
-a, --accept-services-restart
                        accept the restart of the services without having
                        manual prompt confirmation for the same
-r, --dryrun          Run the script and log vulnerable files without
                        mitigating them. The vCenter services are not
                        restarted with this option.
-b BACKUP_DIR, --backup-dir BACKUP_DIR
                        Specify a backup directory to store original files.
-l LOG_DIR, --log-dir LOG_DIR
                        Specify a directory to store log files.

Manual Workaround

VMware highly recommends using the vc_log4j_mitigator.py script instead of these manual steps to avoid errors in this process.

vMON Services
(Applicable to all vCenter 6.5/6.7 embedded, vCenter, and PSC nodes)

Step 1: Stop the services within the Windows vCenter.

C:\Program Files\VMware\vCenter Server\bin> .\service-control --stop --all

Step 2: Make a backup of the below two .json files in a different location than the originals and then manually edit them to remove every line starting with "//". This must be completed before running the script in the next step.

(Note: Step 2 is not applicable to external PSC nodes as these files won't exist on them. The rest of the steps do still apply)

C:\ProgramData\VMware\vCenterServer\cfg\vmware-vmon\svcCfgfiles\vsphere-ui.json

example of types of lines to remove
// Enable remote debugging
// NOTE: Use this option only when you really need it. Don't keep it on by default.
//       It has the potential to cause memory leaks. For further details, see
//       https://bugs.openjdk.java.net/browse/JDK-8164921 as well as our own
//       observations at PR 1878411, comments 21, 33, 34, and 35
//"-Xdebug",
//"-Xnoagent",
//"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8002",
// Enable JMX
//"-Dcom.sun.management.jmxremote",
//"-Dcom.sun.management.jmxremote.port=9876",
//"-Dcom.sun.management.jmxremote.local.only=false",
//"-Dcom.sun.management.jmxremote.authenticate=false",
//"-Dcom.sun.management.jmxremote.ssl=false",5-bri

C:\ProgramData\VMware\vCenterServer\cfg\vmware-vmon\svcCfgfiles\vsphere-client.json

example of types of lines to remove
// This option will be removed soon. See JIRA VSUIP-180
// Enable remote debugging
// NOTE: Use this option only when you really need it. Don't keep it on by default.
//       It has the potential to cause memory leaks. For further details, see
//       https://bugs.openjdk.java.net/browse/JDK-8164921 as well as our own
//       observations at PR 1878411, comments 21, 33, 34, and 35
//"-Xdebug",
//"-Xnoagent",
//"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8001",
// Enable JMX
//"-Dcom.sun.management.jmxremote",
//"-Dcom.sun.management.jmxremote.port=9875",
//"-Dcom.sun.management.jmxremote.local.only=false",
//"-Dcom.sun.management.jmxremote.authenticate=false",
//"-Dcom.sun.management.jmxremote.ssl=false",

Note: Do not simply uncomment these lines. Remove them completely.

Step 3: Execute the python script named vMON.py attached to this KB article

The script can be executed from any path using the following command:

C:\%VMWARE_PYTHON_BIN% vMON.py

Note: If you have issues with running the script in Windows, run the following:

echo "%VMWARE_PYTHON_BIN%"

Use the resulting value for the path to python.exe. For example:
"C:\Program Files\VMware\vCenter Server\python\python.exe" vMON.py

Step 4: Run the following commands to restart all services via vMON:

C:\Program Files\VMware\vCenter Server\bin> .\service-control --stop --all
C:\Program Files\VMware\vCenter Server\bin> .\service-control --start --all
C:\Program Files\VMware\vCenter Server\bin> .\service-control --status

Secure Token Services (STS)
(Applicable only to vCenter 6.5/6.7 embedded and PSC nodes)

Step 1: Backup the file manually from the below location
- C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\conf\wrapper.conf
- Copy this file to C:\old-svcCfgfiles\sts-wrapper.conf
Step 2: Edit and add the following line at the end of "# Java Additional Parameters" section

wrapper.java.additional.27="-Dlog4j2.formatMsgNoLookups=true"

Output of the file wrapper.conf sample, add the RED line below

PSC Client
(Applicable only to vCenter 6.5 embedded and PSC nodes)

Step 1: Manually backup the file c:\ProgramData\VMware\vCenterServer\runtime\vmware-psc-client\conf\wrapper.conf to c:\old-svcCfgfiles\psc-client-wrapper.conf
Step 2: Edit c:\ProgramData\VMware\vCenterServer\runtime\vmware-psc-client\conf\wrapper.conf
and add the following line wrapper.java.additional.23="-Dlog4j2.formatMsgNoLookups=true" at the end of the "# Java Additional Parameters" section.

The below is the output of the file psc-client-wrapper.conf, add the RED line below

#********************************************************************
# Wrapper License Properties (Ignored by Community Edition)
#********************************************************************
#encoding=UTF-8
wrapper.license.type=DEV
wrapper.license.id=201702150000053
wrapper.license.licensee=VMware Global, Inc.
wrapper.license.group=Ops
wrapper.license.dev_application=VMware Platform Services Controller Client
wrapper.license.features=pro, 64bit
wrapper.license.upgrade_term.begin_date=2009-10-27
wrapper.license.upgrade_term.end_date=2018-01-27
wrapper.license.key.1=075b-bf64-8f5d-84af
wrapper.license.key.2=a8b6-b28d-62e3-072c
wrapper.license.key.3=962d-5f1d-f01d-d6ff
wrapper.license.key.4=b460-2f95-3124-0ebb
#********************************************************************
# Wrapper Properties
#********************************************************************
set.CATALINA_HOME=%VMWARE_CIS_HOME%\apachetomcat
set.CATALINA_BASE=%VMWARE_RUNTIME_DATA_DIR%\vmware-psc-client
# Java Main class. This class must implement the WrapperListener interface
# or guarantee that the WrapperManager class is initialized. Helper
# classes are provided to do this for you. See the Integration section
# of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
# Java Home
set.JAVA_HOME=%VMWARE_JAVA_HOME%
# Java Additional Parameters
wrapper.java.additional.1="-Djava.endorsed.dirs=%CATALINA_HOME%\common\endorsed"
wrapper.java.additional.2="-Dcatalina.base=%CATALINA_BASE%"
wrapper.java.additional.3="-Dcatalina.home=%CATALINA_HOME%"
wrapper.java.additional.4="-Djava.io.tmpdir=%CATALINA_BASE%\temp"
wrapper.java.additional.5="-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager"
wrapper.java.additional.6="-Djava.util.logging.config.file=%CATALINA_BASE%\conf\logging.properties"
wrapper.java.additional.7="-Dwrapper.dump.port=-1"
wrapper.java.additional.8="-Xmx512M"
wrapper.java.additional.9="-Xss256K"
wrapper.java.additional.10="-Djdk.map.althashing.threshold=512"
wrapper.java.additional.11="-XX:MaxPermSize=256m"
wrapper.java.additional.12="-XX:+HeapDumpOnOutOfMemoryError"
wrapper.java.additional.13="-XX:HeapDumpPath=%CATALINA_BASE%\logs"
wrapper.java.additional.14="-XX:+PrintGCDetails"
wrapper.java.additional.15="-XX:+PrintGCDateStamps"
wrapper.java.additional.16="-XX:+PrintTenuringDistribution"
wrapper.java.additional.17="-Xloggc:%CATALINA_BASE%\logs\gc.log"
#********************************************************************
# Wrapper License Properties (Ignored by Community Edition)
#********************************************************************
#encoding=UTF-8
wrapper.license.type=DEV
wrapper.license.id=201702150000053
wrapper.license.licensee=VMware Global, Inc.
wrapper.license.group=Ops
wrapper.license.dev_application=VMware Platform Services Controller Client
wrapper.license.features=pro, 64bit
wrapper.license.upgrade_term.begin_date=2009-10-27
wrapper.license.upgrade_term.end_date=2018-01-27
wrapper.license.key.1=075b-bf64-8f5d-84af
wrapper.license.key.2=a8b6-b28d-62e3-072c
wrapper.license.key.3=962d-5f1d-f01d-d6ff
wrapper.license.key.4=b460-2f95-3124-0ebb
#********************************************************************
# Wrapper Properties
#********************************************************************
set.CATALINA_HOME=%VMWARE_CIS_HOME%\apachetomcat
set.CATALINA_BASE=%VMWARE_RUNTIME_DATA_DIR%\vmware-psc-client
# Java Main class. This class must implement the WrapperListener interface
# or guarantee that the WrapperManager class is initialized. Helper
# classes are provided to do this for you. See the Integration section
# of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
# Java Home
set.JAVA_HOME=%VMWARE_JAVA_HOME%
# Java Additional Parameters
wrapper.java.additional.1="-Djava.endorsed.dirs=%CATALINA_HOME%\common\endorsed"
wrapper.java.additional.2="-Dcatalina.base=%CATALINA_BASE%"
wrapper.java.additional.3="-Dcatalina.home=%CATALINA_HOME%"
wrapper.java.additional.4="-Djava.io.tmpdir=%CATALINA_BASE%\temp"
wrapper.java.additional.5="-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager"
wrapper.java.additional.6="-Djava.util.logging.config.file=%CATALINA_BASE%\conf\logging.properties"
wrapper.java.additional.7="-Dwrapper.dump.port=-1"
wrapper.java.additional.8="-Xmx512M"
wrapper.java.additional.9="-Xss256K"
wrapper.java.additional.10="-Djdk.map.althashing.threshold=512"
wrapper.java.additional.11="-XX:MaxPermSize=256m"
wrapper.java.additional.12="-XX:+HeapDumpOnOutOfMemoryError"
wrapper.java.additional.13="-XX:HeapDumpPath=%CATALINA_BASE%\logs"
wrapper.java.additional.14="-XX:+PrintGCDetails"
wrapper.java.additional.15="-XX:+PrintGCDateStamps"
wrapper.java.additional.16="-XX:+PrintTenuringDistribution"
wrapper.java.additional.17="-Xloggc:%CATALINA_BASE%\logs\gc.log"
wrapper.java.additional.18="-XX:+UseGCLogFileRotation"
wrapper.java.additional.19="-XX:NumberOfGCLogFiles=2"
wrapper.java.additional.20="-XX:GCLogFileSize=5M"
wrapper.java.additional.21="-XX:ErrorFile=%CATALINA_BASE%\logs\hs_err_lotus_pid%p.log"
wrapper.java.additional.22="-Djava.security.properties=%VMWARE_CFG_DIR%\java\vmware-override-java.security"
wrapper.java.additional.23="-Dlog4j2.formatMsgNoLookups=true"
....

Identity Management Service
(Applicable only to vCenter Server 6.5/6.7 embedded and PSC nodes)

Step 1: Make sure you have a full a backup of the Registry before moving forward
Step 2: Update registry by launching regedit and change the values :

Regedit HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\VMwareIdentityMgmtService\Parameters\Java

Key to modify:
------------------
key - Options
value - in edit box go to end and after --XX:ThreadStackSize=228k Shift+Enter then append
-Dlog4j2.formatMsgNoLookups=true

Reference: complete value of the registry looks as below

-Djava.security.policy=C:\Program Files\VMware\vCenter Server\VMware Identity Services\\server_policy.txt
-Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true
-Dorg.apache.xml.security.ignoreLineBreaks=true
-XX:ErrorFile=C:\ProgramData\VMware\vCenterServer\logs\sso\hs_err_idm_pid%p.log
-XX:HeapDumpPath=C:\ProgramData\VMware\vCenterServer\logs\sso\
-Dvmware.log.dir=C:\ProgramData\VMware\vCenterServer\logs\sso\
-Xmx168m
-XX:CompressedClassSpaceSize=160m
-Xss256k
-XX:ParallelGCThreads=1
-Dlog4j2.formatMsgNoLookups=true

Alternatively

Use the folliwng Powershell script to update the registry:

$opts = $((Get-ItemProperty 'HKLM:\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\VMwareIdentityMgmtService\Parameters\Java').Options)
$opts1 = $opts + "-Dlog4j2.formatMsgNoLookups=true"
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\VMwareIdentityMgmtService\Parameters\Java' -Name 'Options' -Value $opts1

Component Manager

This is now remediated with the remove_log4j_class.py script at the end of this section.

Run the remove_log4j_class.py script

1. Download the remove_log4j_class.py script attached to this KB to any directory on the vCenter Server.

2. Open an administrator cmd prompt and navigate to the location of the script.

3. Execute the script:

%VMWARE_PYTHON_BIN% remove_log4j_class.py

The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files in the VMware installation directories, and finally start all vCenter services.

Note: If you have issues with running the script in Windows, run the following:

echo "%VMWARE_PYTHON_BIN%"

Use the resulting value for the path to python.exe. For example:

"C:\Program Files\VMware\vCenter Server\python\python.exe" remove_log4j_class.py

Optional Arguments
-h, --help            show this help message and exit
-d dirnames [dirnames ...], --directories dirnames [dirnames ...]
                        space separated list of directories to check
                        recursively for CVE-2021-44228 vulnerable files.
-a, --accept-services-restart
                        accept the restart of the services without having
                        manual confirmation for the same
-r, --dryrun          Run the script and log vulnerable files without
                        mitigating them.
-b BACKUP_DIR, --backup-dir BACKUP_DIR
                        Specify a backup directory to store original files.

vCenter Server 6.0 U3j on Windows

vCenter Server 6.0 is no longer in general support but has also been identified as vulnerable to CVE-2021-44228 & CVE-2021-45046 due to the Performance Charts service. The remove_log4j_class.py script is not verified in vCenter 6.0. Please use the manual steps below to remove the class. Mitigation steps have been identified as follows:

1. Stop the vCenter services

2. Backup and edit C:\Program Files\VMware\vCenter Server\perfcharts\wrapper\conf\wrapper.conf on the appliance and add a new line below "wrapper.java.additional.13=-Dlog4j.configurationFile=file:" with the following content:

wrapper.java.additional.14=-Dlog4j2.formatMsgNoLookups=true

3: Back up the following log4j-core-2.8.2.jar file to a different directory than the original:

C:\Program Files\VMware\vCenter Server\common-jars\log4j-core-2.11.2.jar

4: Rename the log4j-core-2.11.2.jar file to log4j-core-2.11.2.jar.zip

Note: It might be necessary to enable "file name extensions" in the Windows explorer view to navigate into the renamed zip file

5. Double click on the log4j-core-2.11.2.jar.zip

6. Navigate to the folder log4j-core-2.11.2.jar.zip\org\apache\logging\log4j\core\lookup

7. Delete the file JndiLookup.class

Rename the log4j-core-2.11.2.jar.zip file back to log4j-core-2.11.2.jar

8. Start the vCenter services

Note: vCenter Server Windows versions 6.0GA - 6.0U3i are not vulnerable. However, versions 6.0 U3a/b/c/d/e/f were found to contain the following unused vulnerable jar files. No impact on the product has been observed after removing these jar files.

C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-cs\lib\log4j-api-2.1.jar
C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-cs\lib\log4j-core-2.1.jar
C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-p2p\lib\log4j-api-2.1.jar
C:\Program Files\VMware\vCenter Server\tcServer\templates\gemfire-p2p\lib\log4j-core-2.1.jar

Verification Steps

Verify the script successfully removed JndiLookup.class from all java files with the following command:

%VMWARE_PYTHON_BIN% remove_log4j_class.py -r

The list of vulnerable files should now be empty. Re-run the script if it is not.

2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
===== Summary =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.

Look for java services started by vMON for new Log4j option, Go to Windows task manager.

For Tanuki wrapper services, check the wrapper service command, check in process explorer (if you have it).

STS

65_ProcessExplorer_hover_tooltip_wrapper.exe_java_exe_STSService.png

PSC client

65_ProcessExplorer_hover_tooltip_wrapper.exe_java_exe_IDMD_service.png

(OR) Use the following command in power shell to verify all java services

Get-WmiObject Win32_Process -Filter "name = 'java.exe'" | Select-Object CommandLine | fl

Sample output:

Additional Information

To revert the workarounds, replace the modified files with the backups created in each step.

Change log :

December 12th 2021 - 19:45 PST: Added information pertaining to vCenter 6.0. Updated various sections to improve clarity.
December 12th 2021 - 20:45 PST: Updated versions with vulnerable jar files in VCSA 6.0.
December 13th 2021 - 12:45 PST: Added Impacts/Risks section. Updated wording in vmon steps. Changed highlighted examples to red text.
December 15th 2021 - 11:15 PST: Added a notice at the top concerning the recent updates on CVE-2021-44228 from Apache. Updated steps under Component Manager to address "file in use" problem.
December 16th 2021 - 16:15 PST: Added new remove_log4j_class.py script which replaces the Component Manager step and addresses new findings in CVE-2021-44228 and CVE-2021-45046.
December 17th 2021 - 11:45 PST: Added step to vCenter Server 6.0 U3j instructions to manually remove the JndiLookup.class class.
December 17th 2021 - 17:00 PST: Updated remove_log4j_class.py to output a change summary and better handle "File is not a zip file" issues. Added section with new cmd line options for remove_log4j_class.py. Added "--dryrun" verification step.
December 20th 2021 - 18:00 PST: Added vc_log4j_mitigator.py which unifies the previous steps and scripts into a single workaround.
December 21st 2021 - 1:45 PST: Added note about 3rd party plugins reverting to a vulnerable state after the vsphere client is started. Added clarification that CVE-45046 is also addressed with all the changes since the "December 16th 2021 - 16:15 PST" update
December 22nd 2021 - 12:00 PST: Added extra information about known older versions of the Pure Storage plugin which need extra steps to remediate
January 21st 2022 - 09:00 PST: Changed the Automated Workaround (Recommended) step 3 from %VMWARE_PYTHON_BIN% vc_log4j_mitigator.py to "%VMWARE_PYTHON_BIN%" vc_log4j_mitigator.py
January 21st 2022 - 09:00 PST: Changed the Automated Workaround (Recommended) step 5 from %VMWARE_PYTHON_BIN% vc_log4j_mitigator.py -r to "%VMWARE_PYTHON_BIN%" vc_log4j_mitigator.py -r
January 27th 2022 - 19:30 PST: Updated resolution section with the vCenter 7.0 U3c release.
February 8th 2022 - 09:00 PST: Added 6.x versions to resolution section.

Impact/Risks:

This article contains windows registry edit steps. Ensure that a working offline snapshot or backup of the vCenter (or all VCs if ELM mode) exists before starting.
Environments with external PSCs need to have the steps taken on both vCenter and PSC appliances.
After completing the steps, do not attempt to upgrade the Windows vCenter Server. Attempting to upgrade the vCenter Server will revert the remediations taken or fail outright. VMware will provide steps to address this in a future release.

Attachments

remove_log4j_class get_app

vMON get_app

vc_log4j_mitigator get_app

Feedback

thumb_up Yes

thumb_down No