This issue is resolved in:
- vCenter Server 7.0 Update 3c, build 19234570.
- vCenter Server 6.7 Update 3q, build 19300125
- vCenter Server 6.5 Update 3s, build 19261680
Please note that it is not necessary to revert the workaround steps in this article before upgrading to a fixed release of vCenter Server.
Do not use the vc_log4j_mitigator.py script on vCenter Servers that have already been upgraded to a fixed version, such as 7.0 U3c.
Workaround:
The workarounds described in this document are meant to be a temporary solution only.
IMPORTANT: vc_log4j_mitigator.py will now mitigate CVE-2021-44228 and CVE-2021-45046 on vCenter Server end-to-end without extra steps. This script replaces the need to run remove_log4j_class.py and vmsa-2021-0028-kb87081.py independently. However, it is not necessary to run if you've already used those in your environment.
Please use the below 3 scenarios to verify if your previous steps were sufficient:
Completed remediation scenarios:
1. Used vc_log4j_mitigator.py
2. Used vmsa-2021-0028-kb87081.py script from KB 87088 and remove_log4j_class.py from this KB.
3. Used the manual workaround steps in this KB and remove_log4j_class.py.
If you are unsure if the previous steps were sufficient, run the vc_log4j_mitigator.py script with the "--dryrun" option to verify if the environment still has vulnerable files. If any are found, run it without the dryrun flag to correct them.
Automated Workaround (Recommended)
To apply the workaround for CVE-2021-44228 and CVE-2021-45046 with the unified script:
1. Download the script attached to this KB (see file details below):
File Name: vc_log4j_mitigator.py
File Size: 43.2 KB
SHA256 Checksum: 5CE86D29E7D968A61B4FE32F830A233DBC0BF861DCAF564389B7CE228A614616
2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)
3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCP
Note: It's necessary to enable the bash shell before WinSCP will work
4. Execute the script copied in step 1:
python vc_log4j_mitigator.py
This will stop all vCenter services, updates all necessary files with the formatMsgNoLookups flag, removes the JndiLookup.class from all jar/war files on the appliance, and finally starts all vCenter services. The files that the script modifies will be reported as the script runs.
5. To verify that no more vulnerable files exist, execute the script again with the dry run flag:
python vc_log4j_mitigator.py -r
The list of vulnerable files should be zero
Note: Certain 3rd party vSphere Client plugins may be detected as vulnerable even after remediation is done. This is because the jar files are recreated each time the client is started. These must be addressed by either patching the plugin to an unaffected version, marking it incompatible, or removing the plugin via the vCenter MOB. Disabling the plugin from the vSphere Client will not work.
This has been found with older versions of the Pure Storage HTML client plugin 4.3.1 and below (Pure login required) . The file continually marked as vulnerable will be: /usr/lib/vmware-vsphere-ui/server/work/Catalina/localhost/ROOT/eclipse/configuration/org.eclipse.osgi/321/0/.cp/log4j-core-2.11.2.jar.
Optional Arguments
-h, --help show this help message and exit
-d dirnames [dirnames ...], --directories dirnames [dirnames ...]
space separated list of directories to check
recursively for CVE-2021-44228 vulnerable java archive
files.
-a, --accept-services-restart
acccept the restart of the services without having
manual prompt confirmation for the same
-r, --dryrun Run the script and log vulnerable files without
mitigating them. The vCenter services are not
restarted with this option.
-b BACKUP_DIR, --backup-dir BACKUP_DIR
Specify a backup directory to store original files.
-l LOG_DIR, --log-dir LOG_DIR
Specify a directory to store log files.
Manual Workaround
VMware highly recommends using the vc_log4j_mitigator.py script instead of these manual steps to avoid errors in this process.
To manually apply the workaround for CVE-2021-44228 and CVE-2021-45046 to vCenter Server Appliance 7.x and 6.x, skip to the relevant sections below and follow the full instructions there:
Note: For vCenter Cloud Gateway, only the steps for the vMon Service, Analytics Service, and remove_log4j_class.py script are necessary.
vMON Service
1. Backup the existing java-wrapper-vmon file:
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
2. Update the java-wrapper-vmon file with a text editor such as vi:
vi /usr/lib/vmware-vmon/java-wrapper-vmon
3. At the very bottom of the file, replace the very last line with 2 new lines.
NOTE: Execute this step number '3' based on the vCenter version running in your environment. The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below:
- vCenter 7.0 Update 3, 3a, 3b
- vCenter 7.0 Update 2, 2a, 2b, 2c, 2d
Original
exec $java_start_bin $jvm_dynargs $security_dynargs $original_args
Updated
log4j_arg="-Dlog4j2.formatMsgNoLookups=true"
exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args
NOTE: The below update (mentioned in Step number 3) applies ONLY to the vCenter versions listed below:
- vCenter 7.0 GA, 7.0.0a, 7.0.0b, 7.0.0c, 7.0.0d
- vCenter 7.0 Update 1, U1a, U1c, U1d
Original
exec $java_start_bin $jvm_dynargs "$@"
Updated
log4j_arg="-Dlog4j2.formatMsgNoLookups=true"
exec $java_start_bin $jvm_dynargs $log4j_arg "$@"
4. Ensure the file permissions are set correctly with the below commands:
chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
5. Restart vCenter Services
service-control --stop --all
service-control --start --all
Update Manager Service
1. Back up the existing start.ini file.
cp -rfp /usr/lib/vmware-updatemgr/bin/jetty/start.ini /usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak
2. Update the start.ini file.
vi /usr/lib/vmware-updatemgr/bin/jetty/start.ini
3. Add the following line to the end of the file:
-Dlog4j2.formatMsgNoLookups=true
4. Restart the Update Manager Service with this command:
service-control --restart vmware-updatemgr
Analytics Service
This is now remediated with remove_log4j_class.py at the end of this section.
Run the remove_log4j_class.py script
1. Download the script attached to this KB (remove_log4j_class.py)2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCPNote: It's necessary to enable the bash shell before WinSCP will work 4. Execute the script copied in step 1:python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as "VULNERABLE FILE" as the script runs.
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
1. Verify if the vMon services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes includes -Dlog4j2.formatMsgNoLookups=true.
2. Verify the Update Manager changes are shown under "System Properties" in the output of the following two commands:
cd /usr/lib/vmware-updatemgr/bin/jetty/
java -jar start.jar --list-config
System Properties:
------------------
log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)
3. Verify the Analytics Service changes:
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
Note: This should return 0 lines.
4. Verify the script successfully removed JndiLookup.class from all java files with the following command:
python remove_log4j_class.py -r
The list of vulnerable files should now be empty.
2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
===== Summary =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.
vMON Service
1. Backup the existing java-wrapper-vmon file:
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
2. Update the java-wrapper-vmon file with a text editor such as vi:
vi /usr/lib/vmware-vmon/java-wrapper-vmon
3. At the very bottom of the file, replace the very last line with 2 new lines:
Original
exec $java_start_bin $jvm_dynargs "$@"
Updated
log4j_arg="-Dlog4j2.formatMsgNoLookups=true"
exec $java_start_bin $jvm_dynargs $log4j_arg "$@"
4. Restart vCenter Services:
service-control --stop --all
service-control --start --all
Note: If the services do not start, ensure the file permissions are set correctly with these commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
Secure Token Service
Note: These steps are only applicable to embedded vCenter and external PSC nodes
1. Back up and edit the the vmware-stsd file:
cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak
vi /etc/rc.d/init.d/vmware-stsd
2. Find the section labeled start_service(). Insert a new line near line 266, just before "$DAEMON_CLASS start" with "-Dlog4j2.formatMsgNoLookups=true \" as seen in the example:
start_service()
{
perform_pre_startup_actions
local retval
JAVA_MEM_ARGS=`/usr/sbin/cloudvm-ram-size -J vmware-stsd`
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
3. Restart the vmware-stsd service:
service-control --stop vmware-stsd
service-control --start vmware-stsd
Identity Management Service
Note: These steps are only applicable to embedded vCenter and external PSC nodes
1. Back up and edit the the vmware-sts-idmd file:
cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak
vi /etc/rc.d/init.d/vmware-sts-idmd
2. Insert a new line near line 177 before "$DEBUG_OPTS \" with "-Dlog4j2.formatMsgNoLookups=true \" as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
3. Restart the vmware-sts-idmd service:
service-control --stop vmware-sts-idmd
service-control --start vmware-sts-idmd
Analytics Service and CM Service
These are now remediated with remove_log4j_class.py at the end of this section.
Run the remove_log4j_class.py script
1. Download the script attached to this KB (remove_log4j_class.py)2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCPNote: It's necessary to enable the bash shell before WinSCP will work 4. Execute the script copied in step 1:python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as "VULNERABLE FILE" as the script runs.
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
1. Verify if the stsd, idmd, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
Note: This will also verify the workaround for the Analytics service in 6.7 U3p
2. Verify the Analytics Service changes:
grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l
Note: This should return 0 lines. This does not apply to vCenter 6.7 U3p.
3. Verify the CM Service changes:
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l
Note: This should return 0 lines.
4. Verify the script successfully removed JndiLookup.class from all java files with the following command:
python remove_log4j_class.py -r
The list of vulnerable files should now be empty.
2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
===== Summary =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.
vMON Service
1. Backup the existing java-wrapper-vmon file
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
2. Update the java-wrapper-vmon file with a text editor such as vi
vi /usr/lib/vmware-vmon/java-wrapper-vmon
3. At the very bottom of the file, replace the very last line with 2 new lines.
Original
exec $java_start_bin $jvm_dynargs "$@"
Updated
log4j_arg="-Dlog4j2.formatMsgNoLookups=true"
exec $java_start_bin $jvm_dynargs $log4j_arg "$@"
4. Restart vCenter Services:
service-control --stop --all
service-control --start --all
Note: If the services do not start, ensure the file permissions are set correctly with these commands:
- chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
- chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
Secure Token Service
Note: These steps are only applicable to embedded vCenter and external PSC nodes
1. Back up and edit the the vmware-stsd file:
cp /etc/rc.d/init.d/vmware-stsd /root/vmware-stsd.bak
vi /etc/rc.d/init.d/vmware-stsd
2. Find the section labeled start_service(). Insert a new line near line 266, just before "$DAEMON_CLASS start" with "-Dlog4j2.formatMsgNoLookups=true \" as seen in the example:
start_service()
{
perform_pre_startup_actions
local retval
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Dauditlog.dir=/var/log/audit/sso-events \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
3. Restart the vmware-stsd service
service-control --stop vmware-stsd
service-control --start vmware-stsd
Identity Management Service
Note: These steps are only applicable to embedded vCenter and external PSC nodes
1. Back up and edit the the vmware-sts-idmd file:
cp /etc/rc.d/init.d/vmware-sts-idmd /root/vmware-sts-idmd.bak
vi /etc/rc.d/init.d/vmware-sts-idmd
2. Insert a new line near line 177 before "$DEBUG_OPTS \" with "-Dlog4j2.formatMsgNoLookups=true \" as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-wait 120 \
-server \
<snip>
-Dlog4j.configurationFile=file://$PREFIX/share/config/log4j2.xml \
-Dlog4j2.formatMsgNoLookups=true \
$DEBUG_OPTS \
$DAEMON_CLASS
3. Restart the vmware-sts-idmd service:
service-control --stop vmware-sts-idmd
service-control --start vmware-sts-idmd
PSC Client Service
Note: These steps are only applicable to embedded vCenter and external PSC nodes
1. Back up and edit the vmware-psc-client file
cp -rfp /etc/rc.d/init.d/vmware-psc-client /root/vmware-psc-client.bak
vi /etc/rc.d/init.d/vmware-psc-client
2. Insert a new line near line 300, just before "$DAEMON_CLASS start" with "-Dlog4j2.formatMsgNoLookups=true \" as seen in the example:
$JSVC_BIN -procname $SERVICE_NAME \
-home $JAVA_HOME \
-server \
<snip>
-Djava.io.tmpdir="$CATALINA_BASE/temp" \
-Dlog4j2.formatMsgNoLookups=true \
$DAEMON_CLASS start
3. Restart the vmware-psc-client service:
service-control --stop vmware-psc-client
service-control --start vmware-psc-client
CM Service
This is now remediated with remove_log4j_class.py at the end of this section.
Run the remove_log4j_class.py script
1. Download the script attached to this KB (remove_log4j_class.py)2. Login to the vCSA using an SSH Client (using Putty.exe or any similar SSH Client)3. Transfer the file to /tmp folder on vCenter Server Appliance using WinSCPNote: It's necessary to enable the bash shell before WinSCP will work 4. Execute the script copied in step 1:python remove_log4j_class.py
The script will stop all vCenter services, proceed with removing the JndiLookup.class from all jar files on the appliance and finally start all vCenter services. The files that the script modifies will be reported as "VULNERABLE FILE" as the script runs.
Verify the changes
Once all sections are complete, use the following steps to confirm if they were implemented successfully.
1. Verify if the stsd, idmd, psc-client, and vMon controlled services were started with the new -Dlog4j2.formatMsgNoLookups=true parameter:
ps auxww | grep formatMsgNoLookups
Check if the processes include -Dlog4j2.formatMsgNoLookups=true
2. Verify the CM Service changes:
grep -i jndilookup /usr/lib/vmware-cm/lib/log4j-core.jar | wc -l
Note: This should return 0 lines.
3. Verify the script successfully removed JndiLookup.class from all java files with the following command:
python remove_log4j_class.py -r
The list of vulnerable files should now be empty.
2021-12-18T00:04:38 INFO main: Running in dryrun mode
2021-12-18T00:05:04 INFO main:
===== Summary =====
List of vulnerable files:
===========================
2021-12-18T00:05:04 INFO main: Done.
vCenter Server Appliance 6.0 U3j is no longer in general support but has also been identified as vulnerable to CVE-2021-44228 due to the Performance Charts service. The remove_log4j_class.py script is not verified in vCenter 6.0. Please use the manual steps below to remove the class. Mitigation steps have been identified as follows:
1. Back up and edit /usr/lib/vmware-perfcharts/wrapper/conf/wrapper.conf on the appliance and add a new line just below "wrapper.java.additional.13=-Dlog4j.configurationFile=file:/etc/vmware-perfcharts/log4j2.xml" (line 72) with the following content:
wrapper.java.additional.14=-Dlog4j2.formatMsgNoLookups=true
2: Back up the log4j-core-2.11.2.jar file
cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar.bak
3. Run the zip command to disable the class
zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
3. Stop the service and then start it through service-control:
service-control --stop vmware-perfcharts
service-control --start vmware-perfcharts
Note: vCenter Server Appliance versions 6.0GA - 6.0U3i are not vulnerable. However, versions 6.0 U3a/b/c/d/e/f were found to contain the following unused vulnerable jar files. No impact on the product has been observed after removing these jar files.
-
/opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-core-2.1.jar
-
/opt/pivotal/pivotal-tc-server-standard/templates/gemfire-p2p/lib/log4j-api-2.1.jar
-
/opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-core-2.1.jar
-
/opt/pivotal/pivotal-tc-server-standard/templates/gemfire-cs/lib/log4j-api-2.1.jar