- The workarounds described in this document are meant to be a temporary solution only.
- Upgrades documented in the VMSA-2021-0028 should be applied to remediate CVE-2021-44228 & CVE-2021-45046 as and when upgrades become available.
Workaround:
To Apply the workaround for CVE-2021-44228 and CVE-2021-45046 perform the following steps
- Click here for the workaround steps for Cloud Builder
- Click here for the workaround steps for SDDC Manager
- Click here for the workaround steps for Other VMware products within VMware Cloud Foundation BOM
- Click here for further information on VxRail Manager workaround to remediate Log4Shell
Steps for Cloud Builder
To apply the workaround for CVE-2021-44228 follow below steps
Note: These steps are applicable for all VCF 3.x and 4.x Cloud Builders.
- Please make sure all imaging runs are completed and none is in-progress before proceeding with this procedure.
- Login to the Cloud Builder VM
- Change the user to 'root' account using command
- Run the below command, to stop Host Imaging Service
- Run the below command to check and confirm the service is stopped
- Run the below command, to take the backup of the original file before editing it
- cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh.orig
- Modify /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh to add -Dlog4j2.formatMsgNoLookups=true as an extra argument for running the parent imaging service in "nohup /etc/alternatives/jre/bin/java -jar". Make sure this argument is added after -Dserver.port=$VIA_SERVICE_PORT
- The following is given as an example. Please make sure the equivalent command in the file is modified accordingly:
- BEFORE:
- nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$VIA_SERVICE_PORT -Dspring.config.additional-location=$VIA_EXTERNAL_PROPERTIES_PATH,$VIA_DB_PROPERTIES_FILE -Dserver.servlet.context-path=$VIA_CONTEXT_PATH $VIA_SERVICE_PATH < /dev/null >>$LOGFILE 2>&1 &
- AFTER:
- nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$VIA_SERVICE_PORT -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=$VIA_EXTERNAL_PROPERTIES_PATH,$VIA_DB_PROPERTIES_FILE -Dserver.servlet.context-path=$VIA_CONTEXT_PATH $VIA_SERVICE_PATH < /dev/null >>$LOGFILE 2>&1 &
- Run the below command, to take the backup of the original file before editing it
- cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh.orig
- Modify /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh to add -Dlog4j2.formatMsgNoLookups=true as an extra argument for running the imaging service in "nohup /etc/alternatives/jre/bin/java -jar". Make sure this argument is added after -Dserver.port=$SECOND. The following is given as an example. Please make sure the equivalent command in the file is modified accordingly:
- BEFORE:
- nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$SECOND -Dspring.config.additional-location=$VIA_DB_PROPERTIES_FILE $name < /dev/null >>$LOGFILE 2>&1 &
- AFTER:
- nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$SECOND -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=$VIA_DB_PROPERTIES_FILE $name < /dev/null >>$LOGFILE 2>&1 &
- Run the below command, to start the Host Imaging Service
- Run the below command, to confirm Host Imaging service is up and running
- systemctl status imaging
- Note: In the event of service startup failure, please revert the config files by following below steps to bring up the service and you may contact VMware Global Support to apply the workaround steps to mitigate the vulnerability :
- Run the below command, to stop Host Imaging Service
- Run the below command to confirm the service is stopped
- systemctl status imaging
- Copy the Original files back :
- cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh.orig /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh
- cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh.orig /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh
- Change the file permissions
- chmod 755 /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh
- chmod 755 /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh
- Restart the imaging service
- Run the below command to confirm Host Imaging Service is run with -Dlog4j2.formatMsgNoLookups=true option. The output should look something like the following ones
root 6177 1 1 00:36 ? 00:00:26 /etc/alternatives/jre/bin/java -jar -Dserver.port=8445 -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=/opt/vmware/evorack-imaging/config/via.properties,/opt/vmware/evorack-imaging/config/via-db-ext.properties -Dserver.servlet.context-path=/via /opt/vmware/evorack-imaging/services/evorack-imaging-services/via.jar
root 6647 5750 1 00:36 ? 00:00:21 /etc/alternatives/jre/bin/java -jar -Dserver.port=8081 -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=/opt/vmware/evorack-imaging/config/via-db-ext.properties /opt/vmware/evorack-imaging/services/evorack-imaging-services/evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar
root 10247 3857 0 01:01 pts/0 00:00:00 grep --color=auto jar
To apply the workaround for CVE-2021-45046 follow below steps
Note: These steps are applicable for all VCF 3.x and 4.x Cloud Builders.
There are 2 jars that require mitigation in cloud builder - Follow the steps below to fix the same
- Log on to Cloud Builder using SSH
- Change the user to 'root' account using command
- Stop the imaging service
- Handling the vulnerability in evorack-imaging-esxi-service jar
- Create a temporary directory with the command below
- mktemp -d -t log4j-XXXXXXXXXX
- Run the command below to list if log4j-core is present in the jar file evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar
- zipinfo -1 "/opt/vmware/evorack-imaging/services/evorack-imaging-services/evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar" | grep log4j-core-*
- This lists the presence of log4j-core-*.jar in the above jar which confirms the vulnerability
- Copy the temp directory path created as part of Step #4.1 and execute below commands
- In this instance the temp folder gets created with name log4j-KANsTCxgjM
- Example command: unzip "/opt/vmware/evorack-imaging/services/evorack-imaging-services/evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar" "BOOT-INF/lib/log4j-core-2.13.1.jar" -d "/tmp/log4j-KANsTCxgjM"
- Similarly, replace the temporary folder names in below commands where applicable with created temp folder name from Step #5
- zipinfo -1 /tmp/log4j-KANsTCxgjM/BOOT-INF/lib/log4j-core-2.13.1.jar | grep -q "org/apache/logging/log4j/core/lookup/JndiLookup.class"
- Next run command echo $?
- If the returned value is 1 - No further action needed as the class is not present then move to step below "Handling the vulnerability in via.jar" Step#5
- If returned value is 0 - Means the JndiLookup.class is present and continue running commands below -
- zip -q -d "/tmp/log4j-KANsTCxgjM/BOOT-INF/lib/log4j-core-2.13.1.jar" "org/apache/logging/log4j/core/lookup/JndiLookup.class"
- This deletes the JndiLookup.class file from the log4j-core*.jar
- Update the changed log4j-core to the evorack-imaging jar using command below
- cd "/tmp/log4j-KANsTCxgjM/" && zip -ur -0 "/opt/vmware/evorack-imaging/services/evorack-imaging-services/evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar"
5. Handling the vulnerability in via.jar
- Create a temporary directory with the command - mktemp -d -t log4j-XXXXXXXXXX
- Run the command below to list if log4j-core is present in the jar via.jar
- zipinfo -1 "/opt/vmware/evorack-imaging/services/evorack-imaging-services/via.jar" | grep log4j-core-*
- This lists the presence of log4j-core-*.jar in the above jar which confirms the vulnerability
- Copy the temp directory path created as part of Step #5.1 and execute below commands
- unzip "/opt/vmware/evorack-imaging/services/evorack-imaging-services/via.jar" "BOOT-INF/lib/log4j-core-2.13.1.jar" -d "/tmp/log4j-mO2dWNlgef"
- zipinfo -1 /tmp/log4j-mO2dWNlgef/BOOT-INF/lib/log4j-core-2.13.1.jar | grep -q "org/apache/logging/log4j/core/lookup/JndiLookup.class"
- Run the below command
- echo $?
- If the returned value is 1 - No further action needed as the class is not present then move to last step below of "Start the imaging service" Step#5
- If returned value is 0 - Means the JndiLookup.class is present and continue running commands below
- zip -q -d "/tmp/log4j-mO2dWNlgef/BOOT-INF/lib/log4j-core-2.13.1.jar" "org/apache/logging/log4j/core/lookup/JndiLookup.class"
- This deletes the JndiLookup.class file from the log4j-core*.jar
- Update the changed log4j-core to the via.jar using command below
- cd "/tmp/log4j-mO2dWNlgef/" && zip -ur -0 "/opt/vmware/evorack-imaging/services/evorack-imaging-services/via.jar"
- Start the imaging service
Cloud Builder Workaround Automation StepsNOTE :
- This automated script addresses both CVE-2021-44228 and CVE-2021-45046
- It is recommended to run the script even in case any earlier workaround was applied for CVE-2021-44228
- Copy the script file named "log4j_via_v3.sh" attached in this KB article to your Cloud Builder via WinSCP, to the /home/admin directory
- Log on to Cloud Builder using SSH
- Change the user to ‘root’ account using the command below
- Change the directory to the location to /home/admin
- Execute the command below:
Note: script is idempotent and can be repeatedly run: it will figure out that the JndiLookup.class has already been removed and do nothing.
Steps for SDDC ManagerNote: Following steps are applicable for all VCF 3.x (except VCF 3.10.2, VCF 3.10.2.1 & VCF 3.10.2.2) and 4.x versions.
If there has been an upgrade from any of the 3.10.1.x to 3.10.2.x, and if the remediation has not been applied prior to the upgrade, then please follow on the steps to apply remediation even if you are on 3.10.2.x (post upgrade).
To apply the workaround for CVE-2021-44228 follow below steps
- Login to the SDDC-Manager using SSH :
- ssh vcf@<ip/dnsname_of_sddc_manager>
- Enter command su and provide the root user password to get super user privileges
- Backup the file
- /usr/local/vip/bin/start-vip.sh
- cp /usr/local/vip/bin/start-vip.sh /usr/local/vip/bin/start-vip.sh.orig
- Modify the file /usr/local/vip/bin/start-vip.sh using vi editor vi /usr/local/vip/bin/start-vip.sh
Add -DLOG4J_FORMAT_MSG_NO_LOOKUPS=true after -Djava.compiler=NONE as the next flag in the line (Approx. line 32 col 135)
The following is given as an example. Please make sure the equivalent command in the file is modified accordingly
BEFORE:
nohup $JAVA -jar -Dapp.log.home=/var/log/vmware/vip -server -XX:MaxMetaspaceSize=64m -XX:ParallelGCThreads=2 -Djava.compiler=NONE $1 --server.scheme=http --server.http.port=7900> $2 2>&1 &
AFTER:
nohup $JAVA -jar -Dapp.log.home=/var/log/vmware/vip -server -XX:MaxMetaspaceSize=64m -XX:ParallelGCThreads=2 -Djava.compiler=NONE -DLOG4J_FORMAT_MSG_NO_LOOKUPS=true $1 --server.scheme=http --server.http.port=7900> $2 2>&1 &
- Restart the vip-manager-i18n service
systemctl restart vip-manager-i18n.service
- Run the below command to confirm VIP Manager Service is run with -DLOG4J_FORMAT_MSG_NO_LOOKUPS=true option. systemctl status -l vip-manager-i18n.service
- The output should look something like the following ones:
vip-manager-i18n.service - VMware Internationalization Service
Loaded: loaded (/etc/systemd/system/vip-manager-i18n.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2021-12-12 05:00:52 UTC; 16s ago
Process: 61736 ExecStop=/usr/local/vip/bin/init.sh stop (code=exited, status=0/SUCCESS)
Process: 61755 ExecStart=/usr/local/vip/bin/init.sh start (code=exited, status=0/SUCCESS)
Main PID: 61781 (java)
Tasks: 10 (limit: 19191)
Memory: 139.4M
CGroup: /system.slice/vip-manager-i18n.service
`-61781 /usr/bin/java -jar -Dapp.log.home=/var/log/vmware/vip -server -XX:MaxMetaspaceSize=64m -XX:ParallelGCThreads=2 -Djava.compiler=NONE -DLOG4J_FORMAT_MSG_NO_LOOKUPS=true /usr/local/vip/vip-manager-i18n-common.jar --vipservice.cross.domain.alloworigin=sddc-manager.vrack.vsphere.local --server.scheme=http --server.http.port=7900
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local init.sh[61755]: start VIP service
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local init.sh[61755]: execute start function
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local init.sh[61755]: executing: /usr/local/vip/bin/start-vip.sh /usr/local/vip/vip-manager-i18n-common.jar /usr/local/vip/work/vip-runtime.log /usr/local/vip/work
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local init.sh[61755]: =====startup vip=====
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local init.sh[61755]: found java home: /etc/alternatives/jre
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local init.sh[61755]: run vip from: /usr/local/vip/vip-manager-i18n-common.jar
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local init.sh[61755]: log file: /var/log/vmware/vip/vip-runtime.log
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local init.sh[61755]: vip service is started!
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local init.sh[61755]: end of starting VIP service
Dec 12 05:00:52 sddc-manager.vrack.vsphere.local systemd[1]: Started VMware Internationalization Service.
Note:In the event of failure in the above steps revert the files by following below steps to bring up the service and you may contact VMware Global Support to apply the workaround steps to mitigate the vulnerability :
- Revert the config file
- cp /usr/local/vip/bin/start-vip.sh.orig /usr/local/vip/bin/start-vip.sh
- Change the file ownership and permissions
- chown vcf_vip:vcf /usr/local/vip/bin/start-vip.sh
- chmod 775 /usr/local/vip/bin/start-vip.sh
- Restart the service using below command:
- systemctl restart vip-manager-i18n.service
To apply the workaround for CVE-2021-45046 follow below steps
There is one jar that require mitigation in SDDC Manager - Follow the steps below to fix the same
- Log on to SDDC Manager VM and change the user to 'root' account using command shown below and provide the root user password to get super user privileges
- Stop the vip manager i18n service
- systemctl stop vip-manager-i18n.service
- Handling the vulnerability in vip-manager-i18n-common.jar
- Create a temporary directory with the command below
- mktemp -d -t log4j-XXXXXXXXXX
- Run the command below to list if log4j-core is present in the jar vip-manager-i18n-common.jar
- zipinfo -1 "/usr/local/vip/vip-manager-i18n-common.jar" | grep log4j-core-*
- This lists the presence of log4j-core-*.jar in the above jar which confirms the vulnerability
- Copy the temp directory path created as part of Step #3.1 and execute below commands
- unzip "/usr/local/vip/vip-manager-i18n-common.jar" "BOOT-INF/lib/log4j-core-2.13.3.jar" -d "/tmp/<temp-directory-name>"
- In this run - the temp folder gets created with name log4j-hUuV38E2Jc
- Example command:
- unzip "/usr/local/vip/vip-manager-i18n-common.jar" "BOOT-INF/lib/log4j-core-2.13.3.jar" -d "/tmp/log4j-hUuV38E2Jc"
- Similarly, replace the temporary folder names in below commands where applicable with created temp folder name from Step #3.1
- zipinfo -1 /tmp/log4j-hUuV38E2Jc/BOOT-INF/lib/log4j-core-2.13.3.jar | grep -q "org/apache/logging/log4j/core/lookup/JndiLookup.class"
- Run the command below
- echo $?
- If the returned value is 1 - No further action needed as the class is not present the move to Step#4 below to start the vip manager service
- If returned value is 0 - Means the JndiLookup.class is present and continue running commands below
- When the JndiLookup.class is present - run the command below
- zip -q -d "/tmp/log4j-hUuV38E2Jc/BOOT-INF/lib/log4j-core-2.13.3.jar" "org/apache/logging/log4j/core/lookup/JndiLookup.class"
- This deletes the JndiLookup.class file from the log4j-core*.jar
- Update the changed log4j-core to the vip-manager-i18n-common jar using command below
- cd "/tmp/log4j-hUuV38E2Jc/" && zip -ur -0 "/usr/local/vip/vip-manager-i18n-common.jar"
- Start the vip manager i18n service
- systemctl start vip-manager-i18n.service
SDDC Manager Workaround Automation Steps NOTE :
- This automated script addresses both CVE-2021-44228 and CVE-2021-45046
- It is recommended to run the script even in case any earlier workaround was applied for CVE-2021-44228
- This script is applicable for all VCF 3.x (except VCF 3.10.2, VCF 3.10.2.1 & VCF 3.10.2.2) and 4.x versions.
- If there has been an upgrade from any of the 3.10.1.x to 3.10.2.x, and if the remediation has not been applied prior to the upgrade, then please run the script to apply remediation even if you are on 3.10.2.x (post upgrade). The script is idempotent and can be repeatedly run: it will figure out that the JndiLookup.class has already been removed and do nothing in case remediation was already applied before.
- Copy the script file named "log4j_vip_v3.sh" attached in this KB article to your SDDC Manager via WinSCP under /home/vcf directory
- Log on to SDDC Manager using SSH
- Change the user to ‘root’ account using the command below
- Change the directory to location to /home/vcf
- Execute command below
Note: script is idempotent and can be repeatedly run: it will figure out that the
JndiLookup.class has already been removed and do nothing
NOTE:-
- The Knowledge Base Articles listed below has specific workarounds for specific versions for each of the BOM specific components.
- Make sure the Knowledge Base article is followed based on the version of Product running in the VCF environment
- Keep a tab on the below Knowledge Base articles in the related information section on "Change.log" for any new updates
BOM Software Component Product Name | Knowledge Base Article Link |
VMware vCenter Server Appliance | KB87081 |
VMware ESXi (Not Impacted) | NA |
VMware NSX-T Data Center | KB87086 |
VMware vRealize Suite Lifecycle Manager | VCF 4.x - KB87097 VCF 3.x - KB87200 |
VMware vRealize Log Insight | KB87089 |
vRealize Operations Manager | KB87076 |
vRealize Automation | vRA 7.6 - KB87121 vRA 8.x - KB87120 |
VMware Horizon 7 (*applicable for VCF 3.x version only) | KB87073 |
VMware Workspace ONE Access | KB87090 |
VMware Identity Manager | KB87093 |
NSX Data Center for vSphere (*applicable for VCF 3.x version only) | KB87099 |