Workaround instructions to address CVE-2021-44228 and CVE-2021-45046: Apache Log4j for Tanzu Greenplum
Article ID: 296766
Updated On:
Products
Issue/Introduction
CVE-2021-44228 and CVE-2021-45046 does not impact core Tanzu Greenplum (all versions) functionality but it does impact the following add-on products:
- Platform Extension Framework (PXF)
- Tanzu Greenplum Text (GPText)
NOTE: If you are using Tanzu Greenplum and you are not using any of the above two components, then you are not impacted.
UPDATE: If you had previously applied mitigation for CVE-2021-44228, there are additional steps are REQUIRED to mitigate the threat identified by CVE-2021-45046. If you had not previously applied the mitigation for CVE-2021-44228, then you ONLY need to follow the steps for CVE-2021-45046 to address BOTH issues.
Here is this specific impact of new CVE-2021-45046
This could allows attackers with control over Thread Context Map (MDC) input data when the logging configuration uses a non-default Pattern Layout with either a Context Lookup (for example, $${ctx:loginId}) or a Thread Context Map pattern (%X, %mdc, or %MDC) to craft malicious input data using a JNDI Lookup pattern resulting in a denial of service (DOS) attack. Log4j 2.15.0 restricts JNDI LDAP lookups to localhost by default. Note that previous mitigations involving configuration such as to set the system property `log4j2.noFormatMsgLookup` to `true` do NOT mitigate this specific vulnerability.
Environment
Resolution
UPDATE: VMware has released PXF 6.2.1 and GPText 3.8.1 which are available in the latest VMware Tanzu Greenplum releases (5.29.2 & 6.19.0). These two versions update the Apache Log4J component to 2.16.0 which will address both CVE covered in this article.
If you are not able to upgrade to PXF 6.2.1 or GPText 3.8.1, please use the following workaround to mitigate your risk to this vulnerability.
IMPORTANT: Be careful of the hyphen (“-” 0x2d) in “-Dlog4j2.formatMsgNoLookups=true” when you copy and paste. We have noticed due to an unknown issue, it becomes a 0x90, which is rendered like a hyphen and causes startup problems. It's best to manually type in the "-" after copying it from this article.
PXF 5.x: Ships with log4j-1.2.17.jar
Log4J v1 is not impacted by the CVE so no action is needed. However, If you still want to mitigate your risk, upgrade to PXF 6.2.0 and then apply mitigation measures for PXF 6.
PXF 6.x: ships with log4j-core-2.14.1.jar
Immediate Mitigation
Set the system property log4j2.formatMsgNoLookups to true. Follow these steps:
1. Login to the Greenplum master host.
2. Edit $PXF_BASE/conf/pxf-env. [sh]
3. Uncomment and add –Dlog4j2.formatMsgNoLookups=true to PXF_JVM_OPTS property.
For example:
export PXF_JVM_OPTS="-Dlog4j2.formatMsgNoLookups=true"
If you have already specified a custom PXF_JVM_OPTS, append to the existing options.
For example:
export PXF_JVM_OPTS="-Xmx8g -Xms4g -Dlog4j2.formatMsgNoLookups=true"
4. Run the pxf cluster sync command to propagate the change across the Greenplum cluster.
5. Run the pxf cluster restart command to restart the PXF server across the cluster.
UPDATE: PXF 6.x Mitigation steps for CVE-2021-45046
PXF is using %X pattern in its Log4J configuration files, so it is impacted by this CVE and the only mitigation measure available to customers is to remove the JndiLookup class from the log4j-core JAR.
Procedure to remove JndiLookup class from PXF:
1. Login to the machine where Greenplum master where PXF 6.x is installed
2. Stop PXF server across the cluster: pxf cluster stop
3. Navigate to the directory where PXF 6.x application JAR file is installed. For example, assuming PXF6 was installed from the RPM into the default location:
GPDB 5: cd /usr/local/pxf-gp5/application GPDB 6: cd /usr/local/pxf-gp6/application
4. Make a backup copy of the existing PXF application JAR file:
cp pxf-app-*.jar pxf-app.bak
5. Extract the log4j-core JAR file from the PXF application JAR file:
unzip pxf-app-*.jar "BOOT-INF/lib/log4j-core-*.jar"
6. List the newly extracted file, notice its size
ls -l BOOT-INF/lib/log4j-core-*.jar Example: 1714164 Nov 18 2020 BOOT-INF/lib/log4j-core-2.13.3.jar
7. Remove the JndiLookup class from the log4j-core JAR file:
zip -q -d BOOT-INF/lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
8. List the log4j-core JAR file again, verify its size has decreased
ls -l BOOT-INF/lib/log4j-core-*.jar Example output: 1694512 Dec 14 23:26 BOOT-INF/lib/log4j-core-2.13.3.jar
9. Replace the updated log4j-core JAR file into the PXF application JAR file:
zip -q -n .jar -u pxf-app-*.jar BOOT-INF/lib/log4j-core-*.jar
10. Verify the updated JAR file with the updated size was placed into the PXF application JAR file:
unzip -l pxf-app-*.jar BOOT-INF/lib/log4j-core-*.jar Example output: Length Date Time Name --------- ---------- ----- ---- 1694512 12-14-2021 23:26 BOOT-INF/lib/log4j-core-2.13.3.jar --------- ------- 1694512 1 file
11. Remove the extracted intermediate log4j-core JAR file and its parent directories:
rm -rf BOOT-INF
12. Discard or archive PXF JAR backup file
rm pxf-app.bak
13. Repeat this procedure on ALL PXF machines. Alternatively, you can use gpscp command to replace pxf-app-*.jar file with the updated JAR file produced in the steps above:
source ${GPHOME}/greenplum_path.sh
gpscp -f ~/all-hosts -v pxf-app-*.jar =:/usr/local/pxf-gp6/application/
14. Verify the file has been updated on all hosts:
gpssh -f ~/all-hosts 'ls -al /usr/local/pxf-gp6/application/pxf-app-*.jar'
15. Start PXF server across the cluster: pxf cluster start
GPText 3.3 and below: ships with log4j-1.2.17.jar
Log4J v1 is not impacted by the CVE so no action is needed.
GPText 3.4 - 3.8: ships with log4j-core-2.11.0.jar (Apache Solr 7.4)
Immediate Mitigation
Use the method mentioned in CVE of setting a system property log4j2.formatMsgNoLookups to true. Follow these steps:
1. Login to the Greenplum master host.
2. Find out all Solr node’s hosts and directories using, cat $MASTER_DATA_DIRECTORY/gptext.conf.
bash$ cat $MASTER_DATA_DIRECTORY/gptext.conf id,host,port,solrdir,zoocluster 1,host0,18983,/data/primary/solr0,"mdw:2188,sdw1:2189,sdw1:2190/gptext" 2,host1,18984,/data/primary/solr1,"sdw2:2188,sdw0:2189,sdw2:2190/gptext"
3. Modify solr.in.[sh] on all Solr nodes listed in the gptext.conf to append -Dlog4j2.formatMsgNoLookups=True to SOLR_OPTS:
# Before change: SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=20000000" # After change: SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=20000000 -Dlog4j2.formatMsgNoLookups=True"
4. Restart gptext with this command:
bash$ gptext-start ... It might be possible that GPText is already running. Please gptext-stop to stop GPText. Or you want to restart process? (y/n): y
NOTE: If you have many Solr nodes, it can be a very tedious process to update all of them manually. You can use the script (gptext_solr_patch.sh) to automate the patch process. You can follow the steps below on master node:
1. source "<gptext_installation_dir>/greenplum-text_path.[sh]"
2. export MASTER_DATA_DIRECTORY=<master_data_direcotry>
3. bash gptext_solr_patch.[sh]
4. gptext-start
UPDATE: GPText 3.4.0 - 3.8.0 Mitigation steps for CVE-2021-45046
GPText is using %X pattern in its Log4J configuration files, so it is impacted by this CVE and the only mitigation measure available to customers is to remove the JndiLookup class from the log4j-core JAR.
A script (gptext_patch_45056.[sh]) has been created, you can mitigate the CVE by executing this script. Or follow below steps:
Procedure to remove JndiLookup class from GPText
1. Login to the machine where Greenplum master and GPText 3.4.x - 3.8.0 are installed.
2. Stop the GPText cluster with this command:
gptext-stop
3. Get the absolute path of $GPTXTHOME with echo $GPTXTHOME.
bash$ echo $GPTXTHOME
/usr/local/greenplum-text-3.x.x
4. Find out all Solr node’s hosts with cat $MASTER_DATA_DIRECTORY/gptext.conf.
bash$ cat $MASTER_DATA_DIRECTORY/gptext.conf
id,host,port,solrdir,zoocluster
1,host0,18983,/data/primary/solr0,"mdw:2188,sdw1:2189,sdw1:2190/gptext"
2,host1,18984,/data/primary/solr1,"sdw2:2188, sdw0:2189,sdw2:2190/gptext"
5. Backup the log4j-core.jar on the master:
bash$ cp $GPTXTHOME/greenplum-solr/contrib/prometheus-exporter/lib/log4j-core-2.11.0.jar $GPTXTHOME/greenplum-solr/contrib/prometheus-exporter/lib/log4j-core-2.11.0.jar.bak
bash$ cp $GPTXTHOME/greenplum-solr/server/lib/ext/log4j-core-2.11.0.jar $GPTXTHOME/greenplum-solr/server/lib/ext/log4j-core-2.11.0.jar.bak
6. Remove JndiLookup.class from log4j-core.jar in GPTXHOME.
bash$ zip -q -d $GPTXTHOME/greenplum-solr/server/lib/ext/log4j-core-2.11.0.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
Bash$ zip -q -d $GPTXTHOME/greenplum-solr/contrib/prometheus-exporter/lib/log4j-core-2.11.0.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
7. Copy the patched log4j-core.jar to all Solr nodes, remember to append all unique hosts found in step 4 with –h in the gpscp command line:
bash$ gpscp -h host0 –h host1 "$GPTXTHOME/greenplum-solr/server/lib/ext/log4j-core-2.11.0.jar" =:"${GPTXTHOME}/../greenplum-solr/server/lib/ext/log4j-core-2.11.0.jar"
bash$ gpscp -h host0 –h host1 "$GPTXTHOME/greenplum-solr/contrib/prometheus-exporter/lib/log4j-core-2.11.0.jar " =:"${GPTXTHOME}/../greenplum-solr/contrib/prometheus-exporter/lib/log4j-core-2.11.0.jar"
8. Restart gptext:
bash$ gptext-start
...
It might be possible that GPText is already running.
Please gptext-stop to stop GPText.
Or you want to restart process? (y/n): y
Attachments
Feedback
