CVE-2021-44228 & CVE-2021-45046 - log4j vulnerability and APM

Products

APM DX Application Performance Management DX SaaS CA Application Performance Management (APM / Wily / Introscope) CA Application Performance Management Agent (APM / Wily / Introscope) DX APM SaaS

Issue/Introduction

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228

Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints.

An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.

From log4j 2.15.0, this behavior has been disabled by default. In previous releases (>2.10) this behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to “true” or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false".

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046

It was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations. This could allows attackers with control over Thread Context Map (MDC) input data when the logging configuration uses a non-default Pattern Layout with either a Context Lookup (for example, $${ctx:loginId}) or a Thread Context Map pattern (%X, %mdc, or %MDC) to craft malicious input data using a JNDI Lookup pattern resulting in a denial of service (DOS) attack. Log4j 2.15.0 restricts JNDI LDAP lookups to localhost by default. Note that previous mitigations involving configuration such as to set the system property `log4j2.noFormatMsgLookup` to `true` do NOT mitigate this specific vulnerability. Log4j 2.16.0 fixes this issue by removing support for message lookup patterns and disabling JNDI functionality by default. This issue can be mitigated in prior releases (<2.16.0) by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class).

Environment

APM SaaS and APM On-Premise

APM 9.7, 10.0, 10.1, 10.2, 10.3, 10.5, 10.6, 10.7.x, 11.x and 20.x/21.x,22.x

Resolution

APM Engineering has confirmed that APM 9.7 and later servers (Collectors / MOMs / TESS / TIM / WebView / Workstation) and 9.7 (and later)/SaaS/21.x/22.x agents are not vulnerable to CVE-2021-44228

APM 9.7 and later agent versions are not vulnerable to CVE-2021-45046.

The team has also investigated and determined that a few components of the Digital Operational Intelligence contain the log4j versions that may be exposed to the vulnerability.

APM includes the version of Spring Boot that we use (2.5.6) and a link to Spring's website indicating that the Log4j2 jar present is just the API jar, whereas the vulnerability is in the Core jar, which is not included. Thus the vulnerable library is not included in our APM tool.
A scan can indicate only the presence of log4j-api, and the linked article provides evidence for the assertion that log4j-api does not contain the vulnerability, only log4j-core (which is not present) is affected.

https://spring.io/blog/2021/12/10/log4j2-vulnerability-and-spring-boot

We recommend the following steps to mitigate the exposure to this vulnerability:

For release 21.3.1

STEP#1: Add the environment variable “LOG4J_FORMAT_MSG_NO_LOOKUPS” with value “true” in doi-configmap, jarvis-configmap and axa-configmap as described below:

If you are using OpenShift:

- Go to your dxi project

- Click Resources > Config Maps from the left-navigation

- Select each of the config maps and click "Actions", then "Edit YAML"

- Add the below line under the "data" section:

LOG4J_FORMAT_MSG_NO_LOOKUPS: "true"

- Click on "Save"

If you are using Kubernetes

kubectl edit configmaps doi-configmap -n<namespace>
kubectl edit configmaps jarvis-configmap -n<namespace>
kubectl edit configmaps axa-configmap -n<namespace>

- Add the line:

LOG4J_FORMAT_MSG_NO_LOOKUPS: "true",' under "data"

Here is an example for Kubernetes

- Save the file

STEP#2: Restart all Pods:

cd <DX_PLATFORM-HOME>tools

./dx-admin.sh stop

Wait for all pods to stop

./dx-admin.sh start

VERIFICATION:

If OpenShift:

Go to Application > Pods
Open doireadserver
Click Terminal
Execute below command to verify new property has been added:
printenv | grep LOG4J_FORMAT_MSG_NO_LOOKUPS

If Kubernetes:

kubectl get pods -n<namespace> | grep doireadserver
kubectl exec -ti <doireadserver- podname> -n<namespace> sh
printenv | grep LOG4J_FORMAT_MSG_NO_LOOKUPS

NOTE:

Log Analytics is still vulnerable even with the above configuration changes until a hotfix is provided, for now you can scale down Logparser and logcollector as shown below:

kubectl scale --replicas=0 deployment doi-logcollector -n<namespace>
kubectl scale --replicas=0 deployment doi-logparser -n<namespace>

Any additional information will be updated in this KB article as available.

Should you have any further questions or concerns, please open a case with Support.