CVE-2021-44228 - log4j vulnerability and Rally

Products

• Rally On-Premise
• Rally SaaS

Issue/Introduction

A critical vulnerability within the Apache Log4j package (https://nvd.nist.gov/vuln/detail/CVE-2021-44228) discovered on 12/09/2021.  This vulnerability affects all versions of log4j from 2.0-beta9 to 2.14.1

• Rally SaaS and Rally On-Prem are not affected by this vulnerability as Rally Software does not use the affected versions of log4j so there is no reason to take any remediation steps.
• Rally Jira Adapter is using log4j2. After the initial risk assessment performed by Rally Security this issue would not be exploitable from the public network. The adapter is installed behind customer firewalls and does not have any endpoints exposed. The resolution section of this article will further explain the necessary mitigation steps. 
• The information for the LAC integration for Jira is here:  https://support.broadcom.com/security-advisory/content/security-advisories/Symantec-Security-Advisory-for-Log4j-2-CVE-2021-44228-Vulnerability/SYMSA19793

Environment

Rally SaaS

Rally On-premises

Rally Adapter for Jira

LAC

Cause

CVE-2021-44228

Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled

This vulnerability affects all versions of log4j from 2.0-beta9 to 2.14.1

Resolution

Rally SaaS and Rally onPrem are not affected by this vulnerability as Rally Software does not use the affected versions of log4j so there is no reason to take any remediation steps.

Rally Adapter for Jira is leveraging LAC’s External Logging and hence might be susceptible to this vulnerability. Below is the listed set of steps to mitigate this vulnerability based on each version of the Adapter.

 

 

Rally Adapter for Jira 3.4 and higher (To Be Released on 17/12/2021)

This vulnerability is already mitigated through the removal of the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class. No action from customers is needed.

Rally Adapter for Jira 3.3 and lower

Updating to the latest version of Rally Adapter for Jira (version 3.4) is recommended for customers running version 3.3 or lower to properly remediate this vulnerability. 

If an upgrade is not possible, this vulnerability can be mitigated by adding the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS" and setting it to "true" by following either of the two options listed below.

Option 1: Via kots

1. In a kubernetes client that has the kots plugin installed use `kubectl kots download --namespace <KUBERNETES_NAMESPACE> --slug rally-jira-integration-kots` to download the deployment configurations to your filesystem.
2. Modify the file "upstream/adapter.yaml" by adding the environment variable "LOG4J_FORMAT_MSG_NO_LOOKUPS" in the `adapter` section and setting it to "true" like this:

      ``` 

      - name: LOG4J_FORMAT_MSG_NO_LOOKUPS

        value: "true"

      ```

1. Save the file and reupload it to kots via `kubectl kots upload --namespace <KUBERNETES_NAMESPACE> --slug rally-jira-integration-kots ./rally-jira-integration-kots`
2. In the admin console user interface in the version history a new version marked with "KOTS Upload" will be available, this reflects the modification which can be verified via `files changed +/- `
3. Deploy the "KOTS Upload" version.  

      

Alternatively the the kots admin console user interface can be used to generate the download/upload command via the tab "View Files" via the panel `Need to edit these files? Click here to learn how` on top of that screen. 

Option 2: Adapting the deployment

1.  In a kubernetes client use `kubectl edit deployment rji-adapter --namespace <KUBERNETES_NAMESPACE>` 
2. Add the environment variable "LOG4J_FORMAT_MSG_NO_LOOKUPS" in the `adapter` section and setting it to "true" like this:

      ``` 

      - name: LOG4J_FORMAT_MSG_NO_LOOKUPS

        value: "true"

      ```

1. When writing this change the pod should automatically restart.

 

Both Options will be persistent and work regardless of future changes in the configuration or any kind of deployment/redeployment

Note: Beware that yaml insists on 2 spaces as indentation so do not use hard tabs.

Firewall hardening guidelines

Given the fact that Adapter is installed behind customer firewalls and does not have any endpoints exposed - Following firewall hardening recommendations listed below would be required to mitigate risk of exploiting this vulnerability from a public network.

1. Block unnecessary outbound connections (this is something that should already be in place as a general security practice)
1. Add an Access Control rule to block outbound connections from your DMZ hosts on the LDAP port – 389/tcp.
2. Enable logging for this rule and monitor for any attempts by your servers to connect to an external system.
2. Enable outbound connection logging for your DMZ hosts
3. Create a Rule that triggers for any 389/TCP connections initiated from DMZ hosts to anywhere

If you have further questions or need assistance with the mitigation steps, please contact our support team or your Rally Solution Engineer.