CVE-2021-44228: Automic Automation and log4j vulnerability

Products

• CA Automic Workload Automation - Automation Engine
• CA Automic One Automation
• CA Continuous Delivery Automation - Automation Engine

Issue/Introduction

UPDATE 20 DECEMBER: The mitigation steps provided in this article also apply for the newly created CVE-2021-45105

UPDATE 16 DECEMBER: Many packages, Automic Infrastructure Manager, and the RA SOAP and RA JDE agents have new releases to fully patch this issue
UPDATE 15 DECEMBER: We are aware of the update from Apache that the vulnerability was not fully patched with Log4j2 2.15 and will be with 2.16 (CVE-2021-45046).  This does not change any of the mitigation steps or impacted components in this article.  
UPDATE 15 DECEMBER: Automic Infrastructure Manager added to list of affected components

Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. 

There are some Action packs and RA Solutions for Automic Automation (AWA) and Continuous Delivery Automation (ARA) that are affected by the zero-day Apache log4j vulnerability.

The following Action Packs depend on Apache log4j 2 <= 2.10

    Package.DM -> Resolved in PCK.AUTOMIC_DM 1.4.4 (released 16 December)
    Package.IIS -> Resolved in PCK.AUTOMIC_IIS 1.4.2 (released 16 December)
    Package.Tomcat -> Resolved in PCK.AUTOMIC_TOMCAT 1.5.3 (released 16 December)
    Package.Weblogic -> Resolved in PCK.AUTOMIC_WEBLOGIC 1.4.3 (released 17 December)
    Package.Websphere -> Resolved in PCK.AUTOMIC_IBM_WEBSPHERE 1.8.3 (released 16 December)

The following Action Packs, RA Solutions and the AAI Connector depend on log4j > 2.10

    Package.Bond -> Resolved in PCK.AUTOMIC_BOND 1.3.3 (for versions 12.2 and 12.3) and 11.0.1 (for versions 21.x) (released 16 December)
    Package.Atlassia.Jira -> Resolved in PCK.AUTOMIC_ATLASSIAN_JIRA 1.0.2 (released 16 December)
    Package.Jenkins -> Resolved in PCK.AUTOMIC_JENKINS 1.2.0 (released 16 December)
    Package.EXCEL -> Resolved in PCK.AUTOMIC_EXCEL 1.1.1 (released 16 December)
    Package.JDBC -> Resolved in PCK.AUTOMIC_JDBC 1.4.2 (released 16 December)
    Package.Siebel -> released 17 December
    RA.Banner -> Resolved in RA Banner 4.1.1 (released 20 December)
    RA.JD.Edwards -> Resolved with RA JD Edwards 2.2.1 (Released 16 December)
    RA.Web.Service.SOAP -> Resolved with RA Web Service SOAP 4.6.1 (Released 16 December)
    AAI.Automic (Connector) -> Resolved with AAI Automic Connector 2.1.2 (Released 17 December)
    Automic Infrastructure Manager -> Resolved with Infrastructure Manager 2.0.1 (Released 16 December)

Are the JWPs and JCPs impacted by this vulnerability?

JWPs and JCPs are not impacted as they are using a version of log4j that is not impacted by this issue

Is AWI impacted by this vulnerability?

AWI is not impacted as it is not using the log4j-core components that are affected: https://logging.apache.org/log4j/2.x/security.html

How to tell if my system is impacted

Are there packages installed that are impacted?

1. Login to the AWI as a user with Administration perspective Privileges - please note that the installed packages may be different in different clients, so all clients should be checked
2. Go to the Administration Perspective and look for the "Packs" area on the left hand side list of Administration areas - if this does not exist, there are no Packages to be impacted
3. If the Packs area does exist, look for one of the affected Packages above that has a version in the "Installed Version" column - if there are no versions listed, there are no Packages to be impacted
4. If the Package is installed, the "Installed Version" column will have a version and your system could be impacted
   

Are any additional marketplace packages not on this list that are impacted?

There may be packages available on marketplace.automic.com that are not supported by Broadcom - if they're community or partner supported, you will need to reach out to the community or author of the package directly to fully mitigate log4j 2.x vulnerabilities.  These will have something like this under "Supported by"

• • 
  • 

Are there RA agent installed that are impacted?

1. Login to the AWI as a user with Administration perspective Privileges
2. Go to the Administration Perspective and click on the "Agents & Group" sub-item in the list of the left hand side.
3. Sort the list by Software and look for the affected RA agents listed above
   
4. If none exist, your system cannot be impacted by the vulnerabilities in the RA Agent solutions; if they do exist, your system may be impacted

Are any 3rd party components impacted?

There are sure to be 3rd party components that are impacted as the use of log4j2 is wide-spread.  This article has a good breakdown of impacted known applications: https://github.com/NCSC-NL/log4shell/blob/main/software/README.md

It's important to note that Broadcom Support cannot provide an explanation on how to mitigate issues with third party applications.  Those specific vendors will need to be contacted to provide information on mitigation or fixes.

 

AE/RA Components depending on log4j version 1.x :

This vulnerability can only be exploited under very specific circumstances in log4j. Precondition for that is that the log4j components integrates the JMS-listener .

None of our products using the log4J libraries are actually using the JMS-listener. So exploiting this vulnerability with applications from the AE solution depending on log4j 1.x is not possible at this time.

Coming releases of components depending on log4j 1.x will come with an updated log4j module version 2.16 or 2.17 or 2.17.1 if needed. 

"If needed" stands here for the different reasons that might require a log4j update. This could be either an end of support, important technical changes in the component architecture or the detection of an new vulnerability in this component.

To obtain more informations about the handling of log4J 1 in Broadcom Solutions open this article : https://knowledge.broadcom.com/external/article?articleId=233667

Are there any other components impacted by this vulnerability?

There are no other components that we're aware of that are impacted by this vulnerability after numerous scans.  If your security team finds a component they believe to be impacted, please open a case with support pointing out the file that is impacted, the specific version and service pack for that component, and the specific CVE that applies to that component.

Resolution

The Broadcom Automic Automation Engineering team plans on releasing hotfixes containing the latest log4j package by the end of the week of December 13. 

Updating to fixed patches for Packages

The following packages can be updated using the "Packs" section in the Administration perspective in the AWI OR installed from file:

Package.Atlassia.Jira -> Can also be downloaded here
Package.Bond -> Can also be downloaded here
Package.DM -> Can also be downloaded here
Package.IIS -> Can also be downloaded here
Package.JDBC -> Can also be downloaded here
Package.Jenkins -> Can also be downloaded here
Package.Tomcat -> Can also be downloaded here
Package.Weblogic -> can also be downloaded here

The following packages can only be updated using the "Install From File" option:

Package.EXCEL -> Can also be downloaded here
Package.Websphere -> Can also be downloaded here

The following package can only be downloaded by visiting the MarketPlace and requesting via the Contact Us button:

Package.Siebel -> Visit https://marketplace.automic.com/marketplace/browse/packagesiebel and click on the "Contact Us" button

For packages that can be updated via the Packs section of the Administration perspective (please note packages may be different in each client and so these steps may need to be taken in all clients).  Full instructions can be found in Working with Packs and Plug-ins

1. Open the Administration perspective
2. Go to Packs
   
3. Click on the "Update Index" button at the top
   
   Notice that the Latest Version column has been updated compared to the Installed Version
   
4. Select the Pack needing to be udpated, right-click and choose "Upgrade"
   
5. Confirm you'd like to upgrade by clicking "Upgrade"
6. Now notice that the Installed Version and Latest Version updated

Packages can also be updated by downloading them from the Automic marketplace and updated with the following:

1. Download the .zip file from the marketplace
2. Open the Administration perspective
3. Go to Packs
   
4. Click on the "Install from File" button at the top
   
5. Confirm the Installation
6. Now notice that the Installed Version and Latest Version updated

Updating to fixed patches for RA Agents

The following updated RA Agents can be downloaded:

RA JD Edwards 2.2.1 can be downloaded here
RA Web Service SOAP 4.6.1 can be downloaded here

RA Banner 4.1.1 can be downloaded here

Follow the steps to load the respective jar file found in each RA Agent's respective documentation, only the solution needs to be updated, not the RA Core unless the current RA Core installation is incompatible due to being an older version.

Automic Infrastructure Manager

The patched version for Automic Infrastructure Manager can be found here.  Follow the installation instructions for Automic Infrastructure Manager

AAI Automic Connector

For instructions and Guidance for the AAI Automic Connector, please take a look at the article covering the vulnerability with AAI components here.

Workaround/Mitigation

Until the patches are all released, or if the patch is not feasible for the environment (too old, incompatible, etc...), there are some mitigation steps that can be taken:

According to CVE-2021-44228, Java 1.8u121+ has built-in protection due to a default setting and protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false". Customers who are already using this Java version and didn't change the relevant properties should be fine.

For affected Action Packs, RA Solutions and the AAI Connector depending on log4j > 2.10, the vulnerable behavior can be mitigated by setting system property "log4j2.formatMsgNoLookups" to TRUE.  This can be done by altering the startup parameter to include:

-Dlog4j2.formatMsgNoLookups=true

Mitigating the risk with RA Agents

For example, if the service manager command being used to start an RA agent looks like this:

java -Xrs -Xmx1G -jar C:\automic\12.2.9\Agents\RA_AGENT\bin\ucxjcitx.jar

It can be updated to this:

java -Xrs -Xmx1G -Dlog4j2.formatMsgNoLookups=true -jar C:\automic\12.2.9\Agents\RA_AGENT\bin\ucxjcitx.jar

Here's a breakdown of where to find this:

1. Open the servicemanager dialog and put in the Computer Name (and port if applicable) for the server where the RA Agent is running
2. Right-click the agent and choose Properties
   
3. Check the command being used to see if the -D log4j2.formatMsgNoLookups=true is set in the Command textbox:
   
4. Update the Command to include the flag and click OK
   
5. Stop and restart the agent

As of 15 December, another step has been suggested to mitigate even more of the risk.  It is recommended that the JndiLookup.class be removed from jar files of the affected components.  The following steps can be taken for the Banner and JDE agents:

1. If you do not have the [agent name]_solution.jar file that was used to load the solution currently in your system, go to downloads.automic.com and download the solution again.  
   The base links for each agent are:
   RA JD Edwards agent - you will need to choose the solution version from the dropdown
   RA Banner agent - you will need to choose the Sub-Component and Version needed from the dropdowns
2. Download the RA agent .zip file if applicable
3. Unzip the downloaded .zip file and open the jar file with a zip tool (e.g. 7zip)
   Banner agent file is BannerAgent_solution.jar
   JDE agent file is JDEdwardsAgent_solution.jar
4. Navigate into the deploy file
   Banner agent file is BannerAgent_deploy_file.jar
   JDE agent file is JDEdwardsAgent_deploy_file.jar
5. Navigate in the deploy file to the the folder /lib/log4j-core-2.{used verion}.jar/org/apache/logging/log4j/core/lookup
6. Delete the JndiLookup.class
7. Update and close the opened archive (jar file) - you may need to look up documentation for the zip tool you're using to find out how to update the opened archive
8. Install the RA agent again using the modified jar file by loading it into the database
   Banner agent can be loaded following the documentation for Loading the RA Banner .jar File into the Database
   JDE agent can be loaded following the documentation for Loading the .jar File into the Database
9. Restart the agent

Mitigating Packages that might be impacted

The best and easiest way to be sure there is no risk to the integrity of running the installed Packages is to wait for new packages to be released within the week of December 13 and update the packages then.  As a workaround, you could possibly do the following:

1. Log into the AWI where the client where the Package is installed is
2. Go the "Search" textbox in the upper right and choose "Advanced Search"
   
3. Click on the "Add filter criteria" button and choose "Object contents"
   
4. Under Object contents, be sure that just "Process" is checked and put "java -jar" into the Object contents search
   
5. Click the "Filter" button on the bottom
6. Anything listed in one of the affected package folder can be updated by opening it, looking for the java -jar call and updating
   java -jar
   to
   "java -DformatMsgNoLookups=true -jar 
   
   

Can the system-wide environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS be used?

Some articles are pointing to being able to set an environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true - while this may be able to be used, the best is to update the commands as seen in the mitigation steps above

Additional Information

Although there is no known vulnerability, the analytics-backend was updated with 12.3.8 to use log4j 2.17.1 (from 1.2.16).