AAI and log4j vulnerabilities - CVE-2021-44228, CVE-2017-5645, and CVE-2021-45105

Products

• Automic Automation Intelligence

Issue/Introduction

Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. 

Automic Automation Intelligence does have some components that are using Apache log4j 2.x and are therefore affected by the zero-day Apache log4j vulnerability.  The list of confirmed components are:

AAI Server
JobTrac connector
Automic Connector
Control-M Connector
Connector Framework
ESP Connector

 

Below is how you can check the log4j versions on your AAI system.

 

On a Linux AAI server cd to /<Install dir>/jboss/standalone/ and run the command below

 

for i in */*/content; do echo $i; unzip -l $i | grep log4j; done

 

You will likely see files with 2.7 in the name like the files below and more.

 

WEB-INF/lib/log4j-slf4j-impl-2.7.jar

WEB-INF/lib/log4j-core-2.7.jar

WEB-INF/lib/log4j-api-2.7.jar

 

2.7 is vulnerable to some of the recent log4j vulnerabilities raised recently.

 

 

You can also go to /<Install dir>/jboss/standalone/deployments and run:

 

jar tvf customConditionGenerator*.war | grep log4j

jar tvf simulation-service-*.war | grep log4j

jar tvf subscription-service-*.war | grep log4j

jar tvf telemetry-service-*.war | grep log4j

 

 

**Note that you will also see some 1.2.x files in these commands, these are not affected by the vulnerabilities listed above if you are on Java version later then 1.8.121.

Resolution

The Broadcom AAI Engineering team has released a hotfix, 6.4.1 HF2, containing the latest log4j package.

• You can directly upgrade to this patched release directly from previous releases.
• Release notes for this version can be found here: https://docs.automic.com/documentation/webhelp/english/ALL/components/TERMA/latest/AAI%20Guides/Content/AAI_Topics/Release_Notes_6_4_1/Release_Notes_6_4_1.htm
• If upgrading from 6.4.1 to 6.4.1 HF2, there will be no database schema updates.
• If upgrading from an earlier release their may be schema updates, which the upgrade process will notify you about, and an updated upgrade*.sql file will be available in the /<Install Dir>/scripts folder post upgrade.
• The upgrade will replace 2.7 log4j files with 2.17.

 

You can download the specific Upgrade/Install files for your environment by going to: https://downloads.automic.com/downloads/advanced_mode

 

For the AAI Server, enter:

Component: Automic Automation Intelligence

Sub-Component: Installer and/or Upgrader

Version: 6.4.1 H2

 

 

**Post upgrade you may need to delete some left over 2.7 files as noted below:

After upgrading to 6.4.1-1 and before starting up the service, please delete everything under /<Install dir>/jboss/standalone/data/content.

 

For example:

cd /<Install dir>/jboss/standalone/data/content

rm -rf *

 

Then start up AAI.

 

This will remove any left over 2.7 log4j files.

 

You can verify the files are remove by running:

cd /<Install dir>/jboss/standalone/data/content

for i in */*/content; do echo $i; unzip -l $i | grep log4j; done

 

You should now see 2.17 files and no longer see any 2.7 files.

 

**Steps below are only if you cannot upgrade to the patched build as noted above and will help to mitigate the risk of CVE-2021-44228.**

If you cannot upgrade to 6.4.1 HF2 immediately, there are some things that can be done, these steps are not needed if upgrading to 6.4.1 HF2:

According to CVE-2021-44228, Java 1.8u121+ has built-in protection due to a default setting and protects against remote code execution by defaulting "com.sun.jndi.rmi.object.trustURLCodebase" and "com.sun.jndi.cosnaming.object.trustURLCodebase" to "false". Customers who are already using this Java version and didn't change the relevant properties should be fine.

 

 

For AAI Server

 

1:  Ensure that the java 8 version running is > u121.  If it is not, update to the most recent Java 8 version available.  Evidence can be found in the /jboss/standalone/log/jaws.log after a restart and will look as follows:

      2021-12-09 13:28:18,199 INFO  [JavaVersionUtil] Java Version: 1.8.0_292

 

2:  For the AAI Server, please update the following lines in this file:

/jboss/standalone/configuration/logging.properties

Change: formatter.SERVER-PATTERN.pattern=%d %-5p [%C] %m%n

to: formatter.SERVER-PATTERN.pattern=%d %-5p [%C] %m{nolookups}%n

Change: formatter.PATTERN.pattern=%d %-5p [%c{1}] %m%n
to: formatter.PATTERN.pattern=%d %-5p [%c{1}] %m{nolookups}%n

3:  Also, for the AAI Server, please update the following file:

/jboss/standalone/configuration/standalone.xml

Change this block:

<formatter name="PATTERN">

<pattern-formatter pattern="%d %-5p [%c{1}] %m%n"/>

</formatter>

<formatter name="SERVER-PATTERN">

<pattern-formatter pattern="%d %-5p [%C] %m%n"/>

</formatter>

to:

<formatter name="PATTERN">

<pattern-formatter pattern="%d %-5p [%c{1}] %m{nolookups}%n"/>

</formatter>

<formatter name="SERVER-PATTERN">

<pattern-formatter pattern="%d %-5p [%C] %m{nolookups}%n"/>

</formatter> 

4:  If present (not all supported versions will have this file), please update the following file:

/jboss/standalone/configuration/eem.log4j.xml

Change <param name='ConversionPattern'value='%5p %d{ISO8601} [%t] [%c] %m%n'/>

to: <param name='ConversionPattern'value='%5p %d{ISO8601} [%t] [%c] %m{nolookups}%n'/>  

 

Note: there are 4 instances in the file that require the change.

 

5:  Restart AAI Server

 

 

 

 

 

For customers running framework-based connectors for ESP, Automic and/or Control-M:

 

 

Patched version of the Connectors are available for download here: https://downloads.automic.com/downloads/advanced_mode

 

 

Automic Connector version 2.1.2

 

 

 

Control M Connector 1.2.3 HF1

 

To upgrade:

On Windows, run the *-connector.exe on Windows and select the option to upgrade the existing installation, when prompted enter the information from your existing <Install Dir>\application.yml

 

 

 

If you cannot immediately upgrade your Connector you can take the steps below:

 

1:  Locate the service files in both the installed directory and the system directory (there will be a file in two locations).  For ubuntu, by default, the locations are:

/opt/connector.{type}/connector.{type}.service

and

/etc/systemd/system/connector.{type}.service

 

Specifically for Control-M, the files (by default) are:

/etc/systemd/system/connector.control-m.service

/opt/connector.control-m/connector.control-m.service

 

In both files, replace:

ExecStart=/bin/bash -c "java -jar /opt/connector.control-m/connector.control-m.jar"

with:

 

ExecStart=/bin/bash -c "java -DformatMsgNoLookups=true -jar/opt/connector.control-m/connector.control-m.jar"

 

2. Reload the service definition by running:

systemctl daemon-reload

 

 

3.  Restart the connector service

 

Additional Information

Note, that after upgrading to 6.4.1 HF2, you may see some new warnings in server.log as mentioned here: https://knowledge.broadcom.com/external/article?articleId=230884

These can safely be ignored and will be addressed in a future release.

 

**Please note that this KB article may be updated as new information becomes available, please refresh this page to ensure you have the latest information**