Workaround instructions to address CVE-2021-44228, CVE-2021-45046 in vRealize Automation and vRealize Orchestrator 8.x

Products

VMware Aria Suite

Issue/Introduction

CVE-2021-44228 and CVE-2021-45046 have been determined to impact vRA and vRO from 8.0 to 8.6.1 via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

CVE-2021-44228, CVE-2021-45046 - VMSA-2021-0028

Symptoms:

CVE-2021-44228 has been determined to impact vRA and vRO from 8.0 to 8.6.1 via the Apache Log4j open source component it ships. This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

CVE-2021-44228 - VMSA-2021-0028 (link: https://www.vmware.com/security/advisories/VMSA-2021-0028.html)

Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors. 

We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of 8.6.2, as outlined by our software support policies. VMSA-2021-0028 will be updated when these releases are available. In the interim, we will be updating this Knowledge Base article with revised guidance to remove all JndiLookup classes per Apache Software Foundation guidance. Please subscribe to this article to be informed when updates are published.

The workarounds described in this document are meant to be a temporary solution only. Upgrades documented in the aforementioned advisory should be applied to remediate CVE-2021-44228 when available. Long-term resolution will be available in vRA and vRO versions 8.6.2 or later.

Environment

VMware vRealize Orchestrator 8.4.x
VMware vRealize Orchestrator 8.3.x
VMware vRealize Orchestrator 8.1.x
VMware vRealize Automation 8.5.x
VMware vRealize Automation 8.3.x
VMware vRealize Automation 8.4.x
VMware vRealize Orchestrator 8.6.x
VMware vRealize Automation 8.6.x
VMware vRealize Automation 8.1.x
VMware vRealize Orchestrator 8.2.x
VMware vRealize Orchestrator 8.5.x
VMware vRealize Automation 8.2.x

Resolution

The workarounds described in this document can be considered as permanent solution as they update log4j libraries in the VA to 2.17.0.

Future releases will include log4j 2.17.0 or later.

Workaround:

For each vRA and vRO deployments with versions from 8.1 to 8.6.1, execute the following procedure:

Prerequisites

Take simultaneous VM snapshots without memory of all nodes in the cluster.

Procedure

Upload <87120-kb-v2.tar.gz> and <87120-kb-v3-validate.tar.gz> under /root on all nodes.
Validate whether the system is vulnerable by running the command below on all nodes. Error reports related to log4j will show up for the affected artifacts that are vulnerable.

cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYzLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICI4MmY5MDVkYTU1ZDQ2YWJmMDk5YjQ5MTNkMGIyZjI1NjQ2ZWNlOGQ5ODhiMjJlMzYyZDliNzdiYzZkZDhmODU3IiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12My12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi8qOyAvdG1wLzg3MTIwLWtiLzg3MTIwLWtiLXYzLXZhbGlkYXRlLnNoOyBybSAtcmYgL3RtcC84NzEyMC1rYikgfHwgZWNobyAiRmlsZSBub3QgZm91bmQgODcxMjAta2ItdjMtdmFsaWRhdGUudGFyLmd6IG9yIGNoZWNrc3VtIG1pc21hdGNoIgo=" | bash -

3. If the system is vulnerable, install the KB by executing the following command on all nodes:

cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYyLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICJkNDdlYzc4ZWJmZjY1M2JjNTg3Y2I5NTMwOTkwNDhlMTBhMzc5OTUzZGFjZWZlMGY5ODMzODRiOTRiMDAyZTFiIiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12Mi50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi5zaDsgL3RtcC84NzEyMC1rYi84NzEyMC1rYi12Mi5zaDsgcm0gLWZyIC90bXAvODcxMjAta2IpIHx8IGVjaG8gIkZpbGUgbm90IGZvdW5kIDg3MTIwLWtiLXYyLnRhci5neiBvciBjaGVja3N1bSBtaXNtYXRjaCIK" | bash -

4. Make the installation effective by executing /opt/scripts/deploy.sh from the node that is typically used as primary (e.g. where /var/log/deploy.log file exists from previous runs). This is run only once across the vRA/vRO cluster nodes.

5. Verify the KB is active on the system by running the verification command below on all nodes. There should be no error reports related to log4j.

cd /root; base64 -d <<< "W1sgIiQoc2hhMjU2c3VtIDg3MTIwLWtiLXYzLXZhbGlkYXRlLnRhci5neiB8IGN1dCAtZiAxIC1kICcgJykiID09ICI4MmY5MDVkYTU1ZDQ2YWJmMDk5YjQ5MTNkMGIyZjI1NjQ2ZWNlOGQ5ODhiMjJlMzYyZDliNzdiYzZkZDhmODU3IiBdXSAmJiAocm0gLXJmIC90bXAvODcxMjAta2I7IG1rZGlyIC1wIC90bXAvODcxMjAta2IvOyB0YXIgLXh2ZiA4NzEyMC1rYi12My12YWxpZGF0ZS50YXIuZ3ogLUMgL3RtcC84NzEyMC1rYjsgY2htb2QgK3ggL3RtcC84NzEyMC1rYi8qOyAvdG1wLzg3MTIwLWtiLzg3MTIwLWtiLXYzLXZhbGlkYXRlLnNoOyBybSAtcmYgL3RtcC84NzEyMC1rYikgfHwgZWNobyAiRmlsZSBub3QgZm91bmQgODcxMjAta2ItdjMtdmFsaWRhdGUudGFyLmd6IG9yIGNoZWNrc3VtIG1pc21hdGNoIgo=" | bash -

Note: Re-apply this KB, If you have previously applied the KB originally released on 12/13/2021, 12/14/2021. If not re-applied, upgrading to any further release of vRA and vRO will not be possible.
Note: The change is persistent over upgrades and the KB should not be re-applied.

Additional Information

Changelog:
December 13th 2021 - 13:11 MST: Drafted initial document with initial workaround.
December 13th 2021 - 14:30 MST: Modified horizon-service restart due to a backslash incorrectly placed.
December 15th 2021 - 10:42 MST: Modified purpose to include new updates from Apache.
December 20th 2021 - 14:59 MST: Added support for replacing log4j jar files to 2.17.0
December 21st 2021 - 09:45 MST: Added support for patching log4j in vco-cli-java.jar
December 22nd 2021 - 09:12 MST: Added special handling for JARs of type cafe-sdk; Addressed disk space issue errors during validation of vRO directories.

Impact/Risks:

Please note the following prior to executing the workaround procedure:

vRA and vRO versions 8.0, 8.0.1 and 8.1 are no longer supported at the time of this article's publication.
Re-apply this KB, If you have previously applied the workaround released prior to 12/20/2021. If not re-applied, upgrading to a future release will not be possible.
This change is persistent over upgrades and the KB should not be re-applied.
This change applies to vRA and vRO (both standalone and embedded).
Can be applied to all vRA and vRO deployments versions 8.1 through 8.6.1.
For clustered setups, in vRO Control Center/Cluster management page, a warning message "Local changes detected" may appear after applying this KB. This message should be disregarded.

Note: Automated vulnerability scanners may report that vRA/vRO products are still vulnerable to CVE-2021-44228 and CVE-2021-45046 after this KB article has been applied. These findings can be safely ignored.

Attachments

87120-kb-v3-validate.tar get_app

87120-kb-v2.tar get_app