Cumulative Update for vRealize Orchestrator 7.6

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

This article contains a list of known issues that are resolved in the latest patch available for vRealize Orchestrator (vRO) 7.6
Current Builds:
External: 19585838
Embedded: 19345110
Note: The *noarch.rpm file for embedded installations will maintain an offset build #.

See the attached spreadsheet titled:  vRO76-Patch19585838-ResolvedList.xlsx under the Attachment(s) section for a full list of all resolved issues and their associated Symptom(s).

Environment

VMware vRealize Orchestrator 7.6.x

Resolution

Prerequisites

The cumulative update process will rollback the below files to default values.  Ensure to backup these files in case custom Java heap or other similar settings have been applied to the environment.  Commands are provided below to copy the files into the /tmp/cupdate directory.

mkdir /tmp/cupdate
cp -p /etc/vco/app-server/js-io-rights.conf /tmp/cupdate/js-io-rights.conf
cp -p /etc/vco/app-server/vmo.properties /tmp/cupdate/vmo.properties
cp -p /etc/vco/app-server/server.xml /tmp/cupdate/server.xml
cp -p /etc/vco/app-server/sso.properties /tmp/cupdate/sso.properties
cp -p /etc/vco/app-server/cafe.properties /tmp/cupdate/cafe.properties
cp -p /etc/vco/app-server/ldap.properties /tmp/cupdate/ldap.properties
cp -p /etc/vco/app-server/log4j.xml /tmp/cupdate/log4j.xml
cp -p /usr/lib/vco/app-server/bin/setenv.sh /tmp/cupdate/setenv.shAppServer
cp -p /etc/vco/configuration/local.properties /tmp/cupdate/local.properties
cp -p /etc/vco/configuration/controlcenter.properties /tmp/cupdate/controlcenter.properties
cp -p /usr/lib/vco/configuration/bin/setenv.sh /tmp/cupdate/setenv.shConfig
cp -p /etc/logrotate.d/vco.lr /tmp/cupdate/vco.lr 
cp -p /etc/vco/configuration/server.xml /tmp/cupdate/server.xml

 

 

To resolve the listed issues you will want to install the latest vRealize Orchestrator 7.6 patch with the instructions below.

Embedded vRealize Orchestrator Update Procedure

See VMware Customer Connect under Product Patches for further information.

SHA256SUM

F9498BE2ED2E5E9836FB68C7E905121DE710B4FD17F4A71BABE2A3F8758763B4

Standalone and Clustered

1. Create a backup of the appliance(s) (Back Up Your Existing vRealize Automation 7.6 Environment).
2. Download the attached .rpm package named "vco-7.6.0.1664175350-20526662.noarch.rpm".
3. Transfer the .rpm package to each vRealize Automation appliance(s).
4. SSH Login to the primary vRealize Automation appliance.
5. Using root user, run the following command to install the upgrade: (using the proper path and name of the .rpm package previously downloaded.)

   rpm -U vco-7.X.X.XXXXX-XXXXXX.noarch.rpm

6. Execute the rpm -U vco-7.X.X.XXXXX-XXXXXX.noarch.rpm command on any remaining vRealize Automation appliances hosting embedded vRealize Orchestrator services.

Note: The rpm -U command will stop vRealize Orchestrator services before upgrading appliance binaries.  After the .rpm package upgrade completes, services will be started.

7. Restore any files with custom settings from the previously backed up files within /tmp/cupdate to their original location.  Review the Prerequisites section above as reference.

Note:  Commonly updated files with custom settings include those from Secure Hardening Guide (server.xml) and Increasing the memory allocated to a vRealize Orchestrator instance (setenv.sh).

8. Monitor the Services tab within the vRA appliance management interface for vco, advanced-service-designer and o11n-gateway to register.

Note:  Instances still experiencing issues may reference Resetting the Embedded vRealize Orchestrator 7.x configuration on a vRealize Automation 7.x appliance

External vRealize Orchestrator Update Procedure

See VMware Customer Connect under Product Patches for further information.

SHA256SUM

F0A3EF9C3425D7F7A7009DC309D3CE5E83B45D60D42EEF1D328A16C1C563F710

Standalone and Clustered

1. Create a backup of the vRealize Orchestrator appliance(s) (Back Up Your Existing vRealize Automation 7.6 Environment).
2. Download the attached .iso package named "VMware-vRO-Appliance-7.6.0.842-20526659-updaterepo.iso"
3. Stop vco-server and vco-configurator on all participating cluster member(s)

service vco-server stop
service vco-configurator stop

4. In case of a 3 node cluster deployment and SYNCRONOUS replication mode, switch to ASYNC mode before starting the upgrade.
5. Log into the first vRO appliance management interface page (https://FQDN:5480).
6. Navigate to Update > Settings.
7. Change the Update Repository to Use CDROM Updates and Save Settings.
8. Navigate to Update > Status.
9. Click on Check Updates.
10. When the update is shown, click on Install Updates.
11. Once complete, perform steps 4 - 10 for each remaining cluster member not yet updated.​​​
12. Restore any files with custom settings from the previously backed up files within /tmp/cupdate to their original location.  Review the Prerequisites section above as reference.

Note:  Commonly updated files with custom settings include those from Secure Hardening Guide (server.xml) and Increasing the memory allocated to a vRealize Orchestrator instance (setenv.sh).

13. Reboot each appliance.
14. Log in to vRO Control Center (https://FQDN:8283/vco-controlcenter/) to ensure all nodes enter a Running state under Orchestrator Cluster Management

Workaround:

Known Issues

1. Please follow vRealize Orchestrator and Monitoring client logs are not visible in the GUI if the symptoms within are encountered after applying this patch.
2. vRO services are in a failed state; /var/log/vmware/vco/app-server/server.log contain errors similar to:

ERROR {} [KeystoreCache] Cannot get JSSECACertsCache instance: Cannot verify certificate, or certificate is invalid.

java.security.cert.CertificateException: Cannot verify certificate, or certificate is invalid.

• Cause: This issue occurs due to a renamed keystore file during RPM update.
• Resolution:

1. To resolve the issue, restore the keystore file and re-create the symlink:

cp -p /etc/vco/app-server/keystore.password.rpmsave /etc/vco/app-server/keystore.password

ln -s /etc/vco/app-server/keystore.password /var/lib/vco/keystore.password

chown -h vco:vco /var/lib/vco/keystore.password

2. Restart vRO services:

   /etc/init.d/vco-server restart
   /etc/init.d/vco-configurator restart

Additional Information

Differences Between External and Embedded vRealize Orchestrator
Availability and Scalability:  Differences Between Standalone and Clustered vRealize Orchestrator

Impact/Risks:
Important: Cumulative Update Patch # 19258423 / # 19220215 and above includes permanent fixes for Workaround instructions to address CVE-2021-44228 and CVE-2021-45046 in vRealize Orchestrator 7.6.  All instances of Log4j versions 2.x are updated to 2.17.