vco-app pod does not start after upgrade from 8.18.1 to 9.0.0

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Upgrading Standalone Aria Automation Orchestrator 8.18.x to standalone VCF Operations Orchestrator 9.0 following the process in Upgrading Standalone or Clustered VCF Operations Orchestrator Deployments

• The upgrade may show as successful, but running kubectl -n prelude get pods may show that the vco-app pods are not fully running:
  vco-app-##########-#####                  1/2     Running   0          30m
  
  
• Or the upgrade may show as failed after something because the health checks failed
  
  
• In some instances, the vco-app pods may both be running, but running vracli vro authentication shows "No authentication provider configured." 
  
  
• Logs at /services-logs/prelude/vco-app/console-logs/install-rpms.log show errors similar to this:
  "+ run_init_hook
+ '[' -x /usr/lib/vco/app-server/postupdate.sh ']'
+ /bin/bash /usr/lib/vco/app-server/postupdate.sh
Unmatched arguments from index 15: 'Test', 'Admins', 'Group'"
  
  
  • In this example the administrators group used for Authentication was set to use a group named "Automation Test Admins Group"
  • Note: the install-rpms.log may roll after the upgrade attempt and will be in an archive format such as install-rpms.log.20250814191511.xz 

Environment

VCF Operations Orchestrator 9.0 Standalone

Cause

This issue occurs when the upgrade script attempts to reconfigure authentication and Orchestrator authentication has been configured with an administrator group that contains spaces.

Resolution

This is a known issue that is fixed in VCF Operations Orchestrator 9.0.1.

Workaround 1: 
Rollback and change the admin group to one without spaces

1. Roll back to your snapshot from prior to starting the upgrade
2. Connect to the Orchestrator appliance via SSH or console as root
       Note: In a cluster, the commands only need to be run on one node
3. Take note of your current authentication settings by running vracli vro authentication
4. Run vracli vro authentication wizard
       Note: If you prefer not to use the wizard, you can configure authentication as per the documentation: Configuring the Automation Orchestrator Appliance authentication provider with the command line interface
5. Follow the prompts using the wizard, configure authentication and set the administrators group to a group that does not contain spaces
6. Run the upgrade again as per Upgrading Standalone or Clustered VCF Operations Orchestrator Deployments
7. If desired, the group may be changed back to the original group that contained spaces.

Workaround 2:
If you are unable to roll back, authentication can be set after the upgrade completes or fails:

1. Connect to the Orchestrator appliance via SSH or console as root
       Note: In a cluster, the commands only need to be run on one node
2. Run vracli vro authentication wizard
    Note: If you prefer not to use the wizard, you can configure authentication as per the documentation: Configuring the Automation Orchestrator Appliance authentication provider with the command line interface
3. Using the wizard, configure authentication
       Note: The group containing spaces can be used at this time
4. Restart services by running /opt/scripts/deploy.sh.

Additional Information

For other known issues, related to upgrading, please refer to VCF Operations Orchestrator 9 Known Issues