Database Upgrade Fails and RaaS Service Remains Offline After Upgrading Aria Automation Config to 8.18.3

Products

VCF Operations/Automation (formerly VMware Aria Suite) VMware Salt

Issue/Introduction

Upgrading or reinstalling the Returner as a Service (RaaS) daemon requires comprehensive artifact cleanup, secure service account configuration, and credential generation. Failure to properly isolate the raas service account home directory or clear legacy directories can result in upgrade failures, schema upgrade deadlocks, or database credential authentication errors.

Environment

VMware Aria Automation Config (formerly SaltStack Config) 8.18.3

Cause

Architectural enhancements to the application delivery framework introduce new environmental and security isolation prerequisites. When upgrading from legacy versions, the base operating system may retain historical directory mappings and service account profiles that do not align with these updated structural requirements. This environmental divergence prevents the daemon from initializing correctly, leading to sequential startup failures as the service attempts to access missing or incorrectly permissioned state directories, configuration paths, and virtual environments.

Resolution

Step 1: Package Removal and Artifact Cleanup

Remove the legacy RaaS daemon package while preserving dependencies, then identify and clear out legacy state directories.

Bash 
dnf remove --noautoremove raas
find / -iname "raas"
rm -fRv /var/lib/raas

Step 2: Secure State Directory Allocation

Create an isolated home directory for the raas service account. This allows the application to store runtime state without granting write access to the binary execution path (/opt/saltstack/raas/).

Bash 
mkdir -p /app/raas
chown -R raas:raas /app/raas

Step 3: Service Account Profile and Certificate Configuration

Reconfigure the service account profile to utilize the newly created isolated home directory, and enforce explicit ownership over the PKI path.

Bash 
usermod -d /app/raas raas
chown -Rv raas:raas /etc/pki/raas

Note: Ensure the /etc/raas/pki/.raas.key file maintains 640 permissions, is owned by the raas user, and matches the original backup checksum.

Step 4: Secure Database Credential Re-Initialization

Reset the salt_eapi PostgreSQL user password securely. To prevent credential logging in plain-text .bash_history files, utilize the interactive psql password prompt.

Bash 
psql -U postgres
\password salt_eapi

Step 5: Virtual Environment Activation and Security Configuration

Switch the session context to the raas user, set the environment variables, and activate the daemon's Python virtual environment. Generate the raas.secconf file and perform the necessary database schema upgrade.

Bash 
sudo -u raas /bin/bash
export HOME=/app/raas
export PATH=/opt/saltstack/raas/bin:$PATH
source /opt/saltstack/raas/bin/activate

raas -l debug save_creds postgres='{"username": "salt_eapi","password":"<DB_PASS>"}' redis='{"username":"<REDIS_USER>","password":"<REDIS_PASS>"}'
raas -l debug upgrade

Step 6: Service Initialization

Initialize the RaaS daemon and monitor the systemd service status for a successful startup sequence (indicated by the ASCII 'C' logo in the standard output logs).

Bash 
systemctl start raas
systemctl status raas

Additional Information

Additional Information

The startup error referring to ovfenv seems benign if FIPS mode is not enabled
- See https://knowledge.broadcom.com/external/article/400149/ovfenv-command-returns-unable-to-find-th.html for more information.
The /opt/saltstack/raas/ directory must remain owned by root:root with 755 permissions to prevent binary tampering.

Troubleshooting

Command Not Found Errors: If the raas executable is unrecognized after switching users, verify the PATH environment variable correctly includes /opt/saltstack/raas/bin and the virtual environment was sourced properly via the activate script.
Database Connection Failures: Confirm that the credential payload passed in the save_creds JSON object precisely matches the updated interactive string provided to the \password prompt in Step 4.