Failed Aria Operations for Networks Upgrade Requires Node Redeployment Due to Missing OS Dependencies

Products

• VCF Operations for Networks

Issue/Introduction

• Following an upgrade attempt to version 6.14.x, the Aria Operations for Networks UI is entirely inaccessible.
• Attempts to log in via CLI using the consoleuser account fail immediately with a Python traceback error: ModuleNotFoundError: No module named 'lshell'.
• Running service health checks (check-service-health.sh) reports multiple critical Python FIPS binary mismatches (e.g., Python FIPS exe: /usr/local/bin/python3.6 not found!).
• Core Java services are failing or not running:
  
  • Restapilayer
  • TSDB javaservice: NOT running
  • SaasListener: NOT running
  • DatabusGateway: Running but NOT healthy
  • Nginx: Running but NOT healthy

Environment

VCF Operations for Networks 6.13.0

VCF Operations for Networks 6.14.0

Cause

The issue is caused by a package discrepancy or incomplete cleanup of the Python environment during the infrastructure upgrade stage. The system may fail to correctly link new Python FIPS binaries or may lose the lshell module required for the consoleuser shell, resulting in broken CLI access and dependent Java services.

Resolution

For workaround, perform the steps below:

1. Access the appliance backend using the support user via SSH. If SSH is unavailable, use the vSphere console.
2. Verify missing dependencies by checking /var/log/arkin/chef-run.log for dpkg or "File not found" errors.
3. Confirm the missing FIPS path by running ls -l /usr/local/bin/python3.6
4. If snapshots or backups are available, revert the appliance to a state prior to the upgrade attempt.
5. If no snapshots are available, a redeployment is required to version 6.14.x or higher.

Additional Information

This involves manual reconstruction of the underlying OS environment and should only be attempted if no snapshots or backups are available.

Subscribe to this article for further updates.