Error: "An unexpected error occurred in step ops_precheck" while upgrading VCF Operations from 9.1.0.0 to 9.1.0.0100
search cancel

Error: "An unexpected error occurred in step ops_precheck" while upgrading VCF Operations from 9.1.0.0 to 9.1.0.0100

book

Article ID: 452677

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

VCF Operations Upgrade Prechecks error in the user interface, navigate > Build > Lifecycle > VCF Management > Upgrade

Error    An unexpected error occurred in step ops_precheck. Reference Code: 70BACE6F. Please contact support with this reference code. Detail: 401 401: "<!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style>"...
  • /storage/logs/vcops/logs/pakManager/APUAT-for-####-91002543504/apply_system_update_stderr.log
    ... DEBUG ... [ops@4413 threadId="################" threadName="MainThread"] [b2b - validateMoreChecks - run_script] - Script command: "/usr/sbin/httpd -t" failed with exit code: 1 DEBUG:b2b:Script command: "/usr/sbin/httpd -t" failed with exit code: 1
  • /storage/logs/vcops/logs/casa-webapp/casa.log
    INFO casa ...[ops@... threadId="######" threadName="ajp-nio-127.0.0.1-8011-exec-14" requestId="2V000Vbl"] [PAK_MANAGER_QUERY_LOGGER.queryStagedPak:207] - While retrieving PAK status for APUAT-for-814x-910025435043, exception=CommandLineExitException: key=general.failure; args=2,/usr/lib/vmware-python-3/bin/python: can't open file '/usr/lib/vmware-casa-b2b/casa-b2b/pakManager/bin/vcopsPakManager.py': [Errno 2] No such file or directory ; cause=

Environment

VCF Operations 9.1.

Cause

The upgrade preparation (APUAT) ran on all  the analytics nodes, but it fails as one of the analytics nodes is rebooted  manually. The reboot interrupted the step before it could finish,  the affected node upgrade status is permanently "stuck" at that phase and never advanced.

Resolution

If you experience upgrade issues during VCF Operations, collect and download the full log bundle, then revert to your snapshots to avoid downtime. Once completed, raise a case with Technical Support for further assistance.

If you do not have snapshots available, proceed with the workaround below:

  1. Take a VCF Operations snapshot without memory on each node if the upgrade is still ongoing.
  2. Delete all the nested JSON formatted records related to APUAT-for814x-910025435043 from the vcopsPakManagerCommonHistory.json file. They will look like the example below:
      ... 
    {   
        "action": "stage",
        "action_result": "Completed successfully",
        "exit_code": "0",
        "pak": "APUAT-for814x-910025435043",
        "time_updated": "############"
    },
    {   
        "action": "validate",
        "action_result": "Completed successfully-no script to run",
        "exit_code": "0",
        "pak": "APUAT-for814x-910025435043",
        "time_updated": "############"
    }, 
    {   
        "action": "apply_adapter",
        "action_result": "Completed successfully",
        "exit_code": "0",
        "pak": "APUAT-for814x-910025435043",
        "time_updated": "############"
    },
...

    Run the following commands on each VCF Operations analytics node:

    cd /storage/db/pakRepoLocal/
ls -lhtr
cd APUAT-for814x-910025435043
cd ..
mv APUAT-for814x-910025435043 moved-APUAT-for814x-910025435043
mv 9.1.0.0100.25435105 moved-9.1.0.0100.25435105
  3. Retry the VCF Operations upgrade.

Additional Information

Unable to upgrade VCF Operations due to an error "PAK Manager: The PAK action 'deploy_new_upgrade_content' script 'source ./pak_python_wrapper.sh b2bServiceConfigurator.py' failed"

Powered by Wolken Software