Unable to upgrade VCF Operations due to an error "PAK Manager: The PAK action 'deploy_new_upgrade_content' script 'source ./pak_python_wrapper.sh b2bServiceConfigurator.py' failed"
search cancel

Unable to upgrade VCF Operations due to an error "PAK Manager: The PAK action 'deploy_new_upgrade_content' script 'source ./pak_python_wrapper.sh b2bServiceConfigurator.py' failed"

book

Article ID: 374769

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

  • VCF Operations upgrade fails due to an error occurred at staging "PAK Manager: The PAK action 'deploy_new_upgrade_content' script 'source ./pak_python_wrapper.sh b2bServiceConfigurator.py' failed"
  • Cluster shows as going offline with nodes in offline state and button to take node offline or online is greyed out. 
  • Log File /storage/vcops/log/casa/casa.log reports the error:
Command '/usr/bin/sudo -n ... vcopsPakManager.py --action deploy_new_upgrade_content ...' 
threw exception: CommandLineExitException: key=general.failure; args=1

ERROR ... upgrade.pak.PakService:4865 -
com.vmware.vcops.casa.exception.CasaLocalizableException: key=pak_manager.script_failed; 
args=deploy_new_upgrade_content,source ./pak_python_wrapper.sh b2bServiceConfigurator.py

Could not read filename ... casa_pak_history_vRealizeOperationsManagerEnterprise-<version>.json (No such file or directory)
  • Log file "deploy_new_upgrade_content_err.log" shows below error: 
WARNING:root:Open URL: 'https://localhost/casa/cluster/security/ping' failed with a URLERROR returned reason: [SSL:CERTIFICATE VERIFY FAILED] certificate verify failed: certificate has expired ( ssl.c: 1076), exception: <urlopen error [SSL: CERTIFICATE VERIFY FAILED] certificate verify failed: certificate has expired ( SSL.c: 1076)
Traceback (most recent call last):
File "b2bServiceConfigurator.py", line 292, in <module>
  main()
File "b2bServiceConfigurator.py", lune 288, in main
  b2bService.configure()
File "b2bServiceConfigurator.py", line 256, in configure
  is_admin_node = self.ping_node_to_find_admin()
File "b2bServiceConfigurator.py", line 194, in ping_node_to_find_admin
  response = CasaManager().get_casa_call(casa_ping_url, logger)
File "/storage/db/pakRepoLoca1/vRealizeOperationsManagerEnterprise—<version>/extracted/va1idate.py", line 2199, in get_casa_call
  return CasaManager.hand1e_http_response(url, response_code, response, logger, log_response=log_response)
File "/storage/db/pakRepoLoca1/vRealizeOperationsManagerEnterprise—<version>/extracted/va1idate.py", line 2215, in handle_http_response 
  raise CaSARestException(response code, error message)
validate.CaSARestException: Failed to call CaSA ur1 https://localhost/casa/cluster/security/ping. CASA returned status code 9000, response:
  • Below log entry in the vcopsPakManager.root.deploy_new_upgrade_content.log file pointing to exit code: 1:
ERROR 2605355 [ops@####threadId="<ThreadID>" threadName="MainThread"] [vcopsPakManager - vcopsPakManager - HandleError - 630] - Exiting with exit code: 1, message: The deploy_new_upgrade_content script: "source ./pak_python_wrapper.sh b2bServiceConfigurator.py" failed for the Pak file: "vRealizeOperationsManagerEnterprise-<version>"--exiting
  • In deploy_new_upgrade_content_stderr.log file below entries are observed:
INFO 2605362 [ops@#### threadId="<ThreadID>" threadName="MainThread"] [b2b - validate - get_casa_call] - Found status code: 9000, ignore it and try again after 10 seconds
INFO:b2b:Found status code: 9000, ignore it and try again after 10 seconds
WARNING:root:Open URL: 'https://localhost/casa/cluster/security/ping' failed with a URLERROR returned reason: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016), exception: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016)>

..
..
..

INFO 2605362 [ops@#### threadId="<ThreadID>" threadName="MainThread"] [b2b - validate - get_casa_call] - Found status code: 9000, ignore it and try again after 10 seconds
INFO:b2b:Found status code: 9000, ignore it and try again after 10 seconds
ERROR 2605362 [ops@4413 threadId="<ThreadID>" threadName="MainThread"] [b2b - validate - handle_http_response] - Failed to call CaSA url https://localhost/casa/cluster/security/ping. CaSA returned status code 9000, response:
ERROR:b2b:Failed to call CaSA url https://localhost/casa/cluster/security/ping. CaSA returned status code 9000, response:
Traceback (most recent call last):
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-<version>/extracted/b2bServiceConfigurator.py", line 189, in ping_node_to_find_admin
    response = CasaManager().get_casa_call(casa_ping_url, logger)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-<version>/extracted/validate.py", line 1504, in get_casa_call
    return CasaManager.handle_http_response(url, response_code, response, logger, log_response=log_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-<version>/extracted/validate.py", line 1520, in handle_http_response
    raise CaSARestException(response_code, error_message)
validate.CaSARestException: Failed to call CaSA url https://localhost/casa/cluster/security/ping. CaSA returned status code 9000, response:
(END)
  File "/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-<version>/extracted/b2bServiceConfigurator.py", line 287, in <module>
    main()
  File "/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-<version>/extracted/b2bServiceConfigurator.py", line 283, in main
    b2bService.configure()
  File "/storage/db/pakRepoLocal/vRealizeOperationsManagerEnterprise-<version>/extracted/b2bServiceConfigurator.py", line 251, in configure
    is_admin_node = self.ping_node_to_find_admin()

Environment

  • Aria Operations 8.18.x
  • VCF Operations 9.0.x

Cause

Custom/Default web certificate is either expired or has incorrect chain.

Resolution

Recovery via Pre-Upgrade Snapshot

If you have a snapshot taken prior to the upgrade attempt, use this workflow:

  1. Revert the appliance back to the pre-upgrade snapshot.

  2. Follow the appropriate guide below to reload your certificates based on your environment:

  3. Retry the upgrade process.

Manual Recovery and Cleanup Workflow

If a pre-upgrade snapshot is unavailable, the environment must be manually cleared of corrupted states before you can reload the certificates and retry.

Note: Please capture a fresh snapshot of the appliance(s) now before implementing the steps below. This ensures you have a reliable rollback point. (Alternatively, feel free to contact Broadcom Support for assistance with this cleanup, referencing this article).

It's recommended to bring the cluster offline from Admin UI and then restart the Operations nodes from the vCenter as per KB 340318.

1. Sanitize the History Configuration File

  1. Perform these steps on all nodes to remove corrupted or stuck upgrade entries:
  2. Log in to the node via console or SSH using the root account.
  3. Open the file for editing:
    vi /storage/db/pakRepoLocal/vcopsPakManagerCommonHistory.json
  4. Delete the unwanted or corrupted upgrade entries. Need to keep entries till last successful upgrade/patch.
    Important JSON Formatting Check:** Ensure the file maintains valid syntax. The very first non-space character must be an opening bracket `[` and the last non-space character must be a closing bracket `]`. 
    Check for the "pak": "<component-version>" to validate if the entry is for old upgrade/patch or the latest.

    Eg.  
      {
            "action": "stage",
            "action_result": "Completed successfully",
            "exit_code": "0",
            "pak": "vRealizeOperationsManagerEnterprise-<Old-Version>",
            "time_updated": "############"
        },
        {
            "action": "deploy_new_upgrade_content",
            "action_result": "Failed",
            "exit_code": 1,
            "pak": "vRealizeOperationsManagerEnterprise-<New-Version>",
            "time_updated": "###########"
        }
    ]
    
* Save and exit the file:
  ```text
:wq!
  5. Validate and repeat this configuration across all nodes in the cluster.

2. Clear the CASA-B2B Directory and Restart the Service

  1. Navigate to the library directory:
    cd /usr/lib
  2. Move or remove the corrupted directory.
     *Moving the directory to a temporary location is recommended as a safer alternative to direct deletion:*
  # Preferred method: Move the directory to /tmp
  mv /usr/lib/vmware-casa-b2b /tmp

  # Alternative method: Delete the directory entirely
  rm -Rf vmware-casa-b2b
  3. Restart the CASA service to apply the changes:
    service vmware-casa restart 

Once the file system and service cleanup steps are complete, take the cluster online and follow the appropriate certificate KB (linked in the above section) to reload your default or custom certificates, then re-initiate the upgrade process.

Powered by Wolken Software