Aria Operations Upgrade Fails on stage 9/14 with an error "FailedThe PAK action 'apply_system_update' script 'source ./pak_python_wrapper.sh updateCoordinator.py -p'"

Products

• VMware Aria Suite

Issue/Introduction

This article describes an issue encountered during an upgrade to Aria Operations that results in a failure with the error message "FailedThe PAK action 'apply_system_update' script 'source ./pak_python_wrapper.sh updateCoordinator.py -p'" in Admin UI.

When we check the log file under /var/log/vmware/vcops/vcops_upgrade_20240726-072559.log,. It describes an issue that due to an encoded character which could not get decoded in 'UTF-8' format the upgrade or Aria Operation failed. Below is an example:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Environment

Aria Operations 8.10.x and later

Cause

_{The upgrade fails due to the presence of an invalid character in one of the Aria Operations log files. The character causing the issue is encoded as byte 0x8b, which cannot be decoded using the UTF-8 character encoding.}

Resolution

There is no permanent fix available yet to prevent this issue from occurring again, as the cause of the invalid character being written to the log file is unknown. However couple of workaround exists:

Identify the affected log file:

Run the following command on the node where the upgrade failed (Value of " x8b " could be replaced with the value we get in the log files as per your issue ):

grep -c $'\x8b' /storage/log/vcops/log/pakManager/*.log | grep -v ":0$"

This command will search for the character encoded as byte 0x8b within all log files located in the /storage/log/vcops/log/pakManager directory.

The output will display the filename and line number where the invalid character is found.

There are two options to proceed:

Note: Make sure you take a backup of the files before proceeding with either of the below options:

• We can delete or modify the specific file containing the character

Option 1: Delete the affected file(s):

If the identified file is not critical for Aria Operations operation, you can simply delete it.

Option 2: Edit the affected file(s):

If the identified file contains important information, you can edit the file and remove the invalid character. Be cautious when editing system files.

After addressing the affected file(s), retry the Aria Operations upgrade.

• We can also truncate all the log files under /storage/log/vcops/log/pakManger/ as another workaround with below command:

truncate -s 0 /storage/log/vcops/log/pakManger/*.log

Additional Information

_{The provided logs indicate the error occurred during the execution of the script "vsutilities-preb2b.py" or can occur for any other scripts too.}