Patch upgrade fails at the Perform Component Upgrade step on platform nodes
search cancel

Patch upgrade fails at the Perform Component Upgrade step on platform nodes

book

Article ID: 453942

calendar_today

Updated On:

Products

VCF Operations for Networks

Issue/Introduction

A patch upgrade of VMware VCF Operations for Networks (formerly Aria Operations for Networks, vRNI) from 9.1.0 to a 9.1.0.0X00 patch fails at the Perform Component Upgrade subtask.

The stage, precheck, and prepare steps all complete successfully. Only the apply step (networks_apply_upgrade) fails. The full upgrade binary stages from the depot without error, which rules out the depot and the download path.

The VCF Operations interface reports:

Failed to execute sudo -u ubuntu /home/ubuntu/build-target/launcher/upgrade-services.sh /tmp/upgrade_infra-base_########.tar.gz /home/ubuntu/build-target/infra-base ubuntu /home/ubuntu/build-target/deployment infra-base ######## /home/ubuntu/build-target/infra-base >> /var/log/arkin/centralized_upgrade.log 2>&1
The lifecycle service retries the apply step roughly every twenty minutes. Each attempt fails identically with a new bundle file name and a new reference code, so the failure is deterministic rather than transient.

Steps to validate:

On platform node 1, /var/log/arkin/centralized_upgrade.log shows the infra-base upgrade handing off to cinc-solo and exiting non-zero on every node:

Base upgrade components complete, now running upgrade_chefrun.sh ...
cinc_role=upgrade_onprem_platform.json
CheckResult 1 'Fail to upgrade infra-base components'
CheckResult 1 'Failed to perform infra-base upgrade'
Result: {'platform1': 1, 'platform2': 1, 'platform3': 1}
/var/log/arkin/cinc-run.log identifies the failing recipe:

FATAL: Mixlib::ShellOut::ShellCommandFailed: apt_package[purge_old_kernels] (kernel::remove line 124) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '100'

Immediately above that entry, the same log records the kernel state and the package selected for removal. Kernel versions vary by appliance:

INFO: Current Kernel version: 6.8.0-106
INFO: All sorted kernel versions: 6.8.0-94, 6.8.0-106, 6.8.0-111
INFO: Kernels versions to keep : 6.8.0-106, 6.8.0-111
INFO: Kernel versions to purge: 6.8.0-94
INFO: Linux kernel metapkg check output: linux-tools-common 5.15.0-118.128
INFO: Final list of kernel packages to purge: linux-hwe-6.8-tools-6.8.0-94

The apt transaction that follows expands well beyond the single package named for removal, and ends with a missing package file:

---- Begin output of ["apt-get", "-q", "-y", "purge", "linux-hwe-6.8-tools-6.8.0-94"] ----
The following additional packages will be installed:
linux-hwe-6.8-tools-6.8.0-111 linux-tools-6.8.0-111-generic
linux-tools-generic-hwe-22.04
The following packages will be REMOVED:
linux-hwe-6.8-tools-6.8.0-94* linux-tools-6.8.0-94-generic*
The following packages will be upgraded:
linux-tools-generic-hwe-22.04
Err:3 file:/var/cache/apt-cacher-ng/####/main amd64 linux-tools-generic-hwe-22.04 amd64 6.8.0-111.11122.04.1
File not found - /var/cache/apt-cacher-ng/####/pool/main/l/linux-meta-hwe-6.8/linux-tools-generic-hwe-22.04_6.8.0-111.11122.04.1_amd64.deb (2: No such file or directory)

Environment

  • VMware VCF Operations for Networks 9.1.0
  • Platform and collector appliances running Ubuntu 22.04 LTS with the HWE 6.8 kernel
  • Online patch upgrade performed through VCF Operations lifecycle management
  • Clustered and simple deployments are both affected

Cause

The cinc recipe kernel::remove runs during the infra-base stage of the upgrade and purges kernel packages that are no longer needed. On the appliance, the linux-tools-generic-hwe-22.04 metapackage remains installed at the superseded kernel ABI version.

That metapackage depends on the tools package being removed, so apt expands the purge into a larger transaction. The transaction removes the two old tools packages and upgrades linux-tools-generic-hwe-22.04 to the newest installed kernel ABI.

The .deb file for the newer metapackage version is absent from the local apt-cacher-ng cache on the appliance. apt cannot resolve the fetch, exits with code 100, and the cinc run aborts at kernel::remove line 124. upgrade-services.sh then returns non-zero.

The metapackage file is roughly 2 KB. The other two packages in the same transaction fetch from the cache without error, so a single missing file of that size blocks the entire upgrade.

The lifecycle service polls the appliance and relays only the command line it invoked, not the script output. That is why the interface names upgrade-services.sh rather than the package failure underneath it.

Resolution

Fixed in VCF Operations for Networks 9.1.1.

Workaround

Purging the linux-tools-generic-hwe-22.04 metapackage removes the dependency that pulls the missing file into the apt transaction. The kernel::remove recipe then completes and the upgrade proceeds.

Apply the workaround to every platform node and every collector node.

  1. Take a snapshot or backup of each platform node and collector node before making any change.
  2. Open an SSH session to the first node as the support user, then switch to the ubuntu shell:

    ub
  3. Record the running kernel and the installed kernel tools packages:

    uname -r dpkg -l | grep linux-tools
  4. Simulate the purge and review the output before removing anything:

    sudo apt-get -s purge linux-tools-generic-hwe-22.04

    Confirm that linux-tools-generic-hwe-22.04 is the only entry listed under "The following packages will be REMOVED". The older linux-hwe-6.8-tools and linux-tools-generic packages appear under "automatically installed and are no longer required", which is expected and requires no action.
  5. Purge the metapackage:

    sudo apt purge -y linux-tools-generic-hwe-22.04
  6. Repeat steps 2 through 5 on each remaining platform node and collector node.
  7. Allow the lifecycle service to retry the apply step, or re-run the upgrade from Lifecycle > VCF Management. Collector nodes can take up to an hour to finish after the change.
  8. Re-run the upgrade precheck, then confirm the upgrade completes and each appliance reports the target version.

If the error persists after following these steps, contact Broadcom Support for further assistance.

Provide the following information when opening a support request with Broadcom for this issue:

  • A support bundle collected with every platform node and collector node selected
  • A screenshot of the error as shown in the VCF Operations interface
  • The output of dpkg -l | grep linux-tools from each platform node and collector node
Powered by Wolken Software