vSAN Compute Cluster Live Patch Pre-check Fails during Upgrade from 9.1 to 9.1.0.0100

Products

• VMware vCenter Server

Issue/Introduction

When attempting to perform a Live Patch (also known as a Quick Patch) upgrade on a vSAN compute cluster from version 9.1 to 9.1.0.0100, you experience the following symptoms:

• The lifecycle pre-check fails immediately blocking the remediation task.
• The vSphere Client displays an error message similar to:

The health check failed to complete

• The vSAN Cluster Health dashboard indicates no actual health issues or warnings.
• This issue isolated to vSAN compute clusters running version 9.1 GA. Standard vSAN Hyperconverged Infrastructure (HCI) or vSAN storage clusters are unaffected, and Live Patch upgrades on those environments will complete normally.

Environment

• VMware vCenter Server 9.1
• VMware Cloud Foundation 9.1

Cause

During a Live Patch remediation, VMware vSphere Lifecycle Manager (vLCM) triggers unexpected precheck on vSAN compute cluster thereby halting the pre-check loop and blocking the upgrade process

Resolution

Broadcom is aware of the issue and is planning to fix this in a future release.

Workaround

The Live Patch remediation can proceed on a vSAN compute cluster by applying any of the two options below to bypass the failing validation:

 

Option 1 : Apply the workaround through PowerShell script

1. Install PowerShell as prerequisite step. The PowerShell requirements for different platform are listed as

Platform	PowerShell	SSH Client
Windows 10 1803+ / Windows 11	5.2+ (built-in)/7.x	Built-in — enable via Optional Features
Windows Server 2016 / 2019 / 2022	5.2+ (built-in) / 7.x	Run Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
Windows (any, older or no built-in SSH)	5.2+ (built-in)/7.x	Manual install: download OpenSSH-Win64-v*.msi
macOS (any version)	7.x — install by running brew install powershell	Built-in — nothing to install

    2. Download the attached PowerShell script: Set-VlcmCheckExclusions.ps1.
    3. Execute the script against the target vCenter Server (or multiple vCenter Servers) to apply the exclusion configuration by running the following command

.\Set-VlcmCheckExclusions.ps1 -VCHost <vCenter_IP_or_FQDN> -Operation Apply -Password "YourRootPassword"

(Note: For advanced execution parameters, please refer to the script's built-in help guide).

    4. Perform the Live Patch (Quick Patch) remediation on the vSAN compute cluster.
    5. Once the upgrade has successfully completed, revert the workaround configuration to its original state by executing the following command

.\Set-VlcmCheckExclusions.ps1 -VCHost <vCenter_IP_or_FQDN> -Operation Revert -Password "YourRootPassword"

 

Options 2 : Apply the workaround manually

Following the instructions as below to disable the vSAN live patch precheck manually.

Phase 0: Pre-validation:

1. Make sure there is no health issue for both of vSAN compute cluster and the remote vSAN storage cluster.

Phase 1: Skip the vSAN Precheck Before Live Patch:

1. Log in to the vCenter Server Appliance (VCSA) via SSH as an administrator.
2. Navigate to the Update Manager configuration directory and create a backup of the disabled_checks.json file by executing the following command:

cp /usr/lib/vmware-updatemgr/config/disabled_checks.json /usr/lib/vmware-updatemgr/config/disabled_checks.json.orig

     3. Open /usr/lib/vmware-updatemgr/config/disabled_checks.json using a text editor (such as vi) and replace its entire content with the following JSON structure

{
    "host_perspectives": {
        "BEFORE_ENTER_MAINTENANCE": [
            "com.vmware.vcIntegrity.lifecycle.health.vsan.wcp_pmm_health",
            "com.vmware.vcIntegrity.lifecycle.health.vsan.host_qp_pmm_health"
        ],
        "BEFORE_MODIFICATION": [],
        "AFTER_MODIFICATION": [],
        "BEFORE_EXIT_MAINTENANCE": [
            "com.vmware.vcIntegrity.lifecycle.health.vsan.wcp_pmm_health"
        ],
        "FINALIZE": []
    },
    "cluster_perspectives": {
        "BEFORE_ENTER_MAINTENANCE": [
            "com.vmware.vcIntegrity.lifecycle.health.vsan.cluster_qp_pmm_health"
        ],
        "BEFORE_START_APPLY": [],
        "BEFORE_HOST_REMEDIATION": []
    }
}

     4. Restart the VMware Update Manager service to apply the configuration change:

service-control --restart vmware-updatemgr

     5. Return to the vSphere Client and re-run the Live Patch / Quick Patch upgrade for the vSAN compute cluster. The pre-check should now complete successfully.

Phase 2: Revert Configuration After Live Patch:

Once the remediation process has finished successfully for the cluster, you must restore the original configuration immediately to ensure health checks function normally for any other non vSAN compute cluster:

1. Log back into the vCenter Server Appliance via SSH.
2. Revert the configuration file using the backup created in Step 2:

cp /usr/lib/vmware-updatemgr/config/disabled_checks.json.orig /usr/lib/vmware-updatemgr/config/disabled_checks.json

     3. Restart the VMware Update Manager service once more to normalize the environment:

service-control --restart vmware-updatemgr