SDDC Cluster Creation Fails: Hosts are in 'Incompatible' Compliance State During ''Remediate ESXi Host(s) to be compliant with Cluster's Image' Task (VLCM_REMEDIATE_PERSONALITY_FAILED)
search cancel

SDDC Cluster Creation Fails: Hosts are in 'Incompatible' Compliance State During ''Remediate ESXi Host(s) to be compliant with Cluster's Image' Task (VLCM_REMEDIATE_PERSONALITY_FAILED)

book

Article ID: 422030

calendar_today

Updated On:

Products

VMware SDDC Manager VMware Cloud Foundation VMware vCenter Server

Issue/Introduction

  • During the creation or expansion of an SDDC cluster via VMware Cloud Foundation (VCF) SDDC Manager, the task 'Remediate ESXi Host(s) to be compliant with Cluster's Image' fails.
  • The underlying cause, found in the logs, indicates that one or more ESXi hosts are in an 'incompatible' compliance state with the desired vSphere Lifecycle Manager (vLCM) cluster image.
  • This failure typically manifests in the SDDC Manager task history and the relevant logs with the error VLCM_REMEDIATE_PERSONALITY_FAILED.
  • Error in SDDC Manager lcm-debug.log (/var/log/vmware/vcf/lcm/lcm-debug.log) 
    ERROR [vcf_dm,6############################d,5186] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-7]  [V####8] VLCM_REMEDIATE_PERSONALITY_FAILED Applying personality to cluster domain-c#### failed with error: com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vcIntegrity.lifecycle.RemediateClusterTask.IncompatibleHosts,
    defaultMessage = Cluster 'sample-cluster-vsan' remediation failed because hosts 'esxi01.example.com, esxi02.example.com, esxi03.example.com, esxi04.example.com' are in 'incompatible' compliance state.,
    args = [sample-cluster-vsan, esxi01.example.com, esxi02.example.com, esxi03.example.com, esxi04.example.com],
    params = <null>,
    localized = Cluster 'sample-cluster-vsan' remediation failed because hosts 'esxi01.example.com, esxi02.example.com, esxi03.example.com, esxi04.example.com' are in 'incompatible' compliance state.
}],
    data = <null>,
    errorType = ERROR
}
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Applying personality to cluster domain-c#### failed with error: com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vcIntegrity.lifecycle.RemediateClusterTask.IncompatibleHosts,
    defaultMessage = Cluster 'sample-cluster-vsan' remediation failed because hosts esxi01.example.com, esxi02.example.com, esxi03.example.com, esxi04.example.com' are in 'incompatible' compliance state.,
    args = [sample-cluster-vsan, esxi01.example.com, esxi02.example.com, esxi03.example.com, esxi04.example.com],
    params = <null>,
    localized = Cluster 'sample-cluster-vsan' remediation failed because hosts 'esxi01.example.com, esxi02.example.com, esxi03.example.com, esxi04.example.com' are in 'incompatible' compliance state.
}],
    data = <null>,
    errorType = ERROR
}

     

  • Error in vCenter Server Lifecycle Manager Log (/var/log/vmware/vlcm/vlcm.log) 
    info vmware-vum-server[09383] [Originator@6876 sub=PM.AsyncTask.RemediateClusterTask{125}] [vciTaskBase 1493] SerializeToVimFault fault:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vcIntegrity.lifecycle.RemediateClusterTask.IncompatibleHosts",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "1",
-->                value = "sample-cluster-vsan"
-->             },
-->             (vmodl.KeyAnyValue) {
-->                key = "2",
-->                value = "esxi01.example.com, esxi02.example.com, esxi03.example.com, esxi04.example.com"
-->             }
-->          ],
-->          message = <unset>
-->       }
-->    ],
-->    reason = "vLCM Task failed, see Error Stack for details."
-->    msg = "{
-->     "data": null,
-->     "error_type": "ERROR",
-->     "messages": [
-->         {
-->             "args": [
-->                 "sample-cluster-vsan",
-->                 "esxi01.example.com, esxi02.example.com, esxi03.example.com, esxi04.example.com"
-->             ],
-->             "default_message": "Cluster 'sample-cluster-vsan' remediation failed because hosts 'esxi01.example.com, esxi02.example.com, esxi03.example.com, esxi04.example.com' are in 'incompatible' compliance state.",
-->             "id": "com.vmware.vcIntegrity.lifecycle.RemediateClusterTask.IncompatibleHosts"

Environment

  • VMware Cloud Foundation
  • SDDC Manager
  • vCenter Server
  • vSphere Lifecycle Manager (vLCM) enabled clusters

Cause

  • The immediate cause is the hosts incompatibility with the cluster's desired image, which is managed by vLCM.
  • The specific underlying issue identified in this scenario was: The vLCM compliance check reported an error related to the downgrade of one or more Add-on components that were part of the existing ESXi host configuration but were not included in the cluster's target image definition. vLCM prevents component downgrades during remediation to maintain stability.

Resolution

The resolution involves manually updating the vLCM desired image for the cluster in vCenter to include the necessary component versions that are already present on the ESXi hosts. This is done to achieve a compliant state before VCF remediation can proceed.

 

Follow these steps:

  1. Identify the Incompatibility:
    1. Navigate to the cluster in the vSphere Client.
    2. Go to Updates > Image.
    3. Click Run Compliance Check (if not already run).
    4. Review the Compliance section and note the specific component(s) that are causing the downgrade error.

  2. Edit the Desired Image:
    1. Click on Edit Image for the cluster.
    2. In the Add-ons section, search for and add the component(s) identified in the compliance check.
    3. Ensure the version selected for the Add-on is the same or newer than the version currently installed on the hosts to prevent the downgrade error.

  3. Validate Compliance:
    1. After saving the modified image, run another Compliance Check.
    2. The hosts should now transition to a Compliant state.

  4. Retry the VCF Task:
    1. Once vCenter reports the hosts as compliant with the cluster image, return to the SDDC Manager UI.
    2. Retry the failed VCF task, 'Remediate ESXi Host(s) to be compliant with Cluster's Image'

 

This process aligns with the procedures documented in KB Article 320562 . The successful compliance check in vCenter resolves the underlying blocking condition, allowing the VCF SDDC Manager remediation task to complete.

Powered by Wolken Software