vSphere HA Configuration Fails Due to Stale Lifecycle Task Locks or DNS Misconfiguration
search cancel

vSphere HA Configuration Fails Due to Stale Lifecycle Task Locks or DNS Misconfiguration

book

Article ID: 452246

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article outlines the troubleshooting and resolution steps for vSphere High Availability (HA) configuration failures that occur when a host fails to apply cluster tasks or cannot resolve hostnames, specifically after vCenter server upgrades or interrupted lifecycle operations.

Symptoms

    • vSphere HA (Fault Domain Manager) configuration tasks fail with an error similar to ApplyHAVibsOnClusterFailed.
    • The system indicates that configuration components are blocked.
    • Intermittent hostname resolution failures, where DNS entries added to /etc/hosts manually do not persist after an ESXi host reboot.
    • HA configuration tasks consistently fail even after reboots, cluster-level disabling/enabling of HA, or host reconnection.

Environment

vCenter 8.0.3k

Cause

This issue typically arises from one of two root causes:

  1. Stale Lifecycle Task Lock: If a vSphere Lifecycle Manager (vLCM) operation or scan was forcibly terminated (e.g., via kill -9), it may leave behind a persistent task lock file (/var/vmware/lifecycle/lifecycle.task) and status database (task-status.json.db). Because these files reside on persistent storage, the lock persists across reboots, preventing subsequent attempts to apply configuration changes or HA VIBs.
  2. Non-Persistent DNS Configuration: Editing /etc/hosts manually using text editors is not a persistent configuration method on ESXi. The file is regenerated from the ConfigStore on every boot, causing custom entries (such as vCenter hostname mappings) to be wiped, leading to name resolution failures that break the agent communication.

Resolution

Part 1: Correcting Persistent DNS Configuration

If the host cannot resolve the vCenter Server hostname, ensure the DNS entry is added via the proper CLI command to ensure persistence.

  1. Access the ESXi host via SSH.
  2. Add the vCenter Server entry using the esxcli command (do not edit /etc/hosts manually): esxcli network ip hosts add --hostname=vcenter_hostname --ip=vcenter_ip_address
  3. Verify the entry is added: esxcli network ip hosts list

Part 2: Clearing Stale Lifecycle Task Locks

If the HA configuration continues to fail due to a "blocked" state, check for and remove orphaned lifecycle task files.

  1. Access the ESXi host via SSH.
  2. Navigate to the lifecycle directory: cd /var/vmware/lifecycle/
  3. Check for the existence of lock files or status databases that indicate a stuck task:
    • ls -l lifecycle.task
    • cat task-status.json.db
  4. If a task is identified as stuck or orphaned (i.e., you are certain no valid vLCM scan is running), remove the lock files: rm /var/vmware/lifecycle/lifecycle.task rm /var/vmware/lifecycle/task-status.json.db (Note: These files will be automatically recreated by the system when a new, valid lifecycle task is triggered.)

Part 3: Re-enable vSphere HA

  1. Log in to the vSphere Client.
  2. Navigate to the cluster's vSphere Availability settings (Configure > Services > vSphere Availability).
  3. If HA is currently enabled, toggle it Off, wait for the task to complete, and then toggle it back On.
  4. Monitor the cluster tasks. The vCenter Server should now successfully push the HA agent configuration without being blocked by previous task locks.

Additional Information

  • Always use esxcli for system configuration changes (DNS, networking, etc.) to ensure that changes are written to the ConfigStore and survive system reboots.
  • Avoid using kill -9 on system processes unless explicitly advised by support, as this bypasses clean-up routines and leads to orphaned locks or inconsistent system states.
Powered by Wolken Software