Remediation of vLCM Image based cluster failed with error: compliance check reported host as 'non-compliant'. The image on the host does not match the image set for the cluster.
Article ID: 395691
Updated On:
Products
Issue/Introduction
Similar issues related to memory exhaustion and compliance check failures are being reported. These issues typically manifest as follows:
- After a host remediation process completes, the vLCM compliance check reports the host as 'non-compliant.' The error message indicates a mismatch between the image on the host and the image set for the cluster.
- An error occurs when initiating a host compliance check, specifically with the message "Failed to start the task. Please ensure the system has enough resources and retry."
- The issue is consistently observed when third-party VIBs are being remediated using a cluster image. This suggests that the issue is directly related to the way these VIBs interact with the host's memory management, potentially failing to allocate their own memory pools.
Environment
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
Cause
The "Image Compliance Task" is failing with an "Insufficient Resources" message due to memory exhaustion within the 'settingsd' service. This occurs when third-party VIB's fail to create their own dedicated memory pools and instead rely on the existing memory pool of the 'settingsd' service.
Resolution
While the root cause of memory exhaustion during image compliance checks, stemming from third-party VIBs not creating dedicated memory pools, requires a permanent solution, two temporary workarounds have been identified:
1. Host Reboot or Service Restart:
Rebooting the ESXi host or restarting relevant services, particularly the 'settingsd' service, can temporarily alleviate the memory pressure and enable successful compliance checks. This approach effectively clears the memory pool, allowing for a fresh start. However, it is not a sustainable solution as the issue will likely re-emerge as third-party VIBs resume memory consumption.
2. Manual Patching with 3rd party VIB:
In certain cases, manually patching the ESXi host with the 3rd party vib has been observed to resolve the compliance check failure. This suggests a potential conflict or incompatibility issue between the 3rd party vib and the host's memory management, specifically related to the 'settingsd' service. However, this workaround may mask underlying issues and should not be considered a permanent fix.
Conclusion:
Both workarounds offer temporary relief but do not address the fundamental issue of third-party VIBs relying on the 'settingsd' service's memory pool. A long-term solution requires ensuring all third-party VIBs adhere to best practices by creating their own dedicated memory pools. This will prevent memory exhaustion and ensure the stability and performance of the ESXi host during compliance checks.
Additional Information
Prior to version 16.10, NVIDIA VIBs were known to cause vLCM (vSphere Lifecycle Manager) Cluster Image compliance check failures during host remediation. This issue stemmed from the NVIDIA VIB's failure to create its own dedicated memory pool, leading to resource depletion and insufficient memory for the compliance check process.
Similar issues have been observed with other third-party VIBs, such as CISCO UCS VIB and DELL OpenManage Server Administrator, indicating a broader problem with memory management within these VIBs.
To address these issues, the following steps are recommended:
- Contact the respective vendors (NVIDIA, Cisco, Dell) to report the issues and collaborate on potential solutions. Share detailed logs and error messages to assist in troubleshooting.
- Open a support ticket with Broadcom Support to isolate whether the failure is specifically related to third-party VIBs or if other contributing factors are present. Provide detailed information about the affected VIBs, the steps taken to reproduce the issue, and any relevant logs for comprehensive investigation.
- Reference this knowledge base article to the support ticket to provide context and facilitate a thorough understanding of the issue.
By proactively engaging with vendors and Broadcom support, a more efficient resolution can be achieved and future compliance check failures related to third-party VIBs can be mitigated.
Feedback
