Host compliance check tasks stuck on the vCenter Server for a long time.
Article ID: 414311
Updated On:
Products
VMware vCenter Server
VMware vSphere ESXi
Issue/Introduction
- Remediation tasks are not able to proceed due to large number of compliance tasks stuck on the vCenter Server .
- Compliance check tasks are stuck indefinitely on the vCenter Server.
Environment
vCenter Server
VMware vSphere ESXi
Cause
Initiating multiple scan tasks can overload the Lifecycle Manager with tasks that will get queued on the Lifecycle Manager.
Resolution
Check the time taken for the scan tasks to complete on the ESXi host :
- Use command below to check on the lifecycle logs on the ESXi host if there are scan tasks that were initiated around the time of the issue. :
grep "software --scan" lifecycle.* | grep "YYYY-MM-DDT05:39" - You will see an entry showing the task ID
############-####-####-####-########b148:lifecycle.0:YYYY-MM-DDT05:39:04Z lifecycle: 46061757: imagemanagerctl:1171 INFO Calling with arguments: software --scan --software-spec /var/run/lifecycle.scan.swspec --depot http://vcenter.domain.com:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__DEL-ESXi-8.0-Addon-cumulative_metadata__index__.xml --depot http://vcenter.domain.com:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-107__index__.xml --depot http://vcenter.domain.com:9084/vum/repository/hostupdate/__micro-depot__vendor-EMC__metadata-205__index__.xml --depot http://vcenter.domain.com:9084/vum/repository/hostupdate/__micro-depot__vendor-HTI__HTI-ESXi-8.0-Addon-cumulative_metadata__index__.xml --depot http://vcenter.domain.com:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__vmw-ESXi-8.0.3-metadata__index__.xml --depot http://vcenter.domain.com:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-97__index__.xml --depot http://vcenter.domain.com:9084/vum/repository/hostupdate/__micro-depot__vendor-DEL__DEL-ESXi-7.0.3-Addon-cumulative_metadata__index__.xml --depot http://vcenter.domain.com:9084/vum/repository/hostupdate/__micro-depot__vendor-vmw__metadata-413__index__.xml --task-id ############-####-####-####-########b148 - Check the time taken for the individual task to complete using the host id .
grep -ia "############-####-####-####-########b148" vmware-vum-server* | grep -ia "host-####1"YYYY-MM-DDT05:39:03.743Z info vmware-vum-server[90113] [Originator@6876 sub=ScanUtil opID=########-####-####-####-#########5d3] [ScanUtil 678] Got taskId: ############-####-####-####-########b148 from Host :esxi#.domain.com (host-####1) for scan task. YYYY-MM-DDT05:39:52.040Z info vmware-vum-server[90113] [Originator@6876 sub=ScanUtil opID=########-####-####-####-#########5d3] [ScanUtil 902] Fetching the results for task(############-####-####-####-########b148) on host(host-####1)The timestamps observed will show the time taken for one host scan task to complete.
Check the number of scan tasks initiated around the same timestamp :
- On the
/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-###.loglogs, run the below command to check if multiple scan tasks were initiated on the vCenter Server using the timestamp :
grep "Got taskId" vmware-vum-server-###.log | grep -ia "YYYY-MM-DDT05:39"YYYY-MM-DDT05:39:05.108Z info vmware-vum-server[90113] [Originator@6876 sub=ScanUtil opID=########-####-####-####-#########5d3] [ScanUtil 678] Got taskId: ############-####-####-####-########bc85 from Host :esxi#.domain.com (host-####2) for scan task. YYYY-MM-DDT05:39:05.175Z info vmware-vum-server[90113] [Originator@6876 sub=ScanUtil opID=########-####-####-####-#########5d3] [ScanUtil 678] Got taskId: ############-####-####-####-########c00 from Host :esxi##.domain.com (host-####3) for scan task. YYYY-MM-DDT05:39:05.242Z info vmware-vum-server[90113] [Originator@6876 sub=ScanUtil opID=########-####-####-####-#########5d3] [ScanUtil 678] Got taskId: ############-####-####-####-########ddb0 from Host :esxi###.domain.com (host-####4) for scan task. YYYY-MM-DDT05:39:05.305Z info vmware-vum-server[90113] [Originator@6876 sub=ScanUtil opID=########-####-####-####-#########5d3] [ScanUtil 678] Got taskId: ############-####-####-####-########a1a from Host :esxi####.domain.com (host-####5) for scan task. YYYY-MM-DDT05:39:05.370Z info vmware-vum-server[90113] [Originator@6876 sub=ScanUtil opID=########-####-####-####-#########5d3] [ScanUtil 678] Got taskId: ############-####-####-####-########5bf7 from Host :esxi#####.domain.com (host-####6) for scan task. - On the journalctl logs, check for recently initiated scan tasks :
cat /commands/journalctl_-b--* | grep -ia "Task: Check compliance of" | grep "YYYY-MM-DD"Mmm DD 03:44:54 vcenter.domain.com vpxd[4032518]: Event [312961417] [1-1] [YYYY-MM-DDT03:44:54.168284Z] [vim.event.TaskEvent] [info] [[email protected]] [ClusterName] [312961417] [Task: Check compliance of host with image] Mmm DD 03:47:54 vcenter.domain.com vpxd[4032518]: Event [312962416] [1-1] [YYYY-MM-DDT03:47:54.544429Z] [vim.event.TaskEvent] [info] [[email protected]] [ClusterName] [312962416] [Task: Check compliance of cluster with image] Mmm DD 03:49:06 vcenter.domain.com vpxd[4032518]: Event [312962942] [1-1] [YYYY-MM-DDT03:49:06.591391Z] [vim.event.TaskEvent] [info] [[email protected]] [ClusterName] [312962942] [Task: Check compliance of cluster with image] Mmm DD 03:55:41 vcenter.domain.com vpxd[4032518]: Event [312964716] [1-1] [YYYY-MM-DDT03:55:41.571078Z] [vim.event.TaskEvent] [info] [[email protected]] [ClusterName] [312964716] [Task: Check compliance of cluster with image] Mmm DD 03:58:24 vcenter.domain.com vpxd[4032518]: Event [312965709] [1-1] [YYYY-MM-DDT03:58:24.431696Z] [vim.event.TaskEvent] [info] [[email protected]] [ClusterName] [312965709] [Task: Check compliance of cluster with image] Mmm DD 05:01:32 vcenter.domain.com vpxd[4032518]: Event [312983543] [1-1] [YYYY-MM-DDT05:01:32.622611Z] [vim.event.TaskEvent] [info] [[email protected]] [ClusterName] [312983543] [Task: Check compliance of cluster with image] Mmm DD 05:39:02 vcenter.domain.com vpxd[4032518]: Event [312994023] [1-1] [YYYY-MM-DDT05:39:02.610414Z] [vim.event.TaskEvent] [info] [[email protected]] [ClusterName] [312994023] [Task: Check compliance of cluster with image] - Check the entries that define the load present on the vCenter Lifecycle Manager using command below :
grep -ia "VMware vSphere Update Manager new load" ../var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-###.log
The vCenter Lifecycle Manager can be confirmed to be running on a high load and it will continue to increase as per the logs.YYYY-MM-DDT04:30:01.202Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########9981 that imposes 1 for VMware vSphere Update Manager new load: 5, task active Load: 1 YYYY-MM-DDT05:01:32.630Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########1764 that imposes 1 for VMware vSphere Update Manager new load: 5, task active Load: 1 YYYY-MM-DDT05:38:07.465Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########7f84 that imposes 2 for VMware vSphere Update Manager new load: 6, task active Load: 1 YYYY-MM-DDT05:39:02.616Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########3f5a that imposes 0 for VMware vSphere Update Manager new load: 6, parent task (############-####-####-####-########7f84) active load: 2 YYYY-MM-DDT06:27:07.528Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########8b4e that imposes 2 for VMware vSphere Update Manager new load: 8, task active Load: 1 YYYY-MM-DDT06:28:15.665Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########120c that imposes 0 for VMware vSphere Update Manager new load: 8, parent task (############-####-####-####-########8b4e) active load: 2 YYYY-MM-DDT06:31:38.165Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########62f5 that imposes 1 for VMware vSphere Update Manager new load: 7, task active Load: 1 YYYY-MM-DDT06:31:48.212Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########95a3 that imposes 1 for VMware vSphere Update Manager new load: 7, task active Load: 1 YYYY-MM-DDT06:32:03.568Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########1f2 that imposes 2 for VMware vSphere Update Manager new load: 8, task active Load: 1 YYYY-MM-DDT06:33:13.429Z info vmware-vum-server[4041279] [Originator@6876 sub=JobDispatcher] [JobDispatcher 2167] Scheduling task ############-####-####-####-########865c that imposes 0 for VMware vSphere Update Manager new load: 8, parent task (############-####-####-####-########1f2) active load: 2
To perform a fresh scan task on the hosts / cluster :
- Identify if the vCenter Server Lifecycle Manager has a large number using the above steps and restart the '
vmware-updatemgr' service.service-control --restart vmware-updatemgr - Initiate a fresh "Check Compliance" task on the hosts and verify that the check tasks are completing correctly.
Feedback
Yes
No
Powered by
