Enterprise PKS shows continuous Scan and Fix messages in BOSH Tasks
Article ID: 316816
Updated On:
Products
VMware Cloud PKS
Issue/Introduction
- You see continuous "Scan and Fix" messages in the BOSH Tasks, similar to the following:
ID State Started At Last Activity At User Deployment Description Result
636816 done Thu May 2 21:12:33 UTC 2019 Thu May 2 21:14:56 UTC 2019 pivotal-container-service-4a9d72b40a937240e75d service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e run errand telemetry-agent from deployment service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e 1 succeeded, 0 errored, 0 canceled
636815 done Thu May 2 21:09:43 UTC 2019 Thu May 2 21:12:27 UTC 2019 pivotal-container-service-4a9d72b40a937240e75d service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e run errand apply-addons from deployment service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e 1 succeeded, 0 errored, 0 canceled
636814 done Thu May 2 21:00:00 UTC 2019 Thu May 2 21:00:00 UTC 2019 scheduler clean up local dns blobs Deleted 1 dns blob(s) created before 2019-05-02 20:00:00 +0000
636813 done Thu May 2 21:00:00 UTC 2019 Thu May 2 21:00:00 UTC 2019 scheduler clean up events Old events were deleted
636812 done Thu May 2 20:54:15 UTC 2019 Thu May 2 21:09:38 UTC 2019 pivotal-container-service-4a9d72b40a937240e75d service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e create deployment /deployments/service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e
636811 done Thu May 2 20:44:05 UTC 2019 Thu May 2 20:44:11 UTC 2019 health_monitor pivotal-container-service-4a9d72b40a937240e75d scan and fix scan and fix complete
636810 done Thu May 2 20:44:05 UTC 2019 Thu May 2 20:44:12 UTC 2019 health_monitor service-instance_74e887b0-b7d5-42b3-9359-b9f1f8e577d4 scan and fix scan and fix complete
636809 done Thu May 2 20:44:05 UTC 2019 Thu May 2 20:44:12 UTC 2019 health_monitor service-instance_b18c7dea-4b8e-4bb8-8b66-0e29b91c1fc4 scan and fix scan and fix complete
636808 error Thu May 2 20:44:05 UTC 2019 Thu May 2 20:44:05 UTC 2019 health_monitor service-instance_eb48ea6f-3854-42b1-a99a-83222ce89a59 scan and fix Unable to get deployment lock, maybe a deployment is in progress. Try again later.
636807 error Thu May 2 20:44:04 UTC 2019 Thu May 2 20:44:04 UTC 2019 health_monitor service-instance_74e887b0-b7d5-42b3-9359-b9f1f8e577d4 scan and Unable to get deployment lock, maybe a deployment is in progress. Try again later.
636816 done Thu May 2 21:12:33 UTC 2019 Thu May 2 21:14:56 UTC 2019 pivotal-container-service-4a9d72b40a937240e75d service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e run errand telemetry-agent from deployment service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e 1 succeeded, 0 errored, 0 canceled
636815 done Thu May 2 21:09:43 UTC 2019 Thu May 2 21:12:27 UTC 2019 pivotal-container-service-4a9d72b40a937240e75d service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e run errand apply-addons from deployment service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e 1 succeeded, 0 errored, 0 canceled
636814 done Thu May 2 21:00:00 UTC 2019 Thu May 2 21:00:00 UTC 2019 scheduler clean up local dns blobs Deleted 1 dns blob(s) created before 2019-05-02 20:00:00 +0000
636813 done Thu May 2 21:00:00 UTC 2019 Thu May 2 21:00:00 UTC 2019 scheduler clean up events Old events were deleted
636812 done Thu May 2 20:54:15 UTC 2019 Thu May 2 21:09:38 UTC 2019 pivotal-container-service-4a9d72b40a937240e75d service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e create deployment /deployments/service-instance_68fc28fa-6e2c-4a5b-bbd1-f15e71a50d5e
636811 done Thu May 2 20:44:05 UTC 2019 Thu May 2 20:44:11 UTC 2019 health_monitor pivotal-container-service-4a9d72b40a937240e75d scan and fix scan and fix complete
636810 done Thu May 2 20:44:05 UTC 2019 Thu May 2 20:44:12 UTC 2019 health_monitor service-instance_74e887b0-b7d5-42b3-9359-b9f1f8e577d4 scan and fix scan and fix complete
636809 done Thu May 2 20:44:05 UTC 2019 Thu May 2 20:44:12 UTC 2019 health_monitor service-instance_b18c7dea-4b8e-4bb8-8b66-0e29b91c1fc4 scan and fix scan and fix complete
636808 error Thu May 2 20:44:05 UTC 2019 Thu May 2 20:44:05 UTC 2019 health_monitor service-instance_eb48ea6f-3854-42b1-a99a-83222ce89a59 scan and fix Unable to get deployment lock, maybe a deployment is in progress. Try again later.
636807 error Thu May 2 20:44:04 UTC 2019 Thu May 2 20:44:04 UTC 2019 health_monitor service-instance_74e887b0-b7d5-42b3-9359-b9f1f8e577d4 scan and Unable to get deployment lock, maybe a deployment is in progress. Try again later.
- You see no locks present when you run the bosh locks command.
- You see repeated "Scan and Fix" tasks when you run the bosh tasks -ar command.
- Restarting the BOSH or Opsman VMs does not resolve the issue.
- During these repeated 'Scan and FIx' tasks, no new VM / instance is being deleted or created
Environment
VMware PKS 1.x
Resolution
From an SSH session to BOSH, run the following commands:
#sudo –i
#monit restart health_monitor
Once you have restarted the health_monitor run #monit summary to ensure the service is 'Running'
At this time run #bosh -ar and the repeated 'Scan and Fix' message should stop
If this does not resolve the issues you are having please gather the BOSH logs open an SR with VMware Technical Support as further investigation is needed
#sudo –i
#monit restart health_monitor
Once you have restarted the health_monitor run #monit summary to ensure the service is 'Running'
At this time run #bosh -ar and the repeated 'Scan and Fix' message should stop
If this does not resolve the issues you are having please gather the BOSH logs open an SR with VMware Technical Support as further investigation is needed
Feedback
Yes
No
Powered by
