Restarting stuck image synchronization in VMware Aria Automation

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

How to force an image synchronization restart.

Symptoms:

• Synchronizing images has been stuck for more than 40 minutes.
• VMware vCenter is healthy.

Environment

VMware Aria Automation 8.x

Cause

Storage outages or an unhealthy vCenter may lead to this issue.

Resolution

VMware is aware of this issue. See the Workaround below for additional information.

Workaround:

1. Validate that vCenter is healthy: Clone a virtual machine from the same template you are trying to provision from.
   1. If this operation fails or generates an error, resolve this issue in vCenter then move to Step #2.
2. If you are working with VMware Aria Automation 8.x (On-Premises) restart the provisioning service pod.

   kubectl delete pod -n prelude provisioning-service-pod-name

        Note: You can run the following command to monitor the status of provisioning-service-app pod(s) until all provisioning-service-app pod(s) are up and running:
        watch 'kubectl -n prelude get pods | grep provisioning-service-app'
     CTRL+C will end the watch command after they are all showing as Running 1/1.

3. If restarting the service does not alleviate the issue, you may use the following API to mark the image synchronization as FAILED, then, VMware Aria Automation will retry a new image synchronization.

curl --location --request PATCH 'https://<FQDN>/iaas/api/cloud-accounts/<cloud_account_id>?apiVersion=2021-07-15' \
--header 'Authorization: Bearer <access_token>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "customProperties": {
    "imageEnumerationTaskState": "FAILED"
  },
  "privateKey":"<username_password>",
  "privateKeyId":"<username>"
}'

Note:

• <FQDN>: Is the VMware Aria Automation FQDN on premises.
• <access_token>: Generated following the next steps:
  • ​​​​​​​VMware Aria Automation 8.x: Follow Get Your Access Token for the VMware Aria Automation API.
• <cloud_account_id>,<username>: May be obtained from the browser. Otherwise by running the API GET /iaas/api/cloud-accounts

Additional Information

Impact/Risks:
For on-premises deployments, restarting the provisioning-service will fail any deployment in progress.

 

In case of “Service not found” errors in provisioning-service-app.log, restart kube-system and prelude pods by performing commands:

• kubectl -n kube-system delete pod --all (then wait for pods to restart)
• Then run, /opt/scripts/deploy.sh to restart all prelude pods.