Reconciliation for migrated VM's in Aria Automation fails with error: Retry exhausted on reconciliation of migrated virtual machine!
search cancel

Reconciliation for migrated VM's in Aria Automation fails with error: Retry exhausted on reconciliation of migrated virtual machine!

book

Article ID: 403347

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Reconciliation fails with the following error:

migrationMessage

Retry exhausted on reconciliation of migrated virtual machine!

migrationStatus

FAILED

Errors in provisioning-service-app.log will show:

vCenter Prod: Retrieving associated compute state record failed with error java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: One of term, booleanClauses must be provided

Environment

Aria Automation 8.18.1 and later

Cause

Missing vCenter execution tokens will cause this issue.

Resolution

Validate the issue:

  1. SSH into the Aria Automation appliance (primary node).
  2. Connect to the provisioning-db: 
    • vracli dev psql provisioning-db
  3. Run the following sql statement to list all vSphere endpoints with their execution token.  If any of your managed vSpheres are not listed here then you are impacted by this issue: 
    • SELECT es.name, et.node_id FROM execution_token et JOIN endpoint_state es ON (et.key = es.document_self_link) WHERE et.key like '/resources/endpoints/%';
  4. Exit PSQL: 
    • \q

Resolve the issue:

  1. Backup your environment:
    • You must back up all VMware Aria Automation or Orchestrator appliances, at the same time - simultaneously for all nodes.
    • If you are making the snapshots manually, you must start the snapshots of the second and the third node not more than 40 seconds after you start the snapshots for the first node.
    • If the quiesced state was not achieved for all 3 nodes within the ~40 seconds time frame, the following strings will be found in the logs: "Freeze synchronization failed" and "Sync failed, making inconsistent snapshot".
    • Run the following command from one of the nodes to filter for all vmtoolsd messages: journal ctl --identifier=vmtoolsd
    • When you back up the VMware Aria Automation or Orchestrator appliance, disable in-memory snapshots and enable quiescing.
  2. SSH into the Aria Automation appliance (primary node).
  3. Edit the K8s provisioning-service-app deployment using vim: 
    1. kubectl -n prelude edit deploy provisioning-service-app
  4. Scroll to the section: 
    • - name: JAVA_OPTS
                value: |-
                  -Dhttps.proxyHost=$(JAVA_PROXY_HOST)
                  -Dhttp.proxyHost=$(JAVA_PROXY_HOST)
  5. Add the following options to the option list.  BE VERY CAREFUL WITH SPACES. DO NOT USE TABS. It MUST be aligned with other options: 
    • -Debs.connect.upstream.service.unavailability.threshold=250
      -Debs.connect.node.availability.check.initialDelay=P180D
      -Debs.connect.node.availability.check.interval=P180D 
  6. Example: 
    • - name: JAVA_OPTS
                value: |-
                  -Debs.connect.upstream.service.unavailability.threshold=250
                  -Debs.connect.node.availability.check.initialDelay=P180D
                  -Debs.connect.node.availability.check.interval=P180D
                  -Dhttps.proxyHost=$(JAVA_PROXY_HOST)
                  -Dhttp.proxyHost=$(JAVA_PROXY_HOST)
  7. Save the change by using ":wq". (To exit without saving, use ":q!".)
  8. Monitor the provisioning--service pod restart status: 
    • kubect -n prelude get pods | grep provisioning-service
  9. When provisioning-service pod(s) is ready, log into Aria Automation and navigate to Infrastructure,  Cloud Accounts and check the cloud account detail page of vSphere cloud accounts without execution token.
  10. Input the password for the cloud account, then click "validate" button to validate the cloud account. After validating, please click the "SAVE" button to save the cloud account.

 

To make the configuration above persistent after reboot or deploy.sh has been executed, follow the below steps.

1. SSH / PuTTy into one vRA virtual appliance in the cluster.


2. Run the following command to create custom profile.

vracli cluster exec -- bash -c 'base64 -d <<< IyBDcmVhdGUgY3VzdG9tIHByb2ZpbGUgZGlyZWN0b3J5Cm1rZGlyIC1wIC9ldGMvdm13YXJlLXByZWx1ZGUvcHJvZmlsZXMva2I0MDMzNDcvCiMgQ3JlYXRlIHRoZSByZXF1aXJlZCBkaXJlY3RvcnkgdHJlZSB0aGF0IHdpbGwgYmUgdXNlZCB3aGVuIHRoZSBwcm9maWxlIGlzIGFjdGl2ZQpta2RpciAtcCAvZXRjL3Ztd2FyZS1wcmVsdWRlL3Byb2ZpbGVzL2tiNDAzMzQ3L2hlbG0vcHJlbHVkZV9wcm92aXNpb25pbmctc2VydmljZS8KIyBDcmVhdGUgImNoZWNrIiBmaWxlIHRoYXQgaXMgYW4gZXhlY3V0YWJsZSBmaWxlIHJ1biBieSBkZXBsb3kgc2NyaXB0CmNhdCA8PEVPRiA+IC9ldGMvdm13YXJlLXByZWx1ZGUvcHJvZmlsZXMva2I0MDMzNDcvY2hlY2sKIyEvYmluL2Jhc2gKZXhpdCAwCkVPRgpjaG1vZCA3NTUgL2V0Yy92bXdhcmUtcHJlbHVkZS9wcm9maWxlcy9rYjQwMzM0Ny9jaGVjawojIENvcHkgcHJvdmlzaW9uaW5nLXNlcnZpY2UgY29uZmlndXJhdGlvbiBmaWxlIHRvIHRoZSBjdXN0b20gcHJvZmlsZQpjYXQgPDxFT0YgPiAvZXRjL3Ztd2FyZS1wcmVsdWRlL3Byb2ZpbGVzL2tiNDAzMzQ3L2hlbG0vcHJlbHVkZV9wcm92aXNpb25pbmctc2VydmljZS81MC1yZXNvdXJjZXMueWFtbApqYXZhT3B0czoKICAiZWJzLmNvbm5lY3QudXBzdHJlYW0uc2VydmljZS51bmF2YWlsYWJpbGl0eS50aHJlc2hvbGQiOiAiMjUwIgogICJlYnMuY29ubmVjdC5ub2RlLmF2YWlsYWJpbGl0eS5jaGVjay5pbml0aWFsRGVsYXkiOiAiUDE4MEQiCiAgImVicy5jb25uZWN0Lm5vZGUuYXZhaWxhYmlsaXR5LmNoZWNrLmludGVydmFsIjogIlAxODBEIgpFT0YKY2htb2QgNjQ0IC9ldGMvdm13YXJlLXByZWx1ZGUvcHJvZmlsZXMva2I0MDMzNDcvaGVsbS9wcmVsdWRlX3Byb3Zpc2lvbmluZy1zZXJ2aWNlLzUwLXJlc291cmNlcy55YW1s | bash -'


3. Redeploy vRA services by running

/opt/scripts/deploy.sh


4. Removing the custom profile (Optional):

vracli cluster exec -- bash -c 'rm -rf /etc/vmware-prelude/profiles/kb403347'


Note: If the configuration is no longer required, remove the custom profile with the command above and execute deploy.sh from step 3.

Powered by Wolken Software