Retry SDDC Manager workflows by modifying workflow spec files

Products

• VMware Cloud Foundation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When an active SDDC Manager workflow fails due to an incorrect input configuration, SDDC Manager UI does not support editing the parameters before initiating a restart.
  
  
• A workflow may also fail if a password rotation occurs in the interim but the workflow retains the credentials provided at initiation, resulting in an authentication failure due to outdated information.
  
  
• The following errors appear in the /var/log/vmware/vcf/domainmanager/domainmanager.log file on the SDDC Manager:
  Cannot complete login due to incorrect credentials: vCenter_FQDN, [email protected].
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin)
YYYY-MM-SS:THH:MM:SS ERROR [vcf_dm,###############################] [c.v.e.s.c.c.v.vsphere.VcManagerBase,##-####-##]  Cannot complete login due to incorrect credentials: vCenter_FQDN, [email protected].
YYYY-MM-SS:THH:MM:SS ERROR [vcf_dm,###############################] [c.v.e.s.o.model.error.ErrorFactory,##-####-##]  [######] VSPHERE_HOST_EXIT_MAINTENANCE_FAILED Host cannot exit maintenance mode
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Host cannot exit maintenance mode
  
  
• This article details the procedure for extracting a workflow spec file, applying the necessary modifications, and re-injecting it into the failed workflow to allow a retry with the updated parameters.

Environment

VMware Cloud Foundation 5.x

VMware Cloud Foundation 9.x

Cause

This issue occurs when incorrect configuration inputs are supplied through either the JSON input spec or the SDDC Manager UI.

Resolution

1. Take a snapshot of SDDC Manager virtual machine.
   
   
2. Login to the SDDC Manager UI and click the failed task that needs to be retried.
   
   
3. Copy the workflow ID from the URL of the subtask.
   
4. SSH into the SDDC Manager as the vcf user, then elevate to root by typing su root.
   
   
5. Modify the application.properties file to allow curl access, followed by a service restart: 
   echo "vcf.vault.http-access=true" >> /etc/vmware/vcf/domainmanager/application.properties
systemctl restart domainmanager
   
   
6. Use the following GET API call to retrieve the spec file for the workflow:
   curl -k -s http://localhost/domainmanager/internal/vault/<workflow_id_from_url> -X GET | json_pp > /tmp/workflow.json
   
   
7. Update the workflow.json file to correct the input parameters that caused the workflow to fail.
   
   
8. Use the following PUT API call to update the spec file for the workflow:
   curl -k -s http://localhost/domainmanager/internal/vault/<workflow_id_from_url> -X PUT -H "Content-type: text/plain" -d @/tmp/workflow.json
   
   
9. From the SDDC Manager UI, restart the Task.
   
10. Once the task is complete, revert the changes made to the application.properties file and restart the service.
    sed -i '/vcf.vault.http-access=true/d' /etc/vmware/vcf/domainmanager/application.properties
systemctl restart domainmanager

Additional Information

Retrieve the service accounts credentials from SDDC Manager