Retry VCF 9.0 Installer workflow by modifying the deployment workflow JSON SPEC file

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

If the Deployment workflow initiated by VCF 9.0 Installer fails, there is no way to edit and correct the workflow configuration before attempting the retry workflow operation.

Below is the the reference workflow:

E.g. If vCenter server Machine SSL certs are replaced in between the deployment workflow execution, "Deploy and configure NSX" task of VCF 9.0 Installer will fail due thumbprint mismatch of vCenter server.

Failed Sub-Task in execution will reflect as : "Generate input for trusting certificates".

In this case, its necessary to update the deployment workflow with the correct certificate Thumbprint for successful execution.

Failed task error:

Could not establish trust with pre-existing product vCenterServer.domain.com. Provided SSL thumbprint: AD:3F:##:##:##:##:12:8B, actual SSL thumbprint: E6:2A:##:##:##:##:69:23 Reference Token: 5###E

Environment

VMware Cloud Foundation 9.x

Resolution

NOTE:

Depending on the workflow that is being attempted, modifying the workflow manually can have a significant impact to the environment. Please ensure that you have appropriate backups and snapshots of the components involved in the workflow.

Kindly reach out to Broadcom Support if any assistance is required to execute below Resolution Steps.

 

• Get the Failed TASK ID (UUID) using either of the below Methods:

1. VCF Installer UI:

Goto Developer Tools of the browser ( For Chrome Browser: Select Three Dots on extreme right top corner of the Tab -> Select 'More Tools' -> Select 'Developer Tools' -> Select 'Network' -> Select 'sddcs' under Name -> Select 'Preview' -> Note down the TASK 'id' against the elements  )

2. SDDC Domain Manager Logs:

Look for the Key word like "End of Orchestration with FAILURE for Execution ID" in the SDDC Domain Manager Logs for the respective TASK:

 /var/log/vmware/vcf/domain-manager/domainmanager.log

2##6-##-##T11:22:40.181+0000 DEBUG [vcf_dm,69####17e,2##d] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-15]  Contract task Generate Input for Trust Certificates input: {"trustLocally":true,"includeVcenter":false,"_executionUuid":"15####df-####-####-####-d0####63"}
    .
    .
    2##6-##-##T11:22:41.653+0000 INFO  [vcf_dm,69####17e,a##a] [c.v.e.s.o.core.OrchestratorImpl,dm-exec-8]  End of Orchestration for Execution ID 15####df-####-####-####-d0####63
    2##6-##-##T11:22:41.654+0000 INFO  [vcf_dm,69####17e,a##a] [c.v.e.s.o.co
    re.OrchestratorImpl,dm-exec-8]  End of Orchestration with FAILURE for Execution ID 15####df-####-####-####-d0####63

• Take a snapshot of SDDC Manager VM.

• SSH to the SDDC Manager with the 'vcf' user and then type 'su root'.
• Obtain an API TOKEN.

# TOKEN=$(curl -s --location --request POST 'http://localhost/v1/tokens' --header 'Content-Type: application/json' --data-raw '{"username":"admin@local", "password":"ENTER_PASSWORD"}'| jq -r '.accessToken')

• Validate the API TOKEN.

# echo $TOKEN

• Export the Failed deployment workflow specifications to a file (e.g. spec.json)

Note: <uuid> is the TASK ID of the failed workflow

# curl -H "Authorization: Bearer $TOKEN" localhost/v1/sddcs/<uuid>/spec | jq > spec.json 

• Modify the spec file (spec.json) only for the required fields. (Edit spec.json file using 'vi' command and save the file after making necessary changes using ":wq!" command)

Note: Below screenshot is specific to update the vCenter SSL thumbprint (vcenterSpec -> sslThumbprint). After updating the sslThumbprint , verify that all other fields are correct

• Retry the operation using the updated spec file via the API:

# curl -k -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" -X PATCH https://localhost/v1/sddcs/<uuid> -d "@spec.json"

• You can track the progress of the retried task through either the UI or the API:

# curl -H "Authorization: Bearer ${TOKEN}" localhost/v1/sddcs/<uuid>