Failed bringup via the Cloud Builder - Process to restart the workflow via API

Products

• VMware Cloud Foundation

Issue/Introduction

• We have a failed bring-up process via the Cloud Builder appliance.
• We need to make changes to the bringup .xlsx file, and resume the failed bring-up process, without restarting the entire process from the start.
• This article describes the process to use a modified/edited bring-up xlsx file to create a new bring-up json file, and restart the bringup workflow, through the Cloud Builder appliance.

Environment

SDDC Manager  VMware Cloud Manager,

Cause

The cause for failure of the bring-up workflow will be highly varied depending upon the environment and the input parameters.

Resolution

Note: For VCF 5.1, please use the full path /usr/pgsql/13/bin/psql to invoke the psql command.

1. Modify the Bringup XLS with whatever changes need to be made.
2. Copy Bringup XLS to the Cloud Builder VM (using a utility like WinSCP, Filezilla etc) to the /home/admin directory.
3. Login to cloud builder via SSH with admin credentials and switch user (using su) to root.
4. Convert XLS input to JSON.
   • VCF:

     /opt/vmware/sddc-support/sos --jsongenerator --jsongenerator-input <file_name.xlsx> --jsongenerator-design vcf-public-ems

   • VCF with VxRAIL:

     /opt/vmware/sddc-support/sos --jsongenerator --jsongenerator-input <file_name.xlsx> --jsongenerator-design vcf-vxrail

     
     The JSON file will be generated in:
     • For VCF:

       /opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-ems.json

     • For VCF on VxRAIL:

       /opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-vxrail/vcf-vxrail.json

5. Copy the generated JSON file to the current path.

   cp /opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-vxrail/vcf-vxrail.json /home/admin/

6. Get the Bring-up Execution ID.

   psql -h localhost -U postgres -d bringup -c "select id from execution where name like 'Bringup%';"

   
   We can also find the Execution ID in the vcf-bringup-debug.log located at:

   /opt/vmware/bringup/logs/

7. Run the API to Retry the workflow with the same Execution ID:

   curl -k -u admin:<password> -X PATCH https://localhost/v1/sddcs/<execution_id> -H "Content-Type: application/json" -d@/home/admin/vcf-vxrail.json

   
   This will restart the Bring-up process, and we can track the progress via the Cloud Builder UI.

 

If the workflow failed AFTER the SDDC Manager was already deployed, please run the following additional steps.

 

If the SDDC Manager Virtual Machine is already deployed, it contains the original basic auth credentials configured at NGINX.
Once bring-up process is restarted using PATCH API, it re-generates the new basic auth credentials and bring-up process uses these new credentials while calling the SDDC Manager APIs (I.e https://<sddc_fqdn>/lcm/about).

Since the bring-up service uses the newer basic auth credentials (Generated randomly on bring-up retry), the authentication with SDDC API fails, because NGINX contains the older original basic auth credentials. This results in '401 Unauthorized" errors.

To get around this, run through the following steps:

1. From the existing SSH session to the Cloud Builder Virtual Machine, export the DB to make it easier to search:

   pg_dump --data-only --inserts -U postgres -h localhost -d bringup > /tmp/db.db

2. Search for the new randomly generated password:

   cat /tmp/db.db | grep -i basicAuthPassword

   
   Sample

   "admin", "basicAuthPassword":"wE###########WM"

   
   [ Do not pipe the output to less, the password will be closer to the bottom of the JSON body, and is easier to look for without using less ]
   
   
3. SSH to the SDDC Manager Virtual Machine as vcf and then root, and execute the below script to update the basic auth password in the nginx htpasswd file:

   python /opt/vmware/vcf/commonsvcs/scripts/auth/setbasicauthpassword.py admin `openssl passwd -apr1 '<Password>'`

   Note:
   1. The openssl command should be used in punctuation marks, i.e. ` `, and not single quotes, i.e. ' '
   2. The basic auth password to be updated should be used in single quotes and not double quotes
      
      
4. In order to ensure that the updated basic auth password is working, SSH to the Cloud Builder Virtual Machine and execute the below command:

   curl -X GET https://<SDDC_Manager_IP_or_FQDN>/lcm/about -u 'admin:<BASIC_AUTH_PASSWORD_FROM_ABOVE>'

   
   Example:

   curl -X GET https://sddc.example.com/lcm/about -u 'admin:E###########WM'

5. Now login to the cloud builder and retry bringup.
6. The workflow should continue successfully at this point (assuming all configuration is provided correctly in the bring-up xlsx files).

Additional Information

Impact/Risks:
MINIMAL: We are going to be attempting a re-run of the workflow using the same bring-up xlsx with only minor changes as required to resolve the errors in the workflow. A snapshot of the cloud builder appliance is recommended, but not required.

NOTE: If the workflow failed after the SDDC Manager VM was already deployed, additional steps need to be performed, which are time-sensitive (i.e need to be performed within 5 minutes of restarting the workflow). Please read this article and the resolution steps in its entirety before attempting them.