VCF 9.1 Upgrade failure during "Import VCF Operations in Fleet Lifecycle" due to SSL thumbprint mismatch

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

During an upgrade to VMware Cloud Foundation (VCF) 9.1, the upgrade may fail during the Import VCF Operations in Fleet Lifecycle task. This issue blocks the registration of management components and prevents the upgrade from proceeding.

• The task Deploy VCF Management Components fails at the sub-task Import VCF Operations in Fleet Lifecycle.
• Error message  The Fleet lifecycle 'Component import' task with ID <TASK_ID> failed. Internal errors: Unable to register components with VCF Operations. Body: {"type":"Error","message":"The provided username/password or token is not valid. Please try again.","httpStatusCode":401,"apiErrorCode":401} Provided SSL thumbprint '<OLD_THUMBPRINT>' does not match the certificate's thumbprint '<NEW_THUMBPRINT>'.

Environment

VMware Cloud Foundation 9.1.0.0

Upgrading from VCF 9.0.1 to 9.1.

Cause

This is likely caused by a stale SSL thumbprint or expired certificate anchor stored in the SDDC Manager internal vault. When the sddc-build-service attempts to authenticate with the VCF Operations appliance using the stored (stale) thumbprint, the connection is rejected, resulting in an HTTP 401 Unauthorized error.

Resolution

To resolve this issue, you must manually update the SSL thumbprint and certificate fields in the SDDC Manager internal Vault to match the current VCF Operations certificate.

Prerequisites

• Take a snapshot of the SDDC Manager VM.
• Identify the Execution ID of the failed deployment task. This can be gathered from the error log entry(e.g., 6dccc949-5dc6-477b-9c48-c3d73e9597f9).

Steps to Resolve

1. Extract the Current Workflow Specification SSH into the SDDC Manager and elevate to root. Use the following command to retrieve the current JSON specification from the Vault:

   curl -k -s http://localhost/domainmanager/internal/vault/<EXECUTION_ID> -X GET | jq . > /tmp/workflow.json

2. Retrieve the Current SSL Thumbprint Obtain the correct SHA-256 thumbprint from the active VCF Operations appliance:

   openssl s_client -connect <vcf_ops_fqdn>:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -sha256

3. Update the JSON Specification Edit /tmp/workflow.json and locate the vcfOperationsSpec and vcfOperationsCollectorSpec sections. Update the following fields with the current thumbprint and certificate information:

   • sslThumbprint
   • certificate

4. Push the Corrected Specification to the Vault Upload the modified JSON file back to the SDDC Manager Vault:

   curl -k -s http://localhost/domainmanager/internal/vault/<EXECUTION_ID> -X PUT -H "Content-type: text/plain" -d @/tmp/workflow.json

5. Restart SDDC Manager Services Restart the services to ensure the new trust anchor is utilized:

   systemctl restart operations-manager

6. Retry the Failed Task In the SDDC Manager UI, navigate to the failed task and click Retry.

Additional Information

Retry VCF 9.0 Installer workflow by modifying the deployment workflow JSON SPEC file