Unable to Pull from vSphere Supervisor Content Library - "cannot authenticate SSL certificate for host wp-content.vmware.com"
search cancel

Unable to Pull from vSphere Supervisor Content Library - "cannot authenticate SSL certificate for host wp-content.vmware.com"

book

Article ID: 323442

calendar_today

Updated On:

Products

VMware vCenter Server Tanzu Kubernetes Runtime

Issue/Introduction

A Workload cluster upgrade becomes stuck, the creation of a workload cluster is stuck or a new node is stuck in Provisioning state.
 
The namespace for the affected workload cluster is subscribed to the VMware by Broadcom Kubernetes Content Library at wp-content.vmware.com.
 
 
From the vSphere web UI, one or more of the following tasks repeat whenever the desired content library item is fetched, where values in brackets <> will vary by environment:

Task Name Target Status
Deploy OVF package from Content Library to Resource Pool <affected workload cluster namespace> ⊗ A general system error occurred: Unexpected error
Fetch Content of a Library item* <OVF template for TKR> ⊗ A general system error occurred: The export of library item <item ID> has failed. Reason: Failed to retrieve library item source content for this download session
Deploy OVF Template <new VM name> ⊗ Failed to deploy OVF package. Cause: The operation is not allowed in the current state. Cannot instantiate library item <item ID> due to the failure of importing file <OVF file>, detail error: Error exporting file <OVF file>. Reason: HTTP request error: cannot authenticate SSL certificate for host wp-content.vmware.com.
Sync Library Item <OVF template for TKR> ⊗ A general system error occurred: HTTP request error: cannot authenticate SSL certificate for host wp-content.vmware.com

*The task for Fetch Content of a Library item may still show as Completed, but the "cannot authenticate SSL certificate" tasks will be present.

 

While connected to the Supervisor Cluster context, the following symptoms are observed:

  • Any new node using the above library item and TKR version will be stuck in Provisioning state:
    kubectl get machines -n <affected workload cluster namespace>


  • Performing a describe on the virtual machine (vm) for the new node shows the following error message:
    kubectl describe vm -n <affected workload cluster namespace>

Warning  CreateOrUpdateFailure  #s (x# over ##s)  vmware-system-vmop/vmware-system-vmop-controller-manager-<id>/virtualmachine-controller POST https://<vCenter FQDN>:443/rest/com/vmware/vcenter/ovf/library-item/id:<item ID>?~action=deploy: 500 Internal Server Error"

     

  • vmop-controller-manager system pod logs show one or more of the following error messages similar to the below:
    kubectl logs -n vmware-system-vmop <vmop-controller-manager pod name> -c manager

contentsource_controller.go:343] controllers/ContentSource "msg"="failed to difference images" "error"="Error exporting file <OVF file>. Reason: HTTP request error: cannot authenticate SSL certificate for host wp-content.vmware.com."

"CreateVirtualMachine failed" err="POST https://<vCenter FQDN>:443/rest/com/vmware/vcenter/ovf/library-item/id:<item ID>?~action=deploy: 500 Internal Server Error" logger="vsphere" vmName="<affected workload cluster namespace>/<new VM>"

1 virtualmachine_controller.go:243] "Failed to reconcile VirtualMachine" err="POST https://<vCenter FQDN>:443/rest/com/vmware/vcenter/ovf/library-item/id:<item ID>?~action=deploy: 500 Internal Server Error" logger="VirtualMachine" name="<affected workload cluster namespace>/<new VM>"

Environment

VMware vCenter Server

vSphere Supervisor

This issue can occur regardless of whether or not the affected workload cluster is managed by Tanzu Mission Control (TMC)

Cause

This is due to a mandatory certificate change pushed to the VMware by Broadcom Kubernetes Content Library at wp-content.vmware.com 

For any content libraries that are subscribed to the above content library, the SSL Certificate will no longer be trusted which prevents OVF files from being downloaded.
 
This issue does not affect TKRs that have already been downloaded and made available in the environment.
 
But this issue will occur when a new TKR needs to be fetched from the same content library which does not trust the new SSL certificate.

Resolution

All content libraries subscribed to the VMware by Broadcom Kubernetes Content Library at wp-content.vmware.com will need to have the new corresponding SSL certificate trusted.
 
  1. In the vSphere web UI, access the Content Libraries page from the Menu.



  2. Find the content library subscribed to the VMware by Broadcom Kubernetes Content Library, Right-click the library or click on Actions, and click Edit Settings:



  3. Click OK. No changes need to be made.
    After clicking OK, a red banner will appear noting the following: 

    "SSL certificate cannot be trusted. The thumbprint of the certificate is: <thumbprint> Do you want to proceed?"



  4. Click Actions to trust the new certificate, which will prompt you to trust the new certificate.
     

    In vSphere 8.0u3 and higher versions, a pop-up window will appear warning similarly:
    "Unable to verify the identity of the subscription host.

    The SSL thumbprint of the certificate is:
    <thumbprint>

    Connect anyway?
    Click Yes if you trust the subscription host. The SSL thumbprint of the certificate will be remembered until the library is deleted.

    Click No to cancel connecting to the subscription host at this time."


  5. Click OK once more to save the new SSL certificate thumbprint for the VMware by Broadcom Kubernetes Content Library.

    In vSphere 8.0u3 and higher versions, click YES to save the new SSL certificate thumbprint.


  6. The next fetch of the content library item for the desired TKR version will succeed.
    This may take up to 15 minutes.

Additional Information

 

Powered by Wolken Software