Tanzu Hub Foundation vending failure at "Deploying Foundation Core from OVF" step

Products

• VMware Tanzu Platform - Hub

Issue/Introduction

• When attempting to deploy a new foundation from Tanzu Hub using the Foundation Vending workflow, it fails on step:

"Deploying Foundation Core from OVF” with the message “java.io.IOException: HTTP Error: 404 NOT_FOUND Not Found"

• The graphql-rest-provider-service debug logging reports failures with OVF upload:
  
  

  22:00:34.594Z [thread='boundedElastic-5' user='' org='' trace='697bd8286ce65a8d32f1b9a8e6c084e8'] ERROR com.vmware.ensemble.rest.fleet.vending.activities.impls.AbstractDeployOpsManagerActivity - Exception occurred during OVA upload. Aborting lease.
  reactor.core.Exceptions$ReactiveException: java.io.IOException: HTTP Error: 404 NOT_FOUND Not Found
  
  
  

• Specifically on the NFC request sent to the vSphere ESXi host:
  

  Error has been observed at the following site(s):
        *__checkpoint ⇢ 404 NOT_FOUND from POST https://esxi.domain.com/nfc/########-bb80-####-####-6d73########/disk-0.vmdk [DefaultWebClient]
  
  

Environment

Encountered on Tanzu Hub 10.3.4

Cause

In order for Foundation Vending to create a new Opsman VM, direct connectivity is required on port 443 to the destination vCenter and ESXi host in order to deploy the OVF file. If network access to the ESXi host on port 443 is not available from the source Hub deployment, users will encounter the above failure during the OVA upload stage.

Resolution

Ensure the destination vCenter and ESXi hosts are available via network on port 443 from the source Hub installation.

 

To assist with investigation:

 

1. Enable debug logging on the graphql-rest-provider-services pod:
   • Use bosh ssh to connect to the Hub registry/0 VM:
     
     bosh -d hub-<DEPLOYMENT_ID> ssh registry/0
     
     
     
   • Pause the ensemble-helm and sm package installs to allow modification of the deployment:
     
     

     kctrl package installed pause -n tanzusm -i sm 

     kctrl package installed pause -n tanzusm -i ensemble-helm

   • Edit graphql-rest-provider-service deployment to change ENSEMLE_LOG_LEVEL to DEBUG:
     
     kubectl edit deployment/graphql-rest-provider-service -n tanzusm
     
     
     NOTE: This will restart the graphql-rest-provider-services pod
     
     
2. Attempt the Foundation Vending once more to generate new logs.
3. Review the graphql-rest-provider-service pod log for further details to help isolate where the download is failing.