VCF Interactive depot config failing at Step 4 - Token URL showing invalid

Products

• VMware SDDC Manager

Issue/Introduction

• While updating the authenticated download config using interactive method (Authenticated Download Configuration Update Script), issue is coming in Step 4. Script is showing below error:

DepotChange-MM-DD-YYYY.log

[##-##-####_##:##:##] (WARNING) Received "Unknown" error accessing "https://dl.broadcom.com/######################/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml".
[##-##-####_##:##:##] (WARNING) Full error message for URL "https://dl.broadcom.com/######################/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xmll" is No such host is known.
[##-##-####_##:##:##] (INFO) Checking new "vCenter Appliance Depot" URL ...
[##-##-####_##:##:##] (WARNING) Received "Unknown" error accessing "https://dl.broadcom.com/######################/PROD/COMP/VCENTER/vmw/##########-####-####-####-######################/8.0.3.00400/manifest/manifest-latest.xml".
[##-##-####_##:##:##] (WARNING) Full error message for URL "https://dl.broadcom.com/######################/PROD/COMP/VCENTER/vmw/###########-####-####-####-######################/8.0.3.00400/manifest/manifest-latest.xml" is No such host is known.
[##-##-####_##:##:##] (INFO) Beginning depot update operations...
[##-##-####_##:##:##] (INFO) Testing vCenter "###############.##.####.####.###" reachability from script execution system (PowerCLI timeout is configured as 300 seconds)...
[##-##-####_##:##:##] (INFO) "hostname.doamin.com" vCenter Appliance has been configured to use depot "https://dl.broadcom.com/######################/PROD/COMP/VCENTER/vmw/##########-####-####-####-######################/8.0.3.00400".
[##-##-####_##:##:##] (INFO) Please wait 5-10 minutes and check the vCenter Appliance for new updates.
[##-##-####_##:##:##] (INFO) Testing vCenter "###################.####.####.###" reachability from script execution system (PowerCLI timeout is configured as 300 seconds)...
[##-##-####_##:##:##] (INFO) Testing vCenter "###################.####.####.###" reachability from script execution system (PowerCLI timeout is configured as 300 seconds)...
[##-##-####_##:##:##S] (WARNING) The default ESX depot "VMware Certified Async Drivers for ESXi" has already been disabled on vCenter "###################.####.####.###", No changes are required.
[##-##-####_##:##:##] (WARNING) The default ESX depot "Download vSphere ESXi and ESX patches" has already been disabled on vCenter "###################.####.####.###", No changes are required.
[##-##-####_##:##:##] (WARNING) The default ESX depot "VMware Async Releases for VM-tools on ESXi" has already been disabled on vCenter "###################.####.####.###", No changes are required.
[##-##-####_##:##:##] (WARNING) The default ESX depot "Partner provided Addons for ESXi" has already been disabled on vCenter "###################.####.####.###", No changes are required.
[##-##-####_##:##:##] (ERROR) https://dl.broadcom.com/##########################/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml is invalid.  Please make sure your token is correct and re-try.
[##-##-####_##:##:##] (INFO) Testing vCenter "###################.####.####.###" reachability from script execution system (PowerCLI timeout is configured as 300 seconds)...
[##-##-####_##:##:##] (INFO) Beginning SDDC Manager "###################.####.####.###" depot configuration check...
[##-##-####_##:##:##] (INFO) Beginning SDDC Manager "###################.####.####.###" depot update...
[##-##-####_##:##:##] (INFO) Please wait while the SDDC Manager lcm service on "###################.####.####.###" is restarted...
[##-##-####_##:##:##] (INFO) The lcm service on SDDC Manager "3###################.####.####.###" has been restarted successfully.
[##-##-####_##:##:##] (INFO) SDDC Manager "###################.####.####.###" has been successfully updated with the new depot location "dl.broadcom.com/#######################/PROD".
[##-##-####_##:##:##] (WARNING) SDDC Manager "###################.####.####.###" does not require an update to the operations manager service.
[##-##-####_##:##:##] (INFO) Please wait 5-10 minutes and then you may validate that you can download VCF packages using the new depot.

 

• All the activities are performed from the SDDC Manager side (not from the vCenter), however, the following errors are present in the vCenter:

var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-111.log

####-##-##T##:##:##Z error vmware-vum-server[######] [Originator@####sub=DownloadMgr opID=#######-####-####-####-##############] [downloadMgr ###] Executing download job {##############} throws error: curl_easy_perform() failed: cURL Error: Couldn't resolve host name, Could not resolve host: dl.broadcom.com
####-##-##T##:##:##Z info vmware-vum-server[######] [Originator@####sub=DownloadMgr opID=#######-####-####-####-##############] [downloadMgr ###] Download failed but destination file /tmp/******** exists and is valid. Ignoring error
####-##-##T##:##:##Z info vmware-vum-server[######] [Originator@###sub=DownloadMgr opID=#######-####-####-####-##############] [downloadMgr ###] Download job {###############} finished, bytes downloaded = 0
####-##-##T##:##:##Z verbose vmware-vum-server[#######] [Originator@####sub=DownloadMgr opID=#######-####-####-####-##############] [downloadMgr ###] Download job {#############} finished
####-##-##T##:##:##Z verbose vmware-vum-server[#######] [Originator@####sub=DownloadMgr opID=#######-####-####-####-##############] [downloadMgr ###] Removing download job {###########} in queue
####-##-##T##:##:##Z verbose vmware-vum-server[#######] [Originator@####sub=DownloadMgr opID=#######-####-####-####-##############] [downloadMgr ###] Current download count: 0
####-##-##T##:##:##Z error vmware-vum-server[#######] [Originator@####sub=Default opID=#######-####-####-####-##############] [updateDownloaderImpl ###] File download error: Downloading file: https://dl.broadcom.com/################/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml failed, 0 byte downloaded.
####-##-##T##:##:##Z error vmware-vum-server[#######] [Originator@####sub=VumVapi::Utils opID=#] [DepotsUtil 1210] Url https://dl.broadcom.com/##########################/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml is not valid.
####-##-##T##:##:##Z info vmware-vum-server[#######] [Originator@####sub=Telemetry opID=#######-####-####-####-##############] [TelemetryManager ###] New item added to telemetry data queue. Queue size: 1
####-##-##T##:##:##Z info vmware-vum-server[#######] [Originator@####sub=Telemetry opID=#######-####-####-####-##############] [TelemetryManager ###] New item added to telemetry data queue. Queue size: 2
####-##-##T##:##:##Z error vmware-vum-server[#######] [Originator@####sub=VumVapiAsyncTelemetryFilter opID=#######-####-####-####-##############] [TelemetryVapiProvider ##] Method com.vmware.esx.settings.depots.online.create failed with: Error:
--> com.vmware.vapi.std.errors.invalid_argument
--> Messages:
--> com.vmware.vcIntegrity.lifecycle.depots.online.Invalid<Online Depot URL 'https://dl.broadcom.com/#####################/PROD/COMP/ESX_HOST/addon-main/vmw-depot-index.xml' is not valid or cannot be reached now.>
-->
####-##-##T##:##:##Z verbose vmware-vum-server[#####] [Originator@####sub=vmomi.soapStub[#####]] Using VMODL version; internallookup/4.0(#######-####-####-####-##############n4_0), <<<cs p:00007f49900491f0, TCP:#######-####-####-####-##############:443> >, /lookupservice/sdk>
####-##-##T##:##:##Z info vmware-vum-server[#####] [Originator@#####sub=VcIntegrity] Adapter Created for server:https://#######-####-####-####-##############:443, service:/lookupservice/sdk on namespace:internallookup/4.0
-->
####-##-##T##:##:##Z verbose vmware-vum-server[######] [Originator@6876 sub=vmomi.soapStub[#####]] Initial service state request failed, disabling pings; /lookupservice/sdk, <last binding: <<TCP '###.#.#.# : #####'>, <TCP '###.#.#.# : 443'>> >, HTTP Status:404 'Not Found'
####-##-##T##:##:##Z info vmware-vum-server[11452] [Originator@####sub=LookupServiceClient] [lookupServiceClient ###] Found endpoint URL: https://#/sts/STSService/######.####### for Product: ###.######.###, Type: ##.#############, EndPointType: #######-####-####-####-##############
####-##-##T##:##:##Z verbose vmware-vum-server[#####] [Originator@####sub=vmomi.soapStub[#####]] Resetting stub adapter; <<last binding: <<TCP '###.#.#.# : #####'>, <TCP '###.#.#.# : 443'>> >, /lookupservice/sdk>, (null)
####-##-##T##:##:##Z info vmware-vum-server[#####] [Originator@#### sub=StsClient] [ssoUtils ##] Creating STS instance for endpoint: https://#######-####-####-####-##############/sts/STSService/#######.#######
####-##-##T##:##:##Z info vmware-vum-server[#####] [Originator@#### sub=SsoClient] Successfully acquired token: SamlToken [subject={Name: vpxd-extension-#######-####-####-####-##############; Domain:#####.######}, groups=[{Name: Users; Domain:#############.##########}, {Name: #############; Domain:##############.############}, {Name: ###############.#############; Domain:#############.#########}, {Name: ***********; Domain:##########.#########}, {Name: ##########.########; Domain:##############.##########}, {Name: ############.##########s; Domain:##########.##########}, {Name: ##########.##########; Domain:############.#############}, {Name: ###########.##########; Domain:##########.##########}, {Name: ##########.##########; Domain:##########.##########}, {Name: Everyone; Domain:********.******}], delegationChain=[], startTime=##########.##########, endTime=##########.########## , renewCount=0, delegableCount=10, isSolution=true, type=Saml_HOK]
####-##-##T##:##:##Z info vmware-vum-server[#####] [Originator@####sub=Telemetry] [TelemetryManager ###] Sending telemetry data: {"@type":"pman_task","taskId":"#######-####-####-####-##############","opId":"","succeeded":false,"entityId":"#######-####-####-####-##############"|","taskName":"com.vmware.esx.settings.depots.online.create","startTime":"######.#########","endTime":"#####.######"}
####-##-##T##:##:##Z info vmware-vum-server[#####] [Originator@####sub=Telemetry] [TelemetryManager ###] Telemetry data sent to prod endpoint
####-##-##T##:##:##Z verbose vmware-vum-server[#####] [Originator@####sub=vmomi.soapStub[####]] Using VMODL version; internallookup/4.0(lookup.version.version4_0), <<<cs p:00007f499000feb0, TCP:#######-####-####-####-##############:443> >, /lookupservice/sdk>
####-##-##T##:##:##Z info vmware-vum-server[#####] [Originator@####sub=VcIntegrity] Adapter Created for server:https://#######-####-####-####-##############:443, service:/lookupservice/sdk on namespace:internallookup/4.0

Environment

SDDC Manager 5.2.1.0

Cause

The script is designed to update depot configurations for both the SDDC Manager Lifecycle Manager (LCM) and the vCenter Servers (VCs) managed within the VCF environment. When executed with VCF (SDDC Manager) as the target, the script validates connectivity to the new depot URLs from both the SDDC Manager and each managed vCenter.

Since the vCenter does not have internet access while the SDDC Manager does, the connectivity check from the vCenter to the depot URL using the token fails. This results in an expected error during script execution.

Future releases will include improvements to error messages and logs for better clarity in scenarios where the SDDC Manager has internet access but the vCenter does not.

Resolution

No further action is needed. If you just continue with the remaining steps mentioned in the interactive method (Authenticated Download Configuration Update Script), the depot update goes through successfully via SDDC Manager. 

Note:

• Please note that the /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties file is checked, and all relevant properties appear to be updated correctly after using the interactive method.

cd /opt/vmware/vcf/lcm/lcm-app/conf/

less application-prod.properties

lcm.depot.adapter.host=dl.broadcom.com
lcm.depot.adapter.remote.rootDir=/<downloadToken>/PROD
lcm.depot.adapter.remote.repoDir=/COMP/SDDC_MANAGER_VCF
lcm.depot.adapter.remote.lcmManifestDir=/COMP/SDDC_MANAGER_VCF/lcm/manifest
lcm.depot.adapter.remote.lcmProductVersionCatalogDir=/COMP/SDDC_MANAGER_VCF/lcm/productVersionCatalog

• Additionally, the application properties are validated, and all values, including the URL, are confirmed to be updated accurately.

cd /etc/vmware/vcf/operationsmanager/

less application.properties

proxy.configuration.validation.expected.http.statuses=401,403,404
proxy.configuration.validation.test.url=https://dl.broadcom.com

Additional Information

VCF authenticated downloads configuration update instructions