vCenter patching fails with "Error copying packages" and HTTP 403 Forbidden

Products

• VMware vCenter Server

Issue/Introduction

• When attempting to patch vCenter within a major release, the staging of the software is interrupted and the message "Error copying packages" appears in a red alert bar.

• Further investigation into the the vCenter log /var/log/vmware/applmgmt/update_microservice.log shows the following message:

YYYY-MM-DD HH:MM:SS,SSS - 1831823 - update_functions_target::      runCommandAndCheckResult: 428 -    DEBUG - runCommandAndCheckResult failed: '--YYYY-MM-DD HH:MM:SS--  https://dl.broadcom.com/############################/PROD/COMP/VCENTER/vmw/hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh/8.0.3.00600/package-pool/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.blob\nResolving dl.broadcom.com... ###.###.###.###, ###.###.###.###, 2606:4700:7::a5, ...\nConnecting to dl.broadcom.com|###.###.###.###|:443... connected.\nHTTP request sent, awaiting response... \n  HTTP/1.1 403 Forbidden\n  Date: Day, DD Mon YYYY HH:MM:SS GMT\n  Content-Type: text/plain;charset=UTF-8\n  Content-Length: 12\n  Connection: keep-alive\n  CF-RAY: hhhhhhhhhhhhhhhh-IAD\n  Vary: Accept-Encoding\n  Strict-Transport-Security: max-age=31536000; includeSubDomains\n  Server: cloudflare\nYYYY-MM-DD HH:MM:SS ERROR 403: Forbidden.\n\n'

• Repeated attempts fail the same way

Environment

VMware vCenter Server

Cause

• Invalid or incorrectly formatted download token. In this case the token is invalid due to an issue between the self-generation of the token and the product entitlement.
• There is a very similar case covered under "Error when updating vCenter in vLCM "Error copying packages""

Resolution

• Open a case with GCA, please refer to the guide "Create a Non Technical case for Support Portal / License / Contract related issues".
• For further details, consult the  "VCF Authenticated Downloads Token Troubleshooting Guide" 

Additional Information

• A similar error can happen when updating ESXi, see ""Cannot download VIB" error while staging the ESXi patch using lifecycle manager with Token based URL - Token Not Entitled" for more information.
• The same issue with an invalid Download Token can produce a different error in the VAMI interface during the staging phase as seen below:

vCenter Server Update Fails with "[errno 2] No such file or directory: '/storage/updatemgr/software-update-stage/update/patching_backup_config.json'.
"Error copying packages" により vCenter へのパッチ適用が失敗する