vCenter Server online updates via VAMI fail due to lack of specific product version entitlements in the token

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• When attempting to patch the vCenter Server via the vCenter Server Appliance Management Interface (VAMI), the operation fails and displays one of the following errors. This issue is observed after updating the token as instructed in Broadcom KB VCF authenticated downloads configuration update instructions:
  
  
  • Error copying packages:
    
    
    
  • Update is not staged:
    
    
    
  • [Errno 2] No such file or directory: '/storage/updatemgr/software-update######/stage/update/patching_backup_config.json':
    
    
    
  • Error in method invocation Timeout happens while sending message to microservice:
    
    
    
  • [Errno 2] No such file or directory: '/storage/core/software-update/stage/stageDir.json':
    
    
• The /var/log/vmware/applmgmt/update_microservice.log file on the vCenter Server contains errors similar to the following:
  YYYY-MM-DD HH:MM:SS - 23088 - update_functions target ::-- waitretry 1 -- connect-timeout 10 -r -np -nH -nd -A -P /storage/seat/software-update4ftl1ri/stage https://dl.broadcom.com/#####################/PROD/COMP/VCENTER/vmw/#####################/8.0.3.00600/package-pool/010cbb2b1373b111b91e0d827a4423df0499fe206404022b92ca81add4eae3d7.blob
YYYY-MM-DD HH:MM:SS - 23088 - update_functions target :::35:32 -- https://dl.broadcom. com/#####################/PROD/COMP/VCENTER/vmw/#####################/8.0.3.00600/package-pool/010cb2b1373b111b91e0d827a4423df0499fe206404022b92ca81add4eae3d7.blob\nResolving dl.broadcom.com ... , [IP_ADDRESS], [IPV6_ADDRESS]\n Connecting to dl.broadcom.com|[IP_ADDRESS]| :443 ... connected. \nHTTP request sent, awaiting response ... \n HTTP/1.1 403 Forbidden\n Date: YYYY-MM-DD HH:MM:SS GMT\n Content-Type: text/plain; charset=UTF-8\n Content-Length: 12\n Connection: keep-alive\nServer: cloudflare\n YYYY-MM-DD HH:MM:SS ERROR 403: Forbidden. \n\n'ERROR - Exception staging containers 

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

• The provided token does not have the necessary entitlements for the selected version.
• The download token was generated for the wrong Site ID.
• The token has expired.

Resolution

Clear the update state by following these steps, and then attempt the update again with a valid token.

Note: Ensure there is a valid vCenter Server virtual machine snapshot before executing the steps. For snapshot best practices, refer to: Snapshot Best practices for vCenter Server Virtual Machines.

1. Stop the applmgmt service
   service-control --stop applmgmt
   
   
2. Delete the patching db:
   rm /storage/db/patching.db
   
   
3. Verify that the file /storage/core/software-update/stage/stageDir.json exists, then check its contents:
   cat /storage/core/software-update/stage/stageDir.json
   
   
4. If the file exists, it will specify a stageDir location similar to /storage/<subdir>/software-update####. Delete this directory and its contents by running the following command:
   rm -rf /storage/<subdir>/software-update####
rm -rf /storage/core/software-update/stage/stageDir.json
   
   
5. Clean up the state file.
   rm /etc/applmgmt/appliance/software_update_state.conf
   
   
6. Start the applmgmt service
service-control --start applmgmt
   
   
7. Configure a valid token based on the entitlement. Refer to: VCF authenticated downloads configuration update instructions.
   
   
8. Retry the vCenter Server patching.

Alternatively, mounting the vCenter Server patch ISO to the virtual machine allows for patching without a token. For more information, see: Patching/Updating vCenter Server through vCenter Server Appliance Management Interface (VAMI).

Additional Information

• This issue can also occur if a Download Token is generated using an incorrect Site ID.
• The vCenter Server license must be tied to the same Site ID used to generate the Download Token.