VCF Operations upgrade from 9.1.0 to 9.1.1 fails at download_bundle_file stage due to network security restrictions

Products

• VCF Operations

Issue/Introduction

• VCF Operations upgrade from version 9.1.0 to 9.1.1 fails during the download phase. Build → Lifecycle → VCF Management → Tasks reports the following error:
  

  status: FAILED, Failed Stages: download_bundle_file, stage_bundle_file; ops.task.stage.failed

   

• The following exceptions are recorded in the /storage/log/vcops/log/casa/casa.log on the VCF Operations appliance, indicating a premature termination of the download stream because the expected content length is not reached:

  INFO  [FDSRestTemplate.download:231] - Starting download from
  https://<fleet_fqdn>/depot-service/content-gateway/PROD/COMP/VROPS/Operations-Upgrade-9.1.1.0.25679887.pak
  with stream handler
  WARN  [PakService.uploadFileForSlice:1336] - Failed to copy file, basename=reservedTemp_1011############3110;
  org.apache.hc.core5.http.ConnectionClosedException: Premature end of Content-Length delimited message body
  (expected: 75######75; received: 28######80)
  ERROR [FDSRestTemplate.lambda$download$1:253] - Handler failed while consuming stream
  from .../Operations-Upgrade-9.1.1.0.25679887.pak after 427254 ms:
  CasaLocalizableException: key=upgrade.pak.file_copy_failed;
  args=ConnectionClosedException: Premature end of Content-Length delimited message body
  (expected: 75######75; received: 28######80)
  org.springframework.web.client.ResourceAccessException: I/O error on GET request for
  "https://vcf-fleet.iad.ca.inet/depot-service/content-gateway/PROD/COMP/VROPS/Operations-Upgrade-9.1.1.0.25679887.pak":
  Handler failed while consuming stream
  at org.springframework.web.client.RestTemplate.createResourceAccessException(RestTemplate.java:926)
  ...
  Caused by: java.io.IOException: Handler failed while consuming stream
  ...
  Caused by: com.vmware.vcops.casa.exception.CasaLocalizableException: key=upgrade.pak.file_copy_failed;
  args=org.apache.hc.core5.http.ConnectionClosedException: Premature end of Content-Length delimited message body
  (expected: 75######75; received: 28######80)

• Execution of APUAT completes successfully without any issues.
• While trying to manually stage the file from the VCF Operations Admin UI by navigating to the Software update section, the upload completes successfully.
• Firewall logs explicitly show a resources-unavailable state during the file transfer.

Environment

• VCF Operations 9.1.x

Cause

The failure is caused by a network or security restriction enforced by an egress firewall or proxy. A hard fixed byte/size ceiling or content-inspection buffer limit on the egress path forcefully terminates the download stream.  

Resolution

To bypass the network restrictions / limitations:

1. Work with your Network/Security team to configure the egress firewall, proxy, or antivirus/DLP appliance to implement an exclusion or bypass rule for dl.broadcom.com.
2. Disable SSL decryption or content-inspection for the dl.broadcom.com domain to prevent the reassembly-buffer limit from truncating large file transfers.
3. Retry the VCF Operations upgrade once the network bypass rule is active and verified.

Additional Information

It is not recommended to upgrade VCF Operations directly from the Admin UI. The network bypass resolution above should be applied instead.