Aria Automation upgrade fails in Aria Suite Lifecycle with package download error
search cancel

Aria Automation upgrade fails in Aria Suite Lifecycle with package download error

book

Article ID: 345969

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:

  • Aria Automation upgrade fails.

  • You see the following messages in the vami.log located in /opt/vmware/var/log/vami/

10/01/2023 15:57:39 [DEBUG] Open URL complete: https://<vrslcm-fqdn>/repo/productBinariesRepo/vra/8.10.1/upgrade/update/package-pool/prelude-layer-9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0-1-1.noarch.rpm
10/01/2023 15:57:39 [ERROR] Invalid file size. url=/package-pool/prelude-layer-9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0-1-1.noarch.rpm, downloadsize=1842309062, filesize=2905836008
10/01/2023 15:57:39 [DEBUG] Deleting file: /tmp/vmware_update_download_wXXXXX
10/01/2023 15:57:39 [ERROR] Setting job error information. jobid=20, errorCode=9, errorString=Error during package download. Please try again.

  • Other messages can be similar to:

03/02/2023 22:06:01 [ERROR] CURL error: transfer closed with outstanding read data remaining URL: https://<vrslcm-fqdn>/repo/productBinariesRepo/vra/8.9.1/upgrade/update/package-pool/prelude-layer-9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0-1-1.noarch.rpm
03/02/2023 22:06:01 [ERROR] Sleep 10 seconds and retry.
03/02/2023 22:07:23 [ERROR] CURL error: transfer closed with outstanding read data remaining URL: https://<vrslcm-fqdn>/repo/productBinariesRepo/vra/8.9.1/upgrade/update/package-pool/prelude-layer-9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0-1-1.noarch.rpm

Note: You can also encounter issue while downloading Aria Automation support bundle from Aria Suite Lifecycle where support bundle downloaded is either corrupted or downloaded partially.

 

Environment

Aria Automation 8.X

Cause

This issue has been observed on the network where there is slow bandwidth. The download of RPM files from Aria Suite Lifecycle to Aria Automation takes time to download and finally the connection is closed by Aria Suite Lifecycle . Due to this, there is partial download failure in Aria Automation.

The download of RPM's from Aria Suite Lifecycle is done over nginx proxy service which runs inside Aria Suite Lifecycle. With existing nginx configuration settings, it does not work on slow network. If the rate at which data is transferred to Aria Automation is more than the rate at which data is processed by Aria Automation, the connection is closed by Aria Suite Lifecycle.

Resolution

We need to add additional configuration settings in Nginx configuration file to make it work on a slow network. Follow the below steps:

  1. Take a snapshot of Aria Suite Lifecycle appliance.
  2. SSH to Aria Suite Lifecycle appliance.
  3. Edit this file: /etc/nginx/nginx.conf
  4. Replace this section:
location /repo/ {
      proxy_pass;
    }
with this:
location /repo/ {
      proxy_pass;
      proxy_max_temp_file_size 4096m;
    }
  1. Save the file and verify configurations are correct using the command:
nginx -t

It should give the output as below:

root@vrslcm [ /etc/nginx ]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
  1. Restart the nginx service using the command:
systemctl restart nginx

 

Even after making the above changes, in some environments where network bandwidth is very slow, the download of RPM may still fail with the following error in /var/log/nginx/error.log in Aria Suite Lifecycle appliance:


2022/10/13 09:52:56 [crit] 905#0: *11090 pwritev() "/etc/nginx/proxy_temp/7/52/0000000527" failed (28: No space left on device) while reading upstream, client:<IP-Address>, server: <vrslcm-fqdn>, request: "GET /repo/productBinariesRepo/vra/8.8.2/upgrade/update/package-pool/prelude-layer-9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0-1-1.noarch.rpm HTTP/1.1", upstream: "http://X.X.X.X:8080/repo/productBinariesRepo/vra/8.8.2/upgrade/update/package-pool/prelude-layer-9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0-1-1.noarch.rpm", host: "<vrslcm-fqdn>"

The default location for storing buffer data for nginx is /etc/nginx/proxy_temp which resides inside / partition which is of size 8GB in Aria Suite Lifecycle. There is no documented way to extend this partition. Hence, we can override this default buffer location by following below steps:

  1. Take snapshot of Aria Suite Lifecycle appliance.
  2. SSH to vRSLCM appliance.
  3. Edit this file: /etc/nginx/nginx.conf
  4. Replace this section:

location /repo/ {
      proxy_pass;
    }

with this:

location /repo/ {
      proxy_pass;
      proxy_max_temp_file_size 4096m;
      proxy_temp_path /data/temp/;
    }

  1. Save the file and verify configurations are correct using the command:
nginx -t

It should give the output as below:

root@vrslcm [ /etc/nginx ]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
  1. Restart the nginx service using the command:
systemctl restart nginx

As part of the steps mentioned above, we have just added this new entry in the /etc/nginx/nginx.conf file : "proxy_temp_path /data/temp/;"

After performing the above steps, you can retry Aria Automation upgrade from Aria Suite Lifecycle.

 

Powered by Wolken Software