Aria Orchestrator patching process loops indefinitely

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When applying a patch to Aria Orchestrator 8.18, we can see that it takes much longer than expected (numerous hours).
• From the "/var/log/vmware/prelude/upgrade/202#-##-##-##-##-##.log" logs we can see looping behaviour like:
  

  • [email protected]: Permission denied (publickey,password).
    [ERROR][####=##=## ##:##:##][server1.example.com] Remote command failed: mkdir -p /var/vmware/prelude/upgrade at host: server3.example.com
    [ERROR][####=##=## ##:##:##][server1.example.com] Remote command failed: mkdir -p /var/vmware/prelude/upgrade at one or more nodes
    [WARNING][####=##=## ##:##:##][server1.example.com] Sync of upgrade data to node server3.example.com cannot be completed at this time.
    [INFO][####=##=## ##:##:##][server1.example.com] Preparing directory for upgrade data on remote node: server2.example.com
    [INFO][####=##=## ##:##:##][server1.example.com] Running remote command: mkdir -p /var/vmware/prelude/upgrade at host: server2.example.com
    [ERROR][####=##=## ##:##:##][server1.example.com][Exit Code: 255] Attempt failed to run command: mkdir -p /var/vmware/prelude/upgrade.
    Pseudo-terminal will not be allocated because stdin is not a terminal.
    Welcome to VMware Aria Automation Orchestrator Appliance 8.18.1
    [email protected]: Permission denied (publickey,password).
    [ERROR][####=##=## ##:##:##][server1.example.com] Remote command failed: mkdir -p /var/vmware/prelude/upgrade at host: server2.example.com
    [ERROR][####=##=## ##:##:##][server1.example.com] Remote command failed: mkdir -p /var/vmware/prelude/upgrade at one or more nodes
    [WARNING][####=##=## ##:##:##][server1.example.com] Sync of upgrade data to node server2.example.com cannot be completed at this time.
    [INFO][####=##=## ##:##:##][server1.example.com] Deactivating upgrade monitor on the node
    [INFO][####=##=## ##:##:##][server1.example.com] Upgrade monitor deactivated successfully on the node.
    

Environment

Aria Orchestrator 8.18.1 Patch 1 to 4

Cause

The issue is due to a corrupted Patch file being downloaded and bound in the "Product Binaries" section of Aria Suite Lifecycle under Settings > Product Binaries.

Resolution

 Issue can be resolved by re-downloading the patch file from the Broadcom Support Portal and validating the checksum.