VCF Operations Upgrade Error: Operation failed. Please ensure there is an active connection to the server and try again. For troubleshooting, check the logs for more details

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• VCF Operations upgrades to version 9.0.2.x may fail in the UI, displaying this error:
  
  Operation failed. Please ensure there is an active connection to the server and try again. For troubleshooting, check the logs for more details.
  
  
• The /storage/log/vcops/log/casa/admin-ui.log file records the following exception:
  
  

  INFO  admin-ui #### [ops@### threadId="###" threadName="###"] [com.vmware.vcops.util.admin.HTTPSRequester.###] - Response : {"error_message_key":"general.failure","error_arguments":[],"error_message":"Operation failed"}
  ERROR admin-ui #### [ops@### threadId="###" threadName="###"] [com.vmware.vcops.util.admin.solution.SolutionManagement.###] - CaSA response: 500
  ERROR admin-ui #### [ops@### threadId="###" threadName="###"] [com.vmware.vcops.adminui.service.PakUploadService.###] - Error reported while trying to upload the solution (Operations-Upgrade-<Ops_Version>.pak) : Operation failed. Please ensure there is an active connection to the server and try again. For troubleshooting, check the logs for more details

  
  
  
• The /storage/log/vcops/log/casa/casa.log file records the following error:
  
  

  WARN  casa #### [ops@### threadId="###" threadName="###" requestId="###"] [com.vmware.vcops.casa.exception.CasaControllerExceptionHandler.runtimeExceptionHandler:###] - general runtime exception
  com.vmware.vcops.casa.exception.CasaException: java.lang.RuntimeException: java.io.IOException: No space left on device
          at com.vmware.vcops.casa.upgrade.pak.PakService.uploadToMaster(PakService.java:####) ~[classes/:?]

Environment

VMware Cloud Foundation (VCF) Operations 9.0.2.x

Cause

The VCF Operations analytics nodes has run out of disk space during the .pak upload process. When the partition housing the database and temporary upgrade staging files reaches its capacity limit, the CaSA service cannot unpack or write the upgrade binaries

Resolution

To resolve this issue, increase the capacity of the analytics nodes by adding a new virtual disk. For step-by-step instructions on properly adding this space to the appliance, refer to  KB 318768.

Important Notes:

• Do not expand existing disks: Never expand an existing virtual disk directly from vCenter. Doing so will fail to partition the space correctly within the appliance, and storage utilization will not decrease. You must add a completely new virtual disk.

• Maintain cluster uniformity: Ensure that all analytics nodes within the cluster maintain the exact same amount of configured storage space.