Data Retriever Initialization Error Due to High Disk Utilization in VCF Operations

Products

• VCF Operations

Issue/Introduction

• When logging into the VCF Operations UI, the following error is displayed: Data Retriever is not initialized yet, please wait.

• In the Admin UI the cluster status shows one or more node(s) is "Waiting for Analytics" or shows red x .

• VCF Operations UI hangs when taking cluster offline

• The /storage/log partition is full (100% capacity) when checked via df -h over a root SSH connection.

Environment

VCF Operations 9.0.x

Cause

The error occurs because the /storage/log partition on the primary node is at full capacity, preventing the Analytics service from initializing communication with the cluster.

Resolution

1. SSH to all analytics node as root.
2. Manually take the node offline to allow for maintenance by running the following command:

    $VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOffline --offlineReason "Manual"

3. Repeat step 2 for all analytics nodes in the cluster. 
4. Verify the cluster status using the command : 

    vrops-status

5. Take a snapshot according to the following documentation before proceeding: Create a Snapshot Before You Upgrade VCF Operations
6. Run the following command clear unnecessary log files to free up disk space : 

   find /storage/log/ -mount -type f -mtime +1 -exec echo {} \; -exec truncate -cs 0 {} \; 2>&1 | tee /tmp/files_truncated.txt

7. Reboot the cluster nodes in the order listed in the Shutdown and Startup sequence for VCF/Aria Operations cluster.
8. Once all nodes are powered on and reporting status in vCenter, return to the Admin UI and click Bring Cluster Online.

Additional Information

Data Retriever is not initiated yet, please wait' message received on the Aria Operations user console

To speak with a customer representative or a Support Engineer, see Creating and managing Broadcom cases.