Storage throughput spikes and workload latency after upgrading to VCF Operations 9.1

Products

• VCF Operations
• VMware vCenter Server
• VMware vSphere ESX 8.x

Issue/Introduction

After upgrading VCF Operations to 9.1, network monitoring tools show storage network throughput spikes resulting in workload latency, such as in Horizon VDI workloads.

• Bandwidth graphs indicate spikes in throughput on the storage network every 4 hours.
  
  
• Turning off VCF Operations collection for the affected vCenter stops this behavior.

Environment

• VCF Operations 9.1.x
• VMware vCenter Server 8.x
• VMware vSphere ESX 8.x

Cause

VCF Operations diagnostics run every 4 hours. Starting in VCF Operations 9.1, a filesystem list is performed on up to 10 hosts in parallel. In environments with existing storage issues, such as latency and connectivity problems, the filesystem list operations performed by VCF Operations diagnostics may cause performance latency for workloads in the environment.

Resolution

Disable the VCF Operations Diagnostic adapter for the affected vCenter to prevent the filesystem list operations:

1. In VCF Operations, navigate to Operate > Administration > Integrations.

2. Expand the affected vCenter and note the collector or collector group name listed in the Collector column.

3. Navigate to Operate > Administration > Configurations > Inventory Management.

4. Expand Adapter Instances.

5. Select VCF Diagnostics Adapter Instance (do not expand the item).

6. In the right pane, select the adapter associated with the collector or collector group noted in step 2.

7. Click the square stop button in the toolbar to stop the adapter.

Additional Information

To isolate the affected vCenter on its own collector, deploy a cloud proxy and reconfigure the vCenter adapter to use this new cloud proxy.