Automated Storage DRS can lead to impacts for applications running within the VCF Services Runtime Cluster

Products

• VCF Automation
• VCF Operations

Issue/Introduction

• If the issue is hit, an application that is hosted on a VCF Management Services (VCFMS) cluster or on a VCF Automation cluster in a VCF instance may be down or not operating properly.
• When describing pods with a PVC attached (kubectl describe pods pod_name -n namespace), you may see an error as such:

Warning  FailedAttachVolume  21s attachdetach-controller  AttachVolume.Attach failed for volume "pvc-7e7834fa-####-####-####-############" : rpc error: code = Internal desc = failed to attach disk: "79905314-####-####-####-############" with node: "4213a554-####-####-####-############" err failed to attach cns volume: "79905314-####-####-####-############" to node vm: "VirtualMachine:vm-## [VirtualCenterHost: host.example.com, UUID: 4213a554-####-####-####-############, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-#, VirtualCenterHost: host.example.com]]". fault: "(*types.LocalizedMethodFault)(0x18b5489d1e40)({
 DynamicData: (types.DynamicData) {
 },
 Fault: (*types.ResourceInUse)(0x18b5486cb840)({
  VimFault: (types.VimFault) {
   MethodFault: (types.MethodFault) {
    FaultCause: (*types.LocalizedMethodFault)(<nil>),
    FaultMessage: ([]types.LocalizableMessage) <nil>
   }
  },
  Type: (string) \"\",
  Name: (string) (len=6) \"volume\"
 }),
 LocalizedMessage: (string) (len=32) \"The resource 'volume' is in use.\"
})
". opId: "6315####"

Environment

• VCF Management Services Runtime 9.x
• VCF Automation 9.x
• VCF Identity Broker 9.x

Cause

VCF Management Services leverages Cloud Native Storage system (CNS) to provide first-class virtual disk (FCD) for databases and other state storage.

If the disks used to back VCF Management Services are located on datastore clusters leveraging Storage DRS, the SDRS activities may trigger unplanned relocations of files which may impact the applications running within the VCF Services Runtime Cluster.

Resolution

Broadcom recommends that the virtual machines that make up the VCF Services Runtime cluster be added to the Storage DRS VM overrides list in vCenter, with SDRS Disabled and henceforth monitored to ensure they are kept in the exclusion list.

1. Open the vSphere Client.
2. Select your Datastore Cluster.
3. Navigate to Configure > VM Overrides.
4. Click Add.
5. Choose the virtual machines and click Next.
6. Select Override and choose Disabled from the dropdown.
7. Click Finish.

This recommendation aligns with official documentation for the vSphere Container Storage Plugin.

 

Note: This issue will not be present on Clusters deployed on 9.1.0 EP2 (9.1.0.0200), 9.1.1 or later.