RabbitMQ pod fails to mount PVC due to ResourceInUse error in VCF Automation 9.1

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

RabbitMQ pods in VCF Automation (VCFA) 9.1 fail to initialize with a ResourceInUse error when their required Persistent Volume Claim (PVC) remains attached to an old VCFA virtual machine.

Symptoms

• VCF Automation UI is inaccessible and power off task from the Fleet Management fails with timeout error.

• Describing rabbitmq-ha pod using kubectl describe pod -n prelude rabbitmq-ha-# reveals a FailedAttachVolume event from attachdetach-controller.

• CSI attacher container logs ( /var/log/pods/kube-system_vsphere-csi-controller-<UUID>/csi-attacher) shows the error ResourceInUse for the disk UUID.

  Fault: (*types.ResourceInUse)(0x######)({
  n VimFault: (types.VimFault) {
  n MethodFault: (types.MethodFault)
  {\\n FaultCause: (*types.LocalizedMethodFault)(<nil>),\\n FaultMessage: ([]types.LocalizableMessage) <nil>\\n }
  n },
  n Type: (string) (len=9) \\\"CnsVolume\\\",
  n Name: (string) (len=36) \\\"<disk-UUID>\\\"
  n }),
  n LocalizedMessage: (string) (len=62) \\\"The resource '<disk-UUID>' is in use.

• kubectl get volumeattachments shows the ATTACHED status of the PVC as false.

• The vCenter Cloud Native Storage (CNS) service confirms the PVC is attached to an old VCFA VM.

Environment

VCF Automation 9.1

Cause

The CSI controller in VCF Automation 9.1 cannot attach the required PVC to the current VCFA VMs because the underlying disk remains attached to a previous version of VCF Automation VM node.This conflict occurs if the old VCFA VM is accidentally powered on from the vCenter, preventing the CNS service from releasing the volume for the new VCFA cluster.

Resolution

To resolve this issue, detach the disk from the old VCF Automation node, to allow it to attach to the new VCFA 9.1 nodes.

NOTE : While removing the hard disk from the old VCFA VM, ensure to carefully select Remove device, otherwise it may result in data loss. Engage Broadcom Support if you need any assistance in applying the below procedure.

1. Log in to vCenter and navigate to Cluster > Monitor > Cloud Native Storage > Container Volumes > <pvc-###> >Basics. Identify the old VCFA VM name and the specific VMDK disk file path associated with the failing PVC,disk UUID. To confirm the referred VM is old, validate that the name of the VM is not present in VCF Automation component in the Fleet Management 9.1.

2. Power off the old VCFA VM identified in the previous step.

3. Detach the specific VMDK Hard disk from the old VCFA VM within vCenter by selecting "Remove device" .Once the disk is detached from the old VM, the CSI controller will successfully attach it to the correct VCFA 9.1 node, allowing the RabbitMQ pod to start.

4. If logging-operator-fluentd pods fails to start, resolve the issue by following the steps in the KB 438093 .