PV attachment to a pod fails with the error: 'The resource volume is in use'.
search cancel

PV attachment to a pod fails with the error: 'The resource volume is in use'.

book

Article ID: 313112

calendar_today

Updated On:

Products

Tanzu Kubernetes Grid VMware vSphere Kubernetes Service

Issue/Introduction

New Pods remain stuck in the Init or ContainerCreating state.

Describing the pod shows the error: "The resource 'volume' is in use."

kubectl describe pod <pod-name> -n <namespace>

Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedAttachVolume 43m attachdetach-controller AttachVolume.Attach failed for volume "pvc-########-###-#####-###-#########" : rpc error: code = Internal desc = failed to attach disk: "#########-####-####-####-#######" with node: "workload-###-#-#######-#####" err failed to attach cns volume: "#########-####-####-####-#######" to node vm: "VirtualMachine:vm-###### [VirtualCenterHost: ******, UUID: ########-####-####-####-########, Datacenter: ***** [Datacenter: Datacenter:datacenter-##, VirtualCenterHost: *******]]". fault: "(*types.LocalizedMethodFault)(0xc000a70c60)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (*types.ResourceInUse)(0xc000ede640)({\n VimFault: (types.VimFault) {\n MethodFault: (types.MethodFault) {\n FaultCause: (*types.LocalizedMethodFault)(<nil>),\n FaultMessage: ([]types.LocalizableMessage) <nil>\n }\n },\n Type: (string) \"\",\n Name: (string) (len=6) \"volume\"\n }),\n LocalizedMessage: (string) (len=32) \"The resource 'volume' is in use.\"\n})\n". opId: "086a91d5"
Warning FailedMount 87s (x1769 over 2d18h) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data]: timed out waiting for the condition


Logs from the csi-attacher container show the error 'NoPermission', indicating a permission issue during the volume detach operation.

kubectl logs <csi-controller-pod-name> -n <namespace> -c csi-attacher

I1111 04:45:24.730061       1 controller.go:165] Ignoring VolumeAttachment "csi-##############################################################" change
I1111 04:45:24.730066       1 csi_handler.go:624] Saved detach error to "csi-##############################################################"
I1111 04:45:24.730097       1 csi_handler.go:231] Error processing "csi-##############################################################": failed to detach: rpc error: code = Internal desc = queryVolume failed for volumeID: "########-####-####-####-#########" with err=ServerFaultCode: NoPermission


Environment

Tanzu Kubernetes Grid
vSphere with Tanzu.

Cause

The issue arises because the TKG role assigned to the TKG user doesn't have sufficient vSphere permissions to detach the virtual disk (VMDK),
causing it to stay connected to the original node rather than detaching as intended.

Resolution

Ensure that the TKG role has the necessary permissions as outlined in the 'Required Permissions for the vSphere Account' section of the documentation

vSphere Object Permissions Required for Tanzu Kubernetes Grid

vSphere Object

Required Permissions

Cns

- Searchable

Datastore

- Allocate space
- Browse datastore
- Low level file operations

Global
(If using Velero for backup and restore)

- Disable methods
- Enable methods
- Licenses

Network

- Assign network

Profile-driven storage

- Profile-driven storage view

Resource

- Assign virtual machine to resource pool

Sessions

- Message
- Validate session

Virtual machine

- Change Configuration > Add existing disk
- Change Configuration > Add new disk
- Change Configuration > Add or remove device
- Change Configuration > Advanced configuration
- Change Configuration > Change CPU count
- Change Configuration > Change Memory
- Change Configuration > Change Settings
- Change Configuration > Configure Raw device
- Change Configuration > Extend virtual disk
- Change Configuration > Modify device settings
- Change Configuration > Remove disk
- Change Configuration > Toggle disk change tracking *
- Edit Inventory > Create from existing
- Edit Inventory > Remove
- Interaction > Power On
- Interaction > Power Off
- Provisioning > Allow read-only disk access *
- Provisioning > Allow virtual machine download *
- Provisioning > Clone virtual machine
- Provisioning > Deploy template
- Snapshot Management > Create snapshot *
- Snapshot Management > Remove snapshot *

vApp

- Import

 

Assigning the TKG Role to vSphere Objects

Assign the tkg-user with the TKG role to each of the following vSphere objects involved in your Tanzu Kubernetes Grid deployment:

vSphere Inventory Permissions

  • Hosts and Clusters
    • The root vCenter Server object
    • The Datacenter and all Host and Cluster folders — from the Datacenter object down to the cluster managing the TKG deployment
    • Target hosts and clusters
    • Target resource pools (ensure "Propagate to children" is enabled)
  • VMs and Templates
    • The Tanzu Kubernetes Grid base image templates
    • Target VM and Template folders (with "Propagate to children" enabled)
  • Storage
    • Datastores and all storage folders — from the Datacenter object down to the datastores used for TKG deployments
  • Networking
    • Networks or distributed port groups assigned to clusters
    • Distributed switches

Reference Article:

Prepare to Deploy Management Clusters to vSphere

 

Additional Information

  • For a pod to successfully attach a Persistent Volume (PV), the virtual disk (VMDK) must be mounted to the worker node where the pod is running.
  • If the pod stops, the VMDK is detached from the node.
  • If the pod starts again (either on the same node or a different node), the VMDK is re-attached to the new node.

 

Powered by Wolken Software