Error: "AttachVolume.Attach failed for volume \"pvc-########-####-####-####-############\" on VIO controller

Products

• VMware Integrated OpenStack

Issue/Introduction

• VIO services can not start. 
• OpenStack Deployment State: STARTING
• kubelet.log shows:

Jul 11 10:02:11 controller-pzxt64kxf6 kubelet[3138]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'

• kube-controller-manager.log shows:

Jul 11 11:27:28 vioapp.example.local kube-controller-manager[553]: E0711 11:27:28.467182    1 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/vsphere-volume/[datastore_name] kubevols/kubernetes-dynamic-pvc-########-####-####-####-############.vmdk\"" failed. No retries permitted until 2022-07-11 11:29:30.467127953 +0000 UTC m=+1932.403365010 (durationBeforeRetry 2m2s). Error: "AttachVolume.Attach failed for volume \"pvc-########-####-####-####-############.\" (UniqueName: \"kubernetes.io/vsphere-volume/[datastore_name] kubevols/kubernetes-dynamic-pvc-########-####-####-####-############..vmdk\") from node \"controller-vqmgbcmnkq\" : Invalid configuration for device '0'."

Environment

7.x

Cause

The reason why VIO is stuck with starting is that the vmdk files for persistent volumes were moved out of [datastore_name] kubevols/.

This can happen when a svMotion is done on a controller(s).  Please note that this is not supported.

Resolution

Note: Please check the status of your deployment backups.  They may be needed to recover.

1. List persistent volumes on VIO manager using the command:

# osctl get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-data-mariadb-server-0 Bound pvc-########-####-####-####-############. 60Gi RWO general 398d
mysql-data-mariadb-server-1 Bound pvc-########-####-####-####-############ 60Gi RWO general 398d
mysql-data-mariadb-server-2 Bound pvc-########-####-####-####-############ 60Gi RWO general 398d
rabbitmq-data-rabbitmq1-rabbitmq-0 Bound pvc-########-####-####-####-############ 20Gi RWO general 579d
rabbitmq-data-rabbitmq1-rabbitmq-1 Bound pvc-########-####-####-####-############ 20Gi RWO general 579d
rabbitmq-data-rabbitmq1-rabbitmq-2 Bound pvc-########-####-####-####-############ 20Gi RWO general 579d
restore-restore985 Bound pvc-########-####-####-####-############ 60Gi RWO general 398d
vspc-logs-nova-compute-01-compute-0 Bound pvc-########-####-####-####-############ 2Gi RWO general 579d
vspc-logs-nova-compute-02-compute-0 Bound pvc-########-####-####-####-############ 2Gi RWO general 579d
vspc-logs-nova-compute-03-compute-0 Bound pvc-########-####-####-####-############ 2Gi RWO general 579d
vspc-logs-nova-compute-04-compute-0 Bound pvc-########-####-####-####-############ 2Gi RWO general 579d

2. Check controller vms for attached vmdks (exclude Hard disk 1) to see if any vmdk files are not in kubevols/ folder.

Here are the steps to recover

In our example, the following persistent volumes are not in kubevols/ folder but are attached to controller, controller-vqmgbcmnkq, with a different name.

vspc-logs-nova-compute-03-compute-0 Bound pvc-########-####-####-####-############
rabbitmq-data-rabbitmq1-rabbitmq-2 Bound pvc-########-####-####-####-############
mysql-data-mariadb-server-2 Bound pvc-########-####-####-####-############

1. From VIO Manager run: viocli stop services 
2. Go to vCenter and locate controller,  controller-vqmgbcmnkq 
3. Use Edit Settings to remove Hard disks with size 2GB, 20GB and 60GB (DO NOT delete these vmdk files).  Noting which name corresponds to those sizes.  They must be renamed properly.
4. Go to Storage view and select datastore [datastore_name], 
5. Go to folder controller-vqmgbcmnkq
6. Select controller-vqmgbcmnkq_1.vmdk and choose Move to to move it to [datastore_name] kubevols/
7. Do the same for the following files: controller-vqmgbcmnkq_3.vmdk controller-vqmgbcmnkq_4.vmdk
8. Go to folder kubevols/
9. select controller-vqmgbcmnkq_1.vmdkand choose Rename to from the top to rename it to kubernetes-dynamic-pvc-########-####-####-####-############.vmdk
10. Rename ontroller-vqmgbcmnkq_3.vmdk to kubernetes-dynamic-pvc-########-####-####-####-############.vmdk
11. Rename controller-vqmgbcmnkq_4.vmdk to kubernetes-dynamic-pvc-########-####-####-####-############.vmdk
12. From VIO Manager run: viocli start services

Additional Information

The deployment will be down until the persistent volumes can be recovered.  Rabbitmq volumes contain rabbitmq data, mariadb volumes contain the database, vspc volumes contain the vspc logs.

Data loss is possible.