'Attach a virtual disk', and 'Detach a virtual disk' tasks in the vSphere Client result in a message stating "Database temporarily unavailable or has network problems."

Products

• VMware vSphere ESXi
• VMware vCenter Server 8.0

Issue/Introduction

The following error message is being received in the Tasks of the vSphere Client when attempting to attach or detach persistent volumes (PVCs) to VMs that are used for nodes in OpenShift or vanilla Kubernetes clusters.

 

"Database temporarily unavailable or has network problems."

 

Entries such as the following example may be seen in the vCenter vpxd logs:

info vpxd[24437] [Originator@6876 sub=Default opID=<opID>] [VpxLRO] -- ERROR task-<task-ID> -- vm-<vm-ID> -- vim.VirtualMachine.detachDisk: vim.fault.DatabaseError:
--> Result:
--> (vim.fault.DatabaseError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00005585530eb4f8, h:274, <TCP '<IP-addr:port>'>, <TCP 'IP-addr:port'>>), /vpxa>]: retrieveVStorageObject
--> Received SOAP response fault from [<<io_obj p:0x000000e7e49f9058, h:21, <TCP '127.0.0.1 : 15093'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: retrieveVStorageObject
--> Database temporarily unavailable or has network problems."
--> }
--> Args:
-->
--> Arg diskId:
--> (vim.vslm.ID) {
--> id = "<ID>"
--> }

 

Entries such as the following example may be seen in the ESXi vpxa logs:

 info vpxa[2100492] [Originator@6876 sub=Default opID=<opID>] [VpxLRO] -- ERROR task-<task-ID> -- vstorageObjectManager -- vim.vslm.host.VStorageObjectManager.createDisk: vim.fault.DatabaseError:
--> Result:
--> (vim.fault.DatabaseError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = "Database temporarily unavailable or has network problems."
--> }

 

Entries such as the following example may be seen in the ESXi hostd logs stating "Wrong tidy version":

info hostd[2100134] [Originator@6876 sub=Libs opID=<op-ID> user=vpxuser:<ID>] FCDLIB: fcd-catalog: Trying to reconcile datastore /vmfs/volumes/<datastore-ID>/ to get disk storage as it is not subscribed or is in suspened state
info hostd[2100134] [Originator@6876 sub=Vslmsvc.DatastoreValidator opID=<op-ID> user=vpxuser:<ID>] Subscribing/resuming datastore /vmfs/volumes/<datastore-ID>
info hostd[2100134] [Originator@6876 sub=Libs opID=<op-ID> user=vpxuser:<ID>] badVersion(303)
info hostd[2100134] [Originator@6876 sub=Libs opID=<op-ID> user=vpxuser:<ID>] Wrong tidy version: mTidyVersion = 1, header.mVersion = 0
info hostd[2100134] [Originator@6876 sub=Libs opID=<op-ID> user=vpxuser:<ID>] New error before the previous is handled
info hostd[2100134] [Originator@6876 sub=Libs opID=<op-ID> user=vpxuser:<ID>] badVersion(303)
info hostd[2100134] [Originator@6876 sub=Libs opID=<op-ID> user=vpxuser:<ID>] Catalog health is set: unhealthy badVersion 558
info hostd[2100134] [Originator@6876 sub=Libs opID=<op-ID> user=vpxuser:<ID>] Failed to init fcd-catalog for datastore: datastorePath->Value() = /vmfs/volumes/<datastore-ID>/
info hostd[2100134] [Originator@6876 sub=Libs opID=<op-ID> user=vpxuser:<ID>] New error before the previous is handled
info hostd[2100134] [Originator@6876 sub=Default opID=<op-ID> user=vpxuser:<ID>] Transfer to exception eraro code: 601, message: The error code was generalized from badVersion(303) to db(601)
info hostd[2100134] [Originator@6876 sub=AdapterServer opID=<op-ID> user=vpxuser:<ID>] AdapterServer caught exception; <<5240ea9f-a394-6ed1-7a1e-01d19808599b, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 16756'>>, ha-vstorage-object-manager, vim.vslm.host.VStorageObjectManager.retrieveVStorageObject>, N3Vim5Fault13DatabaseError9ExceptionE(Fault cause: vim.fault.DatabaseError

Environment

• vCenter Server 7.x
• vCenter Server 8.x

Cause

• The tidy file of the datastore can become corrupted, causing discrepancies between the CNS database and the datastore that is used for storing the persistent volumes.

Resolution

The FCD Catalog needs to be rebuilt for the affected datastore.

KB article 320790 provides resolution steps to rebuild the first class disk (FCD) catalog to reconcile this particular issue:

• Persistent volume is failing to attach with Error: "The input volume <Volume handle ID> is not registered as a CNS volume"