Symptoms:
When connected to vCenter Server, you experience these symptoms:
- Cannot create virtual machines from templates.
- Cannot create a new virtual machine.
- You see the alarm:
Thin-provisioned storage devices met soft threshold
- The storage utilization on a LUN is above 90%.
- In the
vpxd.log
file, you see entries similar to:
mem> [verbose 'Default' opID=########-######15-3f] [VpxVmomi] Invoke done: vim.Folder.createVm session: ########-####-####-####-########c5b8
mem> [verbose 'SoapAdapter.HTTPService' opID=########-######15-3f] HTTP Response: Complete (processed 495 bytes)
mem> [verbose 'PropertyProvider'] RecordOp ASSIGN: info.startTime, task-29974
mem> [verbose 'Default' opID=########-######13-22] [VpxVmomi] Invoke done: vmodl.query.PropertyCollector.waitForUpdates session: ########-####-####-####-########c5b8
mem> [verbose 'PropertyProvider' opID=########-######15-3f] RecordOp ASSIGN: info.state, task-29974
mem> [verbose 'PropertyProvider' opID=########-######15-3f] RecordOp ASSIGN: info.cancelable, task-29974
mem> [verbose 'SoapAdapter.HTTPService' opID=########-######13-22] HTTP Response: Complete (processed 1390 bytes)
mem> [verbose 'Default' opID=86defaf9] [VpxVmomi] Invoking [waitForUpdatesEx] on [vmodl.query.PropertyCollector:propertyCollector] session [########-####-####-####-########7af4(########-####-####-####-########fdac)]
mem> [verbose 'SSL SoapAdapter.HTTPService.HttpConnection'] User agent is 'VMware vim-java 1.0'
mem> [verbose 'Default' opID=934f3ba3] [VpxVmomi] Invoking [waitForUpdatesEx] on [vmodl.query.PropertyCollector:propertyCollector] session [########-####-####-####-########a79f(########-####-####-####-########95cf)]
mem> [verbose 'SoapAdapter.HTTPService.HttpConnection'] User agent is 'VMware VI Client/4.0.0'
mem> [info 'commonvpxLro' opID=########-######15-3f] [VpxLRO] -- BEGIN task-29974 -- group-v22 -- vim.Folder.createVm -- ########-####-####-####-########c5b8(########-####-####-####-########29ba)
mem> [verbose 'SoapAdapter.HTTPService.HttpConnection'] User agent is 'VMware VI Client/4.0.0'
mem> [verbose 'vpxdvpxdMoDatastore' opID=########-######15-3f] [IsDatastoreActive] datastore DATASTORE_VMFS01 disabled (Low free space)
mem> [error 'VmProv' opID=########-######15-3f] [CreateDestDatastores] moDs is not active for DATASTORE_VMFS01
mem> [error 'Default' opID=########-######15-3f] (Log recursion level 2) vim.fault.InvalidDatastoreState
------ In-memory logs end --------
[error 'Default' opID=########-######15-3f] Section for VMware VirtualCenter, pid=3924, version=5.1.0, build=947673, option=Release
[info 'commonvpxLro' opID=########-######15-3f] [VpxLRO] -- FINISH task-29974 -- group-v22 -- vim.Folder.createVm --
[info 'Default' opID=########-######15-3f] [VpxLRO] -- ERROR task-29974 -- group-v22 -- vim.Folder.createVm: vim.fault.InvalidDatastoreState:
--> Result:
--> (vim.fault.InvalidDatastoreState) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> datastoreName = "DATASTORE_VMFS01",
--> msg = "",
--> }
--> Args:
[info 'commonvpxLro' opID=########-######16-55] [VpxLRO] -- BEGIN task-internal-6918 -- -- vmodl.query.PropertyCollector.createFilter -- ########-####-####-####-########c5b8(########-####-####-####-########29ba)
[info 'commonvpxLro' opID=########-######16-55] [VpxLRO] -- FINISH task-internal-6918 -- -- vmodl.query.PropertyCollector.createFilter --
[info 'commonvpxLro' opID=########-######19-d5] [VpxLRO] -- BEGIN task-internal-6919 -- -- vmodl.query.PropertyCollector.Filter.destroy -- ########-####-####-####-########c5b8(########-####-####-####-########29ba)
[info 'commonvpxLro' opID=########-######19-d5] [VpxLRO] -- FINISH task-internal-6919 -- -- vmodl.query.PropertyCollector.Filter.destroy --
[info 'commonvpxLro' opID=HB-host-64@2686-7be78501] [VpxLRO] -- BEGIN task-internal-6920 -- host-64 -- VpxdInvtHostSyncHostLRO.Synchronize --
[info 'vpxdvpxdInvtHostCnx' opID=HB-host-64@2686-7be78501] [VpxdHostSync] Synchronizing host: host-64 (hostname.dom.com, 10.118.1.234)
[info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (hostname.dom.com, vmodl.query.PropertyCollector.waitForUpdates)
[info 'vpxdvpxdVmomi' opID=HB-host-64@2686-7be78501] [ClientAdapterBase::InvokeOnSoap] Invoke done (hostname.dom.com, vpxapi.VpxaService.retrieveChanges
Note: All the log messages starting with mem> are from a memory dump. Similar looking lines will be in vpxd.log but they wont start with mem>.
Note: This article describes a specific issue. If you experience all of the preceding symptoms on a particular datastore, refer the following sections. If you experience some but not all of these symptoms, your issue is not related to this article.