Unable to Deploy or create VMs on cluster level.

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

Unable to Deploy or create VMs on cluster level.

Not able to perform same even from ESXi UI as well.

Getting below error :

vsphere_client_virgo.log

[2024-11-22T09:30:47.814+02:00] [ERROR] http-nio-5090-exec-1525      71189500 113157 200764 c.v.v.client.vm.storageDrs.impl.StorageRecommendationsValidator   com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:1080/external-vecs/http2/Test.test.test.s.com/443/sdk invocation failed with "java.net.SocketTimeoutException: 600,000 milliseconds timeout on connection http-outgoing-28382 [ACTIVE]"
[

opid.log:

[2024-11-22 09:30:47.814] [ERROR] http-nio-5090-exec-1525      71189500 113157 200764 Invocation of 'void com.vmware.vim.binding.vim.StorageResourceManager.recommendDatastores(com.vmware.vim.binding.vim.storageDrs.StoragePlacementSpec,com.vmware.vim.vmomi.core.Future)' for https://Test.test.test.s.com:443/sdk (guid=3de21686-6c7f-40a3-b237-d2967fc43fe3, id=3000613) for opId 'm2fxe2mf-9889226-auto-5vyks-h5:71189500' failed in 600559 ms, error: com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:1080/external-vecs/http2/Test.test.test.s.com/443/sdk invocation failed with "java.net.SocketTimeoutException: 600,000 milliseconds timeout on connection http-outgoing-28382 [ACTIVE]"

[2024-11-22 09:30:47.818] [WARN ] http-nio-5090-exec-1525       Request processing for opId 'm2fxe2mf-9889226-auto' to URL /ui/mutation/validate/urn%3Avmomi%3AVirtualMachine%3Avm-3018060%3A3de21686-6c7f-40a3-b237-d2967fc43fe3 took too long: 600578 ms

 

vpxd.all:

2024-11-22T09:20:47.268+02:00 info vpxd[39097] [Originator@6876 sub=vpxLro opID=m2fxe2mf-9889226-auto-5vyks-h5:71189500-e9] [VpxLRO] -- BEGIN lro-1214853025 -- StorageResourceManager -- vim.StorageResourceManager.recommendDatastores -- 521db0c8-ac86-f3af-b578-26a0a8930554(52ad7d6a-65fa-9f38-13a2-2589464a02a9)

2024-11-22T09:20:47.352+02:00 info vpxd[39097] [Originator@6876 sub=SsoClient opID=m2fxe2mf-9889226-auto-5vyks-h5:71189500-e9] Successfully acquired token: SamlToken [subject={Name: vpxd-11137d40-dfb6-11e5-aba9-005056924667; Domain:vsphere.local}, groups=[{Name: Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager
.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=2024-11-22 07:20:47.300, expirationTime=2024-11-22 15:20:47.300, renewable=false, delegable=false, isSolution=true,confirmationType=1]
2024-11-22T09:20:47.370+02:00 error vpxd[14140] [Originator@6876 sub=vmomi.soapStub[392875] opID=m2fxe2mf-9889226-auto-5vyks-h5:71189500-e9] Initial service state request failed, disabling pings; /invsvc/vmomi/sdk, , <TCP '127.0.0.1 : 10080'>>>, HTTP Status:400 'Bad Request'

2024-11-22T09:30:43.007+02:00 info vpxd[14066] [Originator@6876 sub=vpxLro opID=sps-Main-328281-438-260643-68] [VpxLRO] -- BEGIN session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]5295ae6f-c3a1-4a97-566b-dbeae5883cc6 -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange -- 5265c884-c1ae-cc2c-2127-8a2431a0fc5a(524bf87c-40ea-11db-1a07-d318762e14a7)
2024-11-22T09:30:43.007+02:00 error vpxd[14066] [Originator@6876 sub=HostPicker opID=sps-Main-328281-438-260643-68] Couldn't find any candidate hosts for the provided urls
2024-11-22T09:30:43.007+02:00 warning vpxd[14066] [Originator@6876 sub=Vmomi opID=sps-Main-328281-438-260643-68] VMOMI activation LRO failed; <<5265c884-c1ae-cc2c-2127-8a2431a0fc5a, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 38242'>>, CatalogSyncManager, vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange>, N3Vim5Fault21InaccessibleDatastore9ExceptionE(Fault cause: vim.fault.InaccessibleDatastore
--> )
--> [context]zKq7AVECAQAAAAZLcQEWdnB4ZAAAAto3bGlidm1hY29yZS5zbwAAmXksABdtLQAf6jIBgfBxdnB4ZAABh1B0AUdKlQEGkZWBx6YaAYEBVhoBgv2fPgFsaWJ2aW0tdHlwZXMuc28AgfoBagGB9QRpAYE2BmkBgWQVaQGBx0JoAYGS7GgBAOdJIwB1nyMAwGU3A4d/AGxpYnB0aHJlYWQuc28uMAAEvzYPbGliYy5zby42AA==[/context]
2024-11-22T09:30:43.011+02:00 info vpxd[14066] [Originator@6876 sub=vpxLro opID=sps-Main-328281-438-260643-68] [VpxLRO] -- FINISH session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]5295ae6f-c3a1-4a97-566b-dbeae5883cc6
2024-11-22T09:30:43.011+02:00 info vpxd[14066] [Originator@6876 sub=Default opID=sps-Main-328281-438-260643-68] [VpxLRO] -- ERROR session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]5295ae6f-c3a1-4a97-566b-dbeae5883cc6 -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange: vim.fault.InaccessibleDatastore:
--> Result:
--> (vim.fault.InaccessibleDatastore) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = ,
-->    datastore = 'vim.Datastore:3de21686-6c7f-40a3-b237-d2967fc43fe3:datastore-4770264',
-->    name = "CSSA-IBOX1811-DEL_BSS_RES_CU1-14",
-->    detail = "notAccessible"
-->    msg = ""
--> }
--> Args:
-->
--> Arg catalogChangeSpec:
--> (vim.vslm.CatalogChangeSpec) {
-->    datastore = 'vim.Datastore:datastore-4770264',
-->    startVClockTime = (vim.vslm.VClockInfo) {
-->       vClockTime = 0
-->    },
-->    fullSync = false
--> }

sps.log:

2024-11-22T09:30:47.124+02:00 [pool-28-thread-5] INFO  opId=sps-Main-328281-438 com.vmware.vim.storage.common.util.OperationIdUtil - OperationID present in invoker thread, adding suffix and re-using it sps-Main-328281-438-260647
2024-11-22T09:30:47.124+02:00 [pool-2-thread-15] INFO  opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Active thread count is: 1, Core Pool size is: 20, Queue size: 0, Time spent waiting in queue: 0 millis
2024-11-22T09:30:47.128+02:00 [pool-2-thread-15] INFO  opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Request took 4 millis to execute.

2024-11-22T09:30:48.588+02:00 [jaeger.RemoteReporter-QueueProcessor] WARN  opId=sps-Main-328281-438 io.jaegertracing.internal.reporters.RemoteReporter - FlushCommand execution failed! Repeated errors of this command will not be logged.
io.jaegertracing.internal.exceptions.SenderException: Failed to flush spans.
        ... 3 more
Caused by: org.apache.thrift.transport.TTransportException: Cannot flush closed transport
        at io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:151) ~[jaeger-thrift-1.6.0.jar:1.6.0]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[libthrift-0.14.1.jar:0.14.1]
        at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) ~[libthrift-0.14.1.jar:0.14.1]
        at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) ~[jaeger-thrift-1.6.0.jar:1.6.0]
        at io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63) ~[jaeger-thrift-1.6.0.jar:1.6.0]
        at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:84) ~[jaeger-thrift-1.6.0.jar:1.6.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[jaeger-thrift-1.6.0.jar:1.6.0]
        ... 3 more
Caused by: java.net.PortUnreachableException: ICMP Port Unreachable
        at java.net.PlainDatagramSocketImpl.send(Native Method) ~[?:1.8.0_402]
        at java.net.DatagramSocket.send(DatagramSocket.java:693) ~[?:1.8.0_402]
        at io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:149) ~[jaeger-thrift-1.6.0.jar:1.6.0]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[libthrift-0.14.1.jar:0.14.1]
        at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) ~[libthrift-0.14.1.jar:0.14.1]
        at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) ~[jaeger-thrift-1.6.0.jar:1.6.0]
        at io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63) ~[jaeger-thrift-1.6.0.jar:1.6.0]
        at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:84) ~[jaeger-thrift-1.6.0.jar:1.6.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[jaeger-thrift-1.6.0.jar:1.6.0]

 

VPXD.log:

     2024-11-22T06:10:39.284+02:00 info vpxd[15009] [Originator@6876 sub=Default opID=sps-Main-328281-438-248639-a3] [VpxLRO] -- ERROR session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]52beeeb1-e5e4-a4ca-500a-6007919ad72d -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange: vim.fault.InaccessibleDatastore:
--> (vim.fault.InaccessibleDatastore) {
2024-11-22T06:10:55.229+02:00 warning vpxd[10433] [Originator@6876 sub=Vmomi opID=sps-Main-328281-438-248655-b1] VMOMI activation LRO failed; <<5265c884-c1ae-cc2c-2127-8a2431a0fc5a, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 38242'>>, CatalogSyncManager, vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange>, N3Vim5Fault21InaccessibleDatastore9ExceptionE(Fault cause: vim.fault.InaccessibleDatastore
2024-11-22T06:10:55.232+02:00 info vpxd[10433] [Originator@6876 sub=Default opID=sps-Main-328281-438-248655-b1] [VpxLRO] -- ERROR session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]5254621c-8381-9351-1086-d0cbea5ac02c -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange: vim.fault.InaccessibleDatastore:
--> (vim.fault.InaccessibleDatastore) {
2024-11-22T06:10:58.620+02:00 warning vpxd[08478] [Originator@6876 sub=Vmomi opID=sps-Main-328281-438-248658-f4] VMOMI activation LRO failed; <<5265c884-c1ae-cc2c-2127-8a2431a0fc5a, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 38242'>>, CatalogSyncManager, vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange>, N3Vim5Fault21InaccessibleDatastore9ExceptionE(Fault cause: vim.fault.InaccessibleDatastore
2024-11-22T06:10:58.624+02:00 info vpxd[08478] [Originator@6876 sub=Default opID=sps-Main-328281-438-248658-f4] [VpxLRO] -- ERROR session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]52b0773e-e498-b7a5-b3fc-eb0b955f4d77 -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange: vim.fault.InaccessibleDatastore:
--> (vim.fault.InaccessibleDatastore) {
2024-11-22T06:11:31.736+02:00 warning vpxd[13565] [Originator@6876 sub=Vmomi opID=sps-Main-328281-438-248691-9d] VMOMI activation LRO failed; <<5265c884-c1ae-cc2c-2127-8a2431a0fc5a, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 38242'>>, CatalogSyncManager, vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange>, N3Vim5Fault21InaccessibleDatastore9ExceptionE(Fault cause: vim.fault.InaccessibleDatastore

Environment

VMware vCenter Server 7.0

VMware vCenter Server 8.0

Cause

This type of issue can cause If Datastore utilization crossing the Utilization threshold value which is set on Storage DRS & its a feature of DRS.

Also, Storage DRS checks for rule violations when you create a virtual machine on a Storage DRS-enabled datastore. If Storage DRS cannot create the new virtual machine's disks in compliance with the rules, it generates a fault. The fault is generated because Storage DRS cannot reference the virtual machine, which is in the process of being created and does not yet exist.

 

Resolution

• Need to add New Datastore or increase the Datastore size.

         or

• Remove the configured Rules or policy which is creating DRS fault.