Unable to Deploy or create VMs on cluster level.
Not able to perform same even from ESXi UI as well.
Getting below error :
vsphere_client_virgo.log
[2024-11-22T09:30:47.814+02:00] [ERROR] http-nio-5090-exec-1525 71189500 113157 200764 c.v.v.client.vm.storageDrs.impl.StorageRecommendationsValidator com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:1080/external-vecs/http2/Test.test.test.s.com/443/sdk invocation failed with "java.net.SocketTimeoutException: 600,000 milliseconds timeout on connection http-outgoing-28382 [ACTIVE]"
[
opid.log:
[2024-11-22 09:30:47.814] [ERROR] http-nio-5090-exec-1525 71189500 113157 200764 Invocation of 'void com.vmware.vim.binding.vim.StorageResourceManager.recommendDatastores(com.vmware.vim.binding.vim.storageDrs.StoragePlacementSpec,com.vmware.vim.vmomi.core.Future)' for https://Test.test.test.s.com:443/sdk (guid=3de21686-6c7f-40a3-b237-d2967fc43fe3, id=3000613) for opId 'm2fxe2mf-9889226-auto-5vyks-h5:71189500' failed in 600559 ms, error: com.vmware.vim.vmomi.client.exception.ConnectionException: http://localhost:1080/external-vecs/http2/Test.test.test.s.com/443/sdk invocation failed with "java.net.SocketTimeoutException: 600,000 milliseconds timeout on connection http-outgoing-28382 [ACTIVE]"
[2024-11-22 09:30:47.818] [WARN ] http-nio-5090-exec-1525 Request processing for opId 'm2fxe2mf-9889226-auto' to URL /ui/mutation/validate/urn%3Avmomi%3AVirtualMachine%3Avm-3018060%3A3de21686-6c7f-40a3-b237-d2967fc43fe3 took too long: 600578 ms
vpxd.all:
2024-11-22T09:20:47.268+02:00 info vpxd[39097] [Originator@6876 sub=vpxLro opID=m2fxe2mf-9889226-auto-5vyks-h5:71189500-e9] [VpxLRO] -- BEGIN lro-1214853025 -- StorageResourceManager -- vim.StorageResourceManager.recommendDatastores -- 521db0c8-ac86-f3af-b578-26a0a8930554(52ad7d6a-65fa-9f38-13a2-2589464a02a9)
2024-11-22T09:20:47.352+02:00 info vpxd[39097] [Originator@6876 sub=SsoClient opID=m2fxe2mf-9889226-auto-5vyks-h5:71189500-e9] Successfully acquired token: SamlToken [subject={Name: vpxd-11137d40-dfb6-11e5-aba9-005056924667; Domain:vsphere.local}, groups=[{Name: Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager
.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=2024-11-22 07:20:47.300, expirationTime=2024-11-22 15:20:47.300, renewable=false, delegable=false, isSolution=true,confirmationType=1]
2024-11-22T09:20:47.370+02:00 error vpxd[14140] [Originator@6876 sub=vmomi.soapStub[392875] opID=m2fxe2mf-9889226-auto-5vyks-h5:71189500-e9] Initial service state request failed, disabling pings; /invsvc/vmomi/sdk, , <TCP '127.0.0.1 : 10080'>>>, HTTP Status:400 'Bad Request'
2024-11-22T09:30:43.007+02:00 info vpxd[14066] [Originator@6876 sub=vpxLro opID=sps-Main-328281-438-260643-68] [VpxLRO] -- BEGIN session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]5295ae6f-c3a1-4a97-566b-dbeae5883cc6 -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange -- 5265c884-c1ae-cc2c-2127-8a2431a0fc5a(524bf87c-40ea-11db-1a07-d318762e14a7)
2024-11-22T09:30:43.007+02:00 error vpxd[14066] [Originator@6876 sub=HostPicker opID=sps-Main-328281-438-260643-68] Couldn't find any candidate hosts for the provided urls
2024-11-22T09:30:43.007+02:00 warning vpxd[14066] [Originator@6876 sub=Vmomi opID=sps-Main-328281-438-260643-68] VMOMI activation LRO failed; <<5265c884-c1ae-cc2c-2127-8a2431a0fc5a, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 38242'>>, CatalogSyncManager, vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange>, N3Vim5Fault21InaccessibleDatastore9ExceptionE(Fault cause: vim.fault.InaccessibleDatastore
--> )
--> [context]zKq7AVECAQAAAAZLcQEWdnB4ZAAAAto3bGlidm1hY29yZS5zbwAAmXksABdtLQAf6jIBgfBxdnB4ZAABh1B0AUdKlQEGkZWBx6YaAYEBVhoBgv2fPgFsaWJ2aW0tdHlwZXMuc28AgfoBagGB9QRpAYE2BmkBgWQVaQGBx0JoAYGS7GgBAOdJIwB1nyMAwGU3A4d/AGxpYnB0aHJlYWQuc28uMAAEvzYPbGliYy5zby42AA==[/context]
2024-11-22T09:30:43.011+02:00 info vpxd[14066] [Originator@6876 sub=vpxLro opID=sps-Main-328281-438-260643-68] [VpxLRO] -- FINISH session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]5295ae6f-c3a1-4a97-566b-dbeae5883cc6
2024-11-22T09:30:43.011+02:00 info vpxd[14066] [Originator@6876 sub=Default opID=sps-Main-328281-438-260643-68] [VpxLRO] -- ERROR session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]5295ae6f-c3a1-4a97-566b-dbeae5883cc6 -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange: vim.fault.InaccessibleDatastore:
--> Result:
--> (vim.fault.InaccessibleDatastore) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = ,
--> datastore = 'vim.Datastore:3de21686-6c7f-40a3-b237-d2967fc43fe3:datastore-4770264',
--> name = "CSSA-IBOX1811-DEL_BSS_RES_CU1-14",
--> detail = "notAccessible"
--> msg = ""
--> }
--> Args:
-->
--> Arg catalogChangeSpec:
--> (vim.vslm.CatalogChangeSpec) {
--> datastore = 'vim.Datastore:datastore-4770264',
--> startVClockTime = (vim.vslm.VClockInfo) {
--> vClockTime = 0
--> },
--> fullSync = false
--> }
sps.log:
2024-11-22T09:30:47.124+02:00 [pool-28-thread-5] INFO opId=sps-Main-328281-438 com.vmware.vim.storage.common.util.OperationIdUtil - OperationID present in invoker thread, adding suffix and re-using it sps-Main-328281-438-260647
2024-11-22T09:30:47.124+02:00 [pool-2-thread-15] INFO opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Active thread count is: 1, Core Pool size is: 20, Queue size: 0, Time spent waiting in queue: 0 millis
2024-11-22T09:30:47.128+02:00 [pool-2-thread-15] INFO opId= com.vmware.vim.storage.common.task.CustomThreadPoolExecutor - [VLSI-client] Request took 4 millis to execute.
2024-11-22T09:30:48.588+02:00 [jaeger.RemoteReporter-QueueProcessor] WARN opId=sps-Main-328281-438 io.jaegertracing.internal.reporters.RemoteReporter - FlushCommand execution failed! Repeated errors of this command will not be logged.
io.jaegertracing.internal.exceptions.SenderException: Failed to flush spans.
... 3 more
Caused by: org.apache.thrift.transport.TTransportException: Cannot flush closed transport
at io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:151) ~[jaeger-thrift-1.6.0.jar:1.6.0]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[libthrift-0.14.1.jar:0.14.1]
at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) ~[libthrift-0.14.1.jar:0.14.1]
at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) ~[jaeger-thrift-1.6.0.jar:1.6.0]
at io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63) ~[jaeger-thrift-1.6.0.jar:1.6.0]
at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:84) ~[jaeger-thrift-1.6.0.jar:1.6.0]
at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[jaeger-thrift-1.6.0.jar:1.6.0]
... 3 more
Caused by: java.net.PortUnreachableException: ICMP Port Unreachable
at java.net.PlainDatagramSocketImpl.send(Native Method) ~[?:1.8.0_402]
at java.net.DatagramSocket.send(DatagramSocket.java:693) ~[?:1.8.0_402]
at io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:149) ~[jaeger-thrift-1.6.0.jar:1.6.0]
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[libthrift-0.14.1.jar:0.14.1]
at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) ~[libthrift-0.14.1.jar:0.14.1]
at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) ~[jaeger-thrift-1.6.0.jar:1.6.0]
at io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63) ~[jaeger-thrift-1.6.0.jar:1.6.0]
at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:84) ~[jaeger-thrift-1.6.0.jar:1.6.0]
at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[jaeger-thrift-1.6.0.jar:1.6.0]
VPXD.log:
2024-11-22T06:10:39.284+02:00 info vpxd[15009] [Originator@6876 sub=Default opID=sps-Main-328281-438-248639-a3] [VpxLRO] -- ERROR session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]52beeeb1-e5e4-a4ca-500a-6007919ad72d -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange: vim.fault.InaccessibleDatastore:
--> (vim.fault.InaccessibleDatastore) {
2024-11-22T06:10:55.229+02:00 warning vpxd[10433] [Originator@6876 sub=Vmomi opID=sps-Main-328281-438-248655-b1] VMOMI activation LRO failed; <<5265c884-c1ae-cc2c-2127-8a2431a0fc5a, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 38242'>>, CatalogSyncManager, vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange>, N3Vim5Fault21InaccessibleDatastore9ExceptionE(Fault cause: vim.fault.InaccessibleDatastore
2024-11-22T06:10:55.232+02:00 info vpxd[10433] [Originator@6876 sub=Default opID=sps-Main-328281-438-248655-b1] [VpxLRO] -- ERROR session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]5254621c-8381-9351-1086-d0cbea5ac02c -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange: vim.fault.InaccessibleDatastore:
--> (vim.fault.InaccessibleDatastore) {
2024-11-22T06:10:58.620+02:00 warning vpxd[08478] [Originator@6876 sub=Vmomi opID=sps-Main-328281-438-248658-f4] VMOMI activation LRO failed; <<5265c884-c1ae-cc2c-2127-8a2431a0fc5a, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 38242'>>, CatalogSyncManager, vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange>, N3Vim5Fault21InaccessibleDatastore9ExceptionE(Fault cause: vim.fault.InaccessibleDatastore
2024-11-22T06:10:58.624+02:00 info vpxd[08478] [Originator@6876 sub=Default opID=sps-Main-328281-438-248658-f4] [VpxLRO] -- ERROR session[5265c884-c1ae-cc2c-2127-8a2431a0fc5a]52b0773e-e498-b7a5-b3fc-eb0b955f4d77 -- CatalogSyncManager -- vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange: vim.fault.InaccessibleDatastore:
--> (vim.fault.InaccessibleDatastore) {
2024-11-22T06:11:31.736+02:00 warning vpxd[13565] [Originator@6876 sub=Vmomi opID=sps-Main-328281-438-248691-9d] VMOMI activation LRO failed; <<5265c884-c1ae-cc2c-2127-8a2431a0fc5a, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 38242'>>, CatalogSyncManager, vim.vslm.vcenter.CatalogSyncManager.queryCatalogChange>, N3Vim5Fault21InaccessibleDatastore9ExceptionE(Fault cause: vim.fault.InaccessibleDatastore
VMware vCenter Server 7.0
VMware vCenter Server 8.0
This type of issue can cause If Datastore utilization crossing the Utilization threshold value which is set on Storage DRS & its a feature of DRS.
Also, Storage DRS checks for rule violations when you create a virtual machine on a Storage DRS-enabled datastore. If Storage DRS cannot create the new virtual machine's disks in compliance with the rules, it generates a fault. The fault is generated because Storage DRS cannot reference the virtual machine, which is in the process of being created and does not yet exist.
or