Error" java.net.SocketException: Too many open files " while validating the test connection for vCenter adapter in Aria Operations.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• In Aria Operations UI > Capacity > Reclaim > Cluster, capacity metrics for a particular cluster are not being populated.

• The vCenter Adapter (associated with the affected cluster) is in a WARNING state.

• Test Connection for the adapter shows "java.net.SocketException: Too many open files" intermittent success or reports "Host is not reachable".

• Authentication credentials may be reported as incorrect.

• vCenter Adapter /storage/log/vcops/log/adapters/logs/VMware_adapter_#### displays an error: "java.net.SocketException: Too many open files"

        ERROR [TasksManager-TaskHandler-26160] (227277) com.vmware.vcops.vlsi.VcSessions.terminateSession - Cannot Obtain the session:
        com.vmware.vim.vmomi.client.exception.ConnectionException: https://##.##.##.##/sdk invocation failed with "java.net.SocketException: Too many open files"
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:265) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:355) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:59) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.core.tracing.NoopTracer$NoopSpan.runWithinSpanContext(NoopTracer.java:120) ~[vlsi-core-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:17) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:54) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:119) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeFetch(MethodInvocationHandlerImpl.java:347) ~[vlsi-client-8.0.2.jar:?]
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:191) ~[vlsi-client-8.0.2.jar:?]
        at com.sun.proxy.$Proxy333.getCurrentSession(Unknown Source) ~[?:?]
        at com.vmware.vcops.vlsi.VcSessions.terminateSession(VcSessions.java:286) ~[vcops-vlsi-1.0-SNAPSHOT.jar:?]
        at com.integrien.adapter.vmware.VMwareAdapter.vcDisconnect(VMwareAdapter.java:4440) ~[vmwarevi_adapter3.jar:?]
        at com.integrien.adapter.vmware.VMwareAdapter.resetVimClient(VMwareAdapter.java:3541) ~[vmwarevi_adapter3.jar:?]
        at com.integrien.adapter.vmware.drs.ConnectionHelper.resetVimClient(ConnectionHelper.java:94) ~[vmwarevi_adapter3.jar:?]
        at com.integrien.adapter.vmware.drs.ClusterRulesHandler.stop(ClusterRulesHandler.java:595) ~[vmwarevi_adapter3.jar:?]
        at com.integrien.adapter.vmware.VMwareAdapter.shutdownDRSRulesManager(VMwareAdapter.java:4662) ~[vmwarevi_adapter3.jar:?]

Environment

Aria Operations 8.18.x 

Cause

• The issue arises due to a connectivity failure between Aria Operations and the configured vCenter endpoint, caused by a 'Too many open files' condition on Aria Operations.
• This resource exhaustion prevents new sessions from being established, leading to a failure in collecting capacity data for the affected cluster.

Resolution

To resolve the issue, follow these steps:

1. Take a snapshot of all Aria Operations nodes following Snapshot Creation in VMware Aria Operations

2. Reboot the Aria Operations cluster

• • If you are running a single-node Aria Operations deployment, simply reboot the node.

  • If you are using a multi-node cluster, follow the KB Shutdown and Startup sequence for Aria Operations cluster for the correct shutdown and startup sequence for Aria Operations clusters to reboot the nodes in the proper order.

After rebooting, bring the cluster online through the admin UI and perform the following:

• 1. Navigate to Administration > Integrations.

  2. Locate and select the affected vCenter Adapter.

  3. Click Edit and then Validate Connection.

  4. When prompted to import the associated certificate, click OK.

  5. Save the configuration.

  6. Finally, Stop and then Start data collection on the adapter.

  7. Verify the cluster metric is visible in Aria Operations UI > Capacity > Reclaim > Cluster > Select cluster.