The current status of Internet connection is disabled. Online health checks are not available if the internet connection is disabled | Unable to query vSphere health information

Products

VMware vCenter Server 8.0 VMware vCenter Server 7.0 VMware vCenter Server 6.0

Issue/Introduction

Impact/Risks:
SSL encryption is used to hide dangerous content such as viruses, spyware, and other malware, thus be cautious and ask the customer to confirm whether their security team will allow to disable SSL inspection for VCSA.

To address the issue that may prevent Skyline Health from accessing the internet to connect with the necessary repositories essential for the proper functioning of Skyline Health.

Symptoms:

Within Skyline Health(either on vCenter level or Host level), the status indicates that online health connectivity is not available. In the Information section, it is reported as "The present condition of the internet connection is inactive. Online health checks cannot be performed in the absence of an active internet connection.

When a RETEST in initiated, it may take a few minutes to complete the task, and the result will be: "Unable to retrieve vSphere health data. For more information, consult the vSphere Client logs."

**Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment where the task is run.

Errors may be seen similar to log messages show below:

**Note: The following logs can be seen if any proxy server conducts SSL negotiations with the user’s browser, i.e it presents its own certificate for the vCSA traffic before the traffic can reach the internet.(See Related section for more details)

/var/log/vmware/vsan-health/vmware-vsan-health-service.log:

2023-09-21T11:29:03.730Z INFO vsan-mgmt[09700] [VsanVcClusterHealthSystemImpl::_generateHealthGroupResult opID=noOpId] Start health check for group wcp
2023-09-21T11:29:04.018Z INFO vsan-mgmt[18540] [VsanSupportBundleHelper::parseSystemProxies opID=noOpId] VCSA proxy is disabled.
2023-09-21T11:29:04.079Z ERROR vsan-mgmt[18540] [VsanHttpRequestWrapper::urlopen opID=noOpId] Exception while sending request : <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)>
2023-09-21T11:29:04.080Z INFO vsan-mgmt[18540] [VsanSupportBundleHelper::parseSystemProxies opID=noOpId] VCSA proxy is disabled.
2023-09-21T11:29:04.141Z ERROR vsan-mgmt[18540] [VsanHttpRequestWrapper::urlopen opID=noOpId] Exception while sending request : <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

/var/log/vmware/analytics/analytics.log:

2023-09-21T12:04:40.326Z phStageLogDrainerTaskExecutor-2 INFO org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi Initializing with trust store at path: /usr/java/jre-vmware/lib/security/cacerts
2023-09-21T12:04:40.332Z phStageLogDrainerTaskExecutor-2 WARN vmware.ph.upload.rest.ProxyAwareHttpExecutor Failed 3 of 3 attempts to get manifest from VMware server. The reason for the last failure was (enable 'debug' level logging to see the error of every failed attempt): com.vmware.ph.upload.exception.ConnectionException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46).
2023-09-21T12:04:40.332Z phStageLogDrainerTaskExecutor-2 ERROR ph.phservice.push.telemetry.DefaultTelemetryLevelService Unexpected error during telemetry level retrieval for CollectorAgent: {collectorId:vcenter-all.vpxd.hdcs.7_0u2, collectorInstanceId:ph-vpxd-a5a6c74b-039f-4a56-97da-99d98bcfd3e1} java.util.concurrent.CompletionException: com.vmware.ph.phservice.common.manifest.ManifestContentProvider$ManifestException: com.vmware.ph.upload.exception.ConnectionException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
        at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:147)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2404)
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1877)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2377)
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2360)
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
        at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:54)
        at com.vmware.ph.phservice.push.telemetry.DefaultTelemetryLevelService.getTelemetryLevelFromCache(DefaultTelemetryLevelService.java:98)
        at com.vmware.ph.phservice.push.telemetry.DefaultTelemetryLevelService.getTelemetryLevel(DefaultTelemetryLevelService.java:91)
        at com.vmware.ph.phservice.push.telemetry.internal.log.LogTelemetryDrainerExecutor$1.run(LogTelemetryDrainerExecutor.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: com.vmware.ph.phservice.common.manifest.ManifestContentProvider$ManifestException: com.vmware.ph.upload.exception.ConnectionException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
        at com.vmware.ph.phservice.common.ph.UnsignedManifestContentProvider.getManifestContent(UnsignedManifestContentProvider.java:50)
        at com.vmware.ph.phservice.common.internal.manifest.PropertyControlledManifestContentProviderWrapper.getManifestContent(PropertyControlledManifestContentProviderWrapper.java:53)
        at com.vmware.ph.phservice.push.telemetry.DefaultTelemetryLevelService.getTelemetryLevelInfo(DefaultTelemetryLevelService.java:122)
        at com.vmware.ph.phservice.push.telemetry.DefaultTelemetryLevelService.access$000(DefaultTelemetryLevelService.java:42)
        at com.vmware.ph.phservice.push.telemetry.DefaultTelemetryLevelService$2.load(DefaultTelemetryLevelService.java:222)
        at com.vmware.ph.phservice.push.telemetry.DefaultTelemetryLevelService$2.load(DefaultTelemetryLevelService.java:218)
        at com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:140)
        ... 12 more
Caused by: com.vmware.ph.upload.exception.ConnectionException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
        at com.vmware.ph.upload.rest.PhRestClientImpl.getManifest(PhRestClientImpl.java:174)
        at com.vmware.ph.client.api.impl.PhClientImpl.lambda$getManifest$2(PhClientImpl.java:334)
        at com.vmware.ph.upload.rest.JitteringRepeatableInvocationStrategy.invoke(JitteringRepeatableInvocationStrategy.java:66)
        at com.vmware.ph.upload.rest.ProxyAwareHttpExecutor.executeWithRetry(ProxyAwareHttpExecutor.java:76)
        at com.vmware.ph.client.api.impl.PhClientImpl.getManifest(PhClientImpl.java:333)
        at com.vmware.ph.phservice.common.ph.UnsignedManifestContentProvider.getManifestContent(UnsignedManifestContentProvider.java:42)
        ... 18 more
Caused by: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)
        at org.bouncycastle.jsse.provider.ProvSSLSocketWrap.checkServerTrusted(Unknown Source)
        at org.bouncycastle.jsse.provider.ProvTlsClient$1.notifyServerCertificate(Unknown Source)
        at org.bouncycastle.tls.TlsUtils.processServerCertificate(Unknown Source)
        at org.bouncycastle.tls.TlsClientProtocol.handleServerCertificate(Unknown Source)
        at org.bouncycastle.tls.TlsClientProtocol.handleHandshakeMessage(Unknown Source)
        at org.bouncycastle.tls.TlsProtocol.processHandshakeQueue(Unknown Source)
        at org.bouncycastle.tls.TlsProtocol.processRecord(Unknown Source)

Environment

vCenter Server 7.x
vCenter Server 8.x

Cause

The vCenter Server Appliance (vCSA) may experience difficulties communicating with the internet, which can impact its proper functionality. This issue can arise due to network configurations or restrictions such as firewalls, gateways, or proxy servers that the vCSA must pass through before accessing the internet. Additionally, if the Customer Improvement Experience Program (CIEP) is disabled at the vCenter (VC) level, this may also lead to similar connectivity issues.

Resolution

- Enable Customer Improvement Experience Program- CIEP

- Ensure that vCSA is able to get on the internet to access whatever it needs, check this using tools like - ping, nc (netcat), opennssl, curl, or wget.

- Add wildcard entries or whitelist the following URL's in the gateway/firewall/proxy server(proxy vendor support required):

vmware.com
vcsa.vmware.com
vapp-updates.vmware.com
shd-download.vmware.com
partnerweb.vmware.com

- *Disable SSL inspection on the Proxy Server(this may or may not be related in every case, this can be checked with the proxy server vendor whether they have SSL Inspection enabled for vCSA traffic) <<<<Do it only if whitelisting the above URL's doesn't fix the issue

- Restart the analytics server: vmoncli -r analytics

- Perform retests on Skyline Health a few times before concluding that the issue remains unresolved, as it might take some time for Skyline Health to establish connections with the repositories and retrieve the necessary data.