Prometheus Data Collection from Kubernetes Cluster Stops Unexpectedly
search cancel

Prometheus Data Collection from Kubernetes Cluster Stops Unexpectedly

book

Article ID: 399369

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Data collection from the Kubernetes cluster via Prometheus stops abruptly.

Example screenshot:

Environment

8.12.1

Cause

The issue could be related to trust or connectivity problems between nodes in the Kubernetes integration. A warning is observed under the Kubernetes integration section in vRealize Operations, and the following error occurs during the "Test Validate Connection" process.

Error: Unable to access the base URL of the Kubernetes master – /api/v1

Integration-level warning message displayed on the adapters:

 

  • The error message during the "Test Validate Connection" points to a certificate mismatch. Also, the certificate thumbprint received from Kubernetes does not match with any of the trusted entries in vROPs, confirming that the new certificate is not yet imported.

 

  • The issue is happening because the certificates on the Kubernetes side have been rotated or changed, and the updated root CA certificate is not available in the vROPs trusted certificate store.
2025-06-02T07:50:03,662+0000 WARN  [Collector worker thread 10] (18941) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call to  failed on https://##.###.#.###:6443. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
2025-06-02T07:50:03,662+0000 ERROR [Collector worker thread 10] (18941) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Failed performGetStatusCode -io.kubernetes.client.openapi.ApiException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:888) ~[client-java-api-12.0.1.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode(KubernetesRestClient.java:501) ~[KubernetesAdapter3.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.testApi(KubernetesRestClient.java:107) ~[KubernetesAdapter3.jar:?]
        at com.vmware.vcops.adapter.main.KubernetesAdapter.onCollect(KubernetesAdapter.java:382) ~[KubernetesAdapter3.jar:?]
        at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:774) ~[vrops-adapters-sdk.jar:?]
        at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:559) ~[vrops-adapters-sdk.jar:?]
        at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:47) ~[vcops-collector-1.0-SNAPSHOT.jar:?]
        at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:275) ~[vrops-adapters-sdk.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
*************
*************
*************
2025-06-03T06:52:31,692+0000 ERROR [Collector worker thread 23] (18941) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call  failed after 2 retries
2025-06-03T06:52:31,692+0000 ERROR [Collector worker thread 23] (18941) com.vmware.vcops.adapter.main.KubernetesAdapter.onCollect - Exception while checking the status k8s status
2025-06-03T06:52:31,692+0000 WARN  [Collector worker thread 23] (18941) com.vmware.vcops.adapter.main.KubernetesAdapter.updateKubeAuthToken - Credentials are in-valid. Re-Obtaining the token
2025-06-03T06:52:31,708+0000 WARN  [Collector worker thread 23] (18941) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetJson - Api call to /api/v1/nodes failed on https://##.###.#.###:6443. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
2025-06-03T06:52:31,710+0000 WARN  [Collector worker thread 23] (18941) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetJson - Retrying...
2025-06-03T06:52:31,727+0000 WARN  [Collector worker thread 23] (18941) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetJson - Api call to /api/v1/nodes failed on https://##.###.#.###:6443. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
2025-06-03T06:52:31,727+0000 ERROR [Collector worker thread 23] (18941) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetJson - Api call /api/v1/nodes failed after 2 retries
2025-06-03T06:52:31,727+0000 ERROR [Collector worker thread 23] (18941) com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.createResourceManagers - Error in fetching nodes/pods/namespaces
2025-06-03T06:52:31,727+0000 ERROR [Collector worker thread 23] (18941) com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.collect - Couldn't find any objects in the target environment. Please make sure that the adapter settings are correct.
2025-06-03T06:57:32,753+0000 WARN  [Collector worker thread 15] (18941) com.vmware.vcops.adapter.main.KubernetesAdapter.onCollect - Collection cycle start
2025-06-03T06:57:32,790+0000 WARN  [Collector worker thread 15] (18941) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call to  failed on https://##.###.#.###:6443. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
2025-06-03T06:57:32,790+0000 ERROR [Collector worker thread 15] (18941) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Failed performGetStatusCode-


cd logs/collector.log.11

2025-05-15T13:58:46,003+0000 WARN  [Task Processor worker thread 1]  com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Retrying...
2025-05-15T13:58:46,019+0000 WARN  [Task Processor worker thread 1]  com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call to  failed on https://##.###.#.###:6443. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
2025-05-15T13:58:46,019+0000 ERROR [Task Processor worker thread 1]  com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Failed performGetStatusCode -
io.kubernetes.client.openapi.ApiException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:888) ~[client-java-api-12.0.1.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode(KubernetesRestClient.java:501) ~[KubernetesAdapter3.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.testApi(KubernetesRestClient.java:107) ~[KubernetesAdapter3.jar:?]
        at com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.test(KubernetesDataProvider.java:88) ~[KubernetesAdapter3.jar:?]
        at com.vmware.vcops.adapter.main.KubernetesAdapter.onTest(KubernetesAdapter.java:267) ~[KubernetesAdapter3.jar:?]
        at com.integrien.alive.common.adapter3.AdapterBase.test(AdapterBase.java:1305) ~[vrops-adapters-sdk.jar:?]
        at com.integrien.alive.collector.Collector.testConnection(Collector.java:1275) ~[vcops-collector-1.0-SNAPSHOT.jar:?]
        at com.integrien.alive.collector.CollectorTaskHandler.handleTestConnection(CollectorTaskHandler.java:271) ~[vcops-collector-1.0-SNAPSHOT.jar:?]
        at com.integrien.alive.common.communication.task.TaskTest.processTask(TaskTest.java:39) ~[alive_platform.jar:?]
        at com.integrien.alive.common.communication.CommunicatorWorkItem.run(CommunicatorWorkItem.java:37) ~[alive_platform.jar:?]
        at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:275) ~[vrops-adapters-sdk.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(Unknown Source) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(Unknown Source) ~[?:?]
*************
*************
*************
2025-05-15T13:58:46,021+0000 ERROR [Task Processor worker thread 1]  com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call  failed after 2 retries
2025-05-15T13:58:46,021+0000 ERROR [Task Processor worker thread 1]  com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.test - Error in accessing the base url of Kubernetes master - /api/v1java.lang.NullPointerException: null
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode(KubernetesRestClient.java:519) ~[KubernetesAdapter3.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.testApi(KubernetesRestClient.java:107) ~[KubernetesAdapter3.jar:?]
        at com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.test(KubernetesDataProvider.java:88) ~[KubernetesAdapter3.jar:?]
        at com.vmware.vcops.adapter.main.KubernetesAdapter.onTest(KubernetesAdapter.java:267) ~[KubernetesAdapter3.jar:?]
        at com.integrien.alive.common.adapter3.AdapterBase.test(AdapterBase.java:1305) ~[vrops-adapters-sdk.jar:?]
        at com.integrien.alive.collector.Collector.testConnection(Collector.java:1275) ~[vcops-collector-1.0-SNAPSHOT.jar:?]
        at com.integrien.alive.collector.CollectorTaskHandler.handleTestConnection(CollectorTaskHandler.java:271) ~[vcops-collector-1.0-SNAPSHOT.jar:?]
        at com.integrien.alive.common.communication.task.TaskTest.processTask(TaskTest.java:39) ~[alive_platform.jar:?]
        at com.integrien.alive.common.communication.CommunicatorWorkItem.run(CommunicatorWorkItem.java:37) ~[alive_platform.jar:?]
        at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:275) ~[vrops-adapters-sdk.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]

Resolution

  • Import the root CA certificate in vROPs using the option under Control Panel → Trusted Certificates.

  • After importing, navigate to the adapter configuration, verify the adapter status by expanding Data Sources → Integrations → Kubernetes, and use the Test Connection option for each nodes and accept the certificate when prompted and save the settings. 

  • This allows vROPs to establish a secure and trusted connection with the Kubernetes API, which clears the adapter warning, enables successful communication during validation, and potentially resolves the data collection issue.

 

Additionally, wait for 2-3 collection cycles until the adapter instance is in a data-receiving state. This allows the system to stabilize and ensures that the integration is functioning properly. While restarting the Prometheus deployment temporarily restores the collection, validating the connection and allowing the adapter to sync will provide a more reliable long-term solution.

VMware Aria Operations for Integrations 2.2

Powered by Wolken Software