Prometheus Data Collection from Kubernetes Cluster Stops Unexpectedly
search cancel

Prometheus Data Collection from Kubernetes Cluster Stops Unexpectedly

book

Article ID: 399369

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  1. Data collection from the Kubernetes cluster via Prometheus stops abruptly.

Example screenshot:

Environment

8.12.1

Cause

  1. The issue could be related to trust or connectivity problems between nodes in the Kubernetes integration. A warning is observed under the Kubernetes integration section in vRealize Operations, and the following error occurs during the "Test Validate Connection" process.

    Error: Unable to access the base URL of the Kubernetes master – /api/v1

    Integration-level warning message displayed on the adapters:




  2. The error message during the "Test Validate Connection" points to a certificate mismatch. Also, the certificate thumbprint received from Kubernetes does not match with any of the trusted entries in vROPs, confirming that the new certificate is not yet imported.e



  3. The issue is happening because the certificates on the Kubernetes side have been rotated or changed, and the updated root CA certificate is not available in the vROPs trusted certificate store.
####-##-##T##:##:##,###+#### WARN  [Collector worker thread ##] (#####) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call to  failed on https://##.###.#.###:#####. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
####-##-##T##:##:##,###+#### ERROR [Collector worker thread ##] (#####) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Failed performGetStatusCode -io.kubernetes.client.openapi.ApiException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:###) ~[client-java-api-##.#.#.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode(KubernetesRestClient.java:###) ~[KubernetesAdapter3.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.testApi(KubernetesRestClient.java:###) ~[KubernetesAdapter#.jar:?]
        at com.vmware.vcops.adapter.main.KubernetesAdapter.onCollect(KubernetesAdapter.java:###) ~[KubernetesAdapter#.jar:?]
        at com.integrien.alive.common.adapter#.AdapterBase.collectBase(AdapterBase.java:###) ~[vrops-adapters-sdk.jar:?]
        at com.integrien.alive.common.adapter#.AdapterBase.collect(AdapterBase.java:###) ~[vrops-adapters-sdk.jar:?]
        at com.integrien.alive.collector.CollectorWorkItem#.run(CollectorWorkItem#.java:##) ~[vcops-collector-#.#-SNAPSHOT.jar:?]
        at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:###) ~[vrops-adapters-sdk.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
*************
*************
*************
####-##-##T##:##:##,###+#### ERROR [Collector worker thread ##] (#####) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call  failed after # retries
####-##-##T##:##:##,###+#### ERROR [Collector worker thread ##] (#####) com.vmware.vcops.adapter.main.KubernetesAdapter.onCollect - Exception while checking the status k8s status
####-##-##T##:##:##,###+#### WARN  [Collector worker thread ##] (#####) com.vmware.vcops.adapter.main.KubernetesAdapter.updateKubeAuthToken - Credentials are in-valid. Re-Obtaining the token
####-##-##T##:##:##,###+#### WARN  [Collector worker thread ##] (#####) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetJson - Api call to /api/v1/nodes failed on https://##.###.#.###:####. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
####-##-##T##:##:##,###+#### WARN  [Collector worker thread ##] (#####) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetJson - Retrying...
####-##-##T##:##:##,###+#### WARN  [Collector worker thread ##] (#####) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetJson - Api call to /api/v1/nodes failed on https://##.###.#.###:####. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
####-##-##T##:##:##,###+#### ERROR [Collector worker thread ##] (#####) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetJson - Api call /api/v1/nodes failed after # retries
####-##-##T##:##:##,###+#### ERROR [Collector worker thread ##] (#####) com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.createResourceManagers - Error in fetching nodes/pods/namespaces
####-##-##T##:##:##,###+#### ERROR [Collector worker thread ##] (#####) com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.collect - Couldn't find any objects in the target environment. Please make sure that the adapter settings are correct.
####-##-##T##:##:##,###+#### WARN  [Collector worker thread ##] (#####) com.vmware.vcops.adapter.main.KubernetesAdapter.onCollect - Collection cycle start
####-##-##T##:##:##,###+#### WARN  [Collector worker thread ##] (#####) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call to  failed on https://##.###.#.###:6443. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
####-##-##T##:##:##,###+#### ERROR [Collector worker thread ##] (#####) com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Failed performGetStatusCode-


cd logs/collector.log.##

####-##-##T##:##:##,###+#### WARN  [Task Processor worker thread #]  com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Retrying...
####-##-##T##:##:##,###+#### WARN  [Task Processor worker thread #]  com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call to  failed on https://##.###.#.###:####. Exception - javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors.
####-##-##T##:##:##,###+#### ERROR [Task Processor worker thread #]  com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Failed performGetStatusCode -
io.kubernetes.client.openapi.ApiException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:###) ~[client-java-api-##.#.#.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode(KubernetesRestClient.java:###) ~[KubernetesAdapter#.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.testApi(KubernetesRestClient.java:###) ~[KubernetesAdapter#.jar:?]
        at com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.test(KubernetesDataProvider.java:##) ~[KubernetesAdapter#.jar:?]
        at com.vmware.vcops.adapter.main.KubernetesAdapter.onTest(KubernetesAdapter.java:###) ~[KubernetesAdapter#.jar:?]
        at com.integrien.alive.common.adapter#.AdapterBase.test(AdapterBase.java:####) ~[vrops-adapters-sdk.jar:?]
        at com.integrien.alive.collector.Collector.testConnection(Collector.java:####) ~[vcops-collector-#.#-SNAPSHOT.jar:?]
        at com.integrien.alive.collector.CollectorTaskHandler.handleTestConnection(CollectorTaskHandler.java:###) ~[vcops-collector-#.#-SNAPSHOT.jar:?]
        at com.integrien.alive.common.communication.task.TaskTest.processTask(TaskTest.java:##) ~[alive_platform.jar:?]
        at com.integrien.alive.common.communication.CommunicatorWorkItem.run(CommunicatorWorkItem.java:##) ~[alive_platform.jar:?]
        at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:###) ~[vrops-adapters-sdk.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(Unknown Source) ~[?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(Unknown Source) ~[?:?]
*************
*************
*************
####-##-##T##:##:##,###+#### ERROR [Task Processor worker thread #]  com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode - Api call  failed after # retries
####-##-##T##:##:##,###+#### ERROR [Task Processor worker thread #]  com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.test - Error in accessing the base url of Kubernetes master - /api/v1java.lang.NullPointerException: null
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.performGetStatusCode(KubernetesRestClient.java:###) ~[KubernetesAdapter#.jar:?]
        at com.vmware.vcops.adapter.utils.KubernetesRestClient.testApi(KubernetesRestClient.java:###) ~[KubernetesAdapter#.jar:?]
        at com.vmware.vcops.adapter.dataprovider.KubernetesDataProvider.test(KubernetesDataProvider.java:##) ~[KubernetesAdapter#.jar:?]
        at com.vmware.vcops.adapter.main.KubernetesAdapter.onTest(KubernetesAdapter.java:###) ~[KubernetesAdapter#.jar:?]
        at com.integrien.alive.common.adapter#.AdapterBase.test(AdapterBase.java:####) ~[vrops-adapters-sdk.jar:?]
        at com.integrien.alive.collector.Collector.testConnection(Collector.java:####) ~[vcops-collector-#.#-SNAPSHOT.jar:?]
        at com.integrien.alive.collector.CollectorTaskHandler.handleTestConnection(CollectorTaskHandler.java:###) ~[vcops-collector-#.#-SNAPSHOT.jar:?]
        at com.integrien.alive.common.communication.task.TaskTest.processTask(TaskTest.java:##) ~[alive_platform.jar:?]
        at com.integrien.alive.common.communication.CommunicatorWorkItem.run(CommunicatorWorkItem.java:##) ~[alive_platform.jar:?]
        at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:###) ~[vrops-adapters-sdk.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]

Resolution

  1. Import the root CA certificate in vROPs using the option under Control Panel → Trusted Certificates.

  2. After importing, navigate to the adapter configuration, verify the adapter status by expanding Data Sources → Integrations → Kubernetes, and use the Test Connection option for each nodes and accept the certificate when prompted and save the settings. 

  3. This allows vROPs to establish a secure and trusted connection with the Kubernetes API, which clears the adapter warning, enables successful communication during validation, and potentially resolves the data collection issue.

Note: Additionally, wait for 2-3 collection cycles until the adapter instance is in a data-receiving state. This allows the system to stabilize and ensures that the integration is functioning properly. While restarting the Prometheus deployment temporarily restores the collection, validating the connection and allowing the adapter to sync will provide a more reliable long-term solution.

VMware Aria Operations for Integrations 2.2

Powered by Wolken Software