Product Managed Telegraf agents disappear from the Manage Telegraf Agents page and stop collecting metrics
search cancel

Product Managed Telegraf agents disappear from the Manage Telegraf Agents page and stop collecting metrics

book

Article ID: 443211

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

Telegraf agents installed on Product-Managed Windows Server virtual machines intermittently fail to collect application metrics (such as IIS). These managed virtual machine objects experience UI desynchronization, where they stop collecting, disappear from the "Manage Telegraf Agents" page, and reappear hours later. The local Telegraf agent services on the virtual machines remain in a healthy state during these drops.

Manually deleting the failing Microsoft IIS object from the UI and re-adding it immediately restores collection.

Additionally, the APPOSUCPAdapter logs on the Cloud Proxy reveal a high volume of rejected payloads with the following error:

com.vmware.adapter3.applicationdiscovery.handler.EndpointStatusHandler.lambda$handleEndPointStatusBasedOnUcpMinion$3] - Ignoring data received from - {resName=[WINDOWS_SERVER_NAME], resKind=Endpoint, adKind=APPOSUCP, ids={NAME=[WINDOWS_SERVER_NAME], VCID=_, VMMOR=[GUID]}}, since corresponding resource in vROps is not found

NOTE: [WINDOWS_SERVER_NAME] is a physical Windows server with an open-source telegraf agent installed

Environment

  • VCF Operations 9.0.x

  • OS and Application Monitoring Adapter

  • Product-managed Telegraf agents (Virtual Machines)

  • Open-source Telegraf agents (Physical Servers)

Cause

The intermittent UI drops are caused by APPOSUCPAdapter worker thread exhaustion due to an inventory mapping failure originating from physical servers that have a misconfigured Open Source Telegraf agent.

Physical servers deployed with Open-Source Telegraf agents send payloads with identifiers that do not match any Endpoint inventory objects. The Cloud Proxy that contains the OS and Application Monitoring Adapter successfully receives the network traffic but actively drops it because the corresponding resource in VCF Operations is not found. The continuous influx of this unmapped open-source data bombards the UCP listener on the Cloud Proxy, exhausting the adapter's processing queues. Consequently, legitimate Managed Endpoint Services (MEpS) heartbeats and Service Discovery payloads from healthy Product-Managed virtual machines are delayed or dropped, leading to UI state desynchronization.

Resolution

Standardize the architecture by replacing the unmanaged open-source agents with the Product-Managed distribution across all physical endpoints. Follow these steps in order:

  1. Uninstall the open-source Telegraf agents from all physical Windows servers to immediately relieve the APPOSUCPAdapter queues.

    Official uninstallation procedures for the open-source Telegraf agent are not documented, as support is currently limited to the initial installation. However, the following methods can be used to attempt removing the agent:

    • Method 1: Helper Script Flag Since both product-managed and open-source leverage the same underlying script (https://<CloudProxy-IP>/downloads/salt/telegraf-utils.ps1), you can attempt to pass the -o uninstall parameter which is documented for the product-managed agent: Install/Uninstall an Agent Using a Script on a Windows Platform

      Example: telegraf-utils.ps1 opensource -o uninstall ...

    • Method 2: Legacy Manual Removal If the script flag does not work, follow the manual removal steps outlined in this 8.x KB article: Manually Remove Telegraf Agents from Endpoints.

    NOTE: To completely clear the open source agent remnants after following the method(s) above, it may require a reboot of the physical Windows server.

  2. Install the Product-Managed Telegraf agent onto the physical Windows servers using the documented Windows deployment script. 

     

  3. In the VCF Operations UI, navigate to Administration > Integrations > OS and Application Monitoring.

  4. Restart the affected adapter instance by expanding the adapter instance > click the ellipses (3 dots) > Stop Collecting. Wait for up to 5 minutes for it to show Stopped status, then repeat, choosing Start Collecting

  5. (Optional) Log into the affected Product-Managed Windows Server virtual machines experiencing UI drops and restart the local minion service using PowerShell to force an immediate data collection:

    Restart-Service -Name "ucp-minion"
Restart-Service -DisplayName "Application Remote Collector Minion"

Additional Information

Powered by Wolken Software