Endpoint VMs Unable to Send Metrics to Cloud Proxy via Telegraf in Brownfield Deployments

Products

• VCF Operations

Issue/Introduction

In brownfield deployments of VCF Operations, Telegraf agent configured Virtual Machines (VMs) are unable to successfully transmit metrics to the Cloud Proxy. This metric transmission failure does not occur in newly provisioned greenfield deployments.

Environment

VCF Operations 9.0.x
VCF Operations Cloud Proxy 9.0.x

Cause

In brownfield deployments, a newly generated CA certificate on the Cloud Proxy invalidates the existing certificate trust established by the endpoint VMs. The endpoints are subsequently unable to authenticate and send metrics.

Resolution

This is a known issue in 9.0.x and will be fixed in future releases.

Workaround: 

To restore metric collection, you must manually delete the stale certificate artifacts for the specific endpoint VM from the Cloud Proxy and reinstall the Telegraf agent.

1. Identify the targeted VM's vCenter Server ID and VM Managed Object Reference ID (MoRef ID / vmoid) from the vCenter Server UI. The resulting string format will be <vCenterID>_<vm-id>.



2. Connect to the VCF Operations Cloud Proxy via SSH and authenticate as the root user.

3. Execute the following commands to remove the existing certificate, Certificate Signing Request (CSR), and private key associated with the endpoint VM. Replace <vCenterID_vm-id> with the string identified in Step 1:

   cd /ucp/ssl/root/ca/certs/

  rm -rf <vCenterID_vm-id>.cert.pem

  cd /ucp/ssl/root/ca/csr/

  rm -rf <vCenterID_vm-id>.csr.pem

  cd /ucp/ssl/root/ca/private/

  rm -rf <vCenterID_vm-id>.*

  cd /ucp/ssl/root/ca/

  vi index.txt

Note: If vcenter ID is 6d012033-133f-44cb-bae8-0e29b2684837 and VM ID is vm-32, then <vCenter_vm-id> is 6d012033-133f-44cb-bae8-0e29b2684837_vm-32
   
4. Locate the text entry corresponding to the targeted <vCenterID_vm-id>. Delete that entry, then save and close the file.

5. Remove any remaining SSL artifacts for the VM from the primary SSL directory:

    rm -rf /ucp/ssl/<vCenterID_vm-id>*
   
6. Reinstall the Telegraf agent on the affected endpoint VM via the UI. When prompted, ensure the checkbox to retain existing Telegraf configurations is set to True to successfully restore the brownfield configurations.