App monitoring agent stops post content upgrade in vROps

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Telegraf agents have been installed on VM's on vRealize Operations 7.5, 8.0.x, or 8.1.x with plugin(s) activated for applications like activemq, java, jboss, pivotalserver, tomcat and weblogic.
• vRealize Operations has been upgraded to 8.4 or later, with a cloud proxy node deployed and ARC migration performed.
• A Content Upgrade has been preformed on the VMs.
• Post content upgrade, agent turns to stopped state and telegraf stops posting metrics to vRealize Operations for VM's.

Environment

VMware vRealize Operations 8.x

Cause

Applications supporting jolokia inputs.jolokia2_agent.metrics telegraf plugins in 7.5, 8.0.x, or 8.1.x versions of vRealize Operations are no longer supported in latest telegraf version.

When a content upgrade of an endpoint is performed post vRealize Operations upgrade from 7.5, 8.0.x, or 8.1.x to 8.4 or later, the endpoints telegraf will update and the telegraf service version will fail to restart due to the unsupported jolokia plugin.

Resolution

As part of the content upgrade, remove all occurrences of unsupported plugins.

Workaround:
To attempt to workaround this issue, update the update telegraf.conf from jolokia2_agent.metrics to jolokia2_agent.metric using sls scripts.

1. Identify the vc_id and vm_id for all the Virtual Machines where the issue has occurred.

Note: vc_id and vm_id can be identified either from vRealize Operations UI or from the endpoint itself.

• vRealize Operations UI: Navigate to Administration > Inventory.  Click the Columns icon in the lower left corner of the right pane and check Identifier 2 and Identifier 3, then click OK.  Enter the VM name in the Filter field and note the vc_id from the Identifier 3 column, and vm_id from the Identifier 2 column.
• Linux VM: View the /opt/vmware/ucp/salt-minion/etc/salt/grains file and check for the vc_id and vm_id values.
• Windows VM: View the C:\VMware\UCP\salt\conf\grains file and check for the vc_id and vm_id values.

2. Log into the Cloud Proxy node as root via SSH.
3. Download the attached jolokiaapplicationupgrade.sls file and place it in the /ucp/salt/srv/salt/ folder on the Cloud Proxy node using an SCP utility.
4. Apply the salt state by running the following command:

docker exec -it ucp-controlplane-saltmaster bash -c "salt 'vcid_vmid' state.apply jolokiaapplicationupgrade"

Note: Replace vcid and vmid with the vc_id and vm_id on a Virtual Machine you identified in step 1.

Example: docker exec -it ucp-controlplane-saltmaster bash -c "salt '4439a09d-19ac-40ce-921a-1e7b1db71467_vm-155' state.apply jolokiaapplicationupgrade"

5. Repeat step 4 for any other Virtual Machines experiencing the issue.