Tanzu Application Service (TAS) for VMs update or upgrade failing at loggregator_trafficcontroller on reverse_log_proxy job

Products

• Operations Manager

Issue/Introduction

This issue has been observed in Tanzu Application Service (TAS) for VMs 2.7.x and TAS for VMs 2.8.x, and may affect previous or later versions of TAS for VMs.

Symptoms

During "Install Pivotal App Service" stage, Apply Changes fails with the following error while updating loggregator_trafficcontroller:

Task 115131 | 17:40:20 | Updating instance loggregator_trafficcontroller: loggregator_trafficcontroller/fb22f8d4-258c-4fe2-9148-a9bea968767b (1) (00:06:43)
                      L Error: 'loggregator_trafficcontroller/fb22f8d4-258c-4fe2-9148-a9bea968767b (1)' is not running after update. Review logs for failed jobs: reverse_log_proxy

Cause

One of the potential causes is because the reverse_log_proxy (RLP) job can not stop while updating the Traffic Controller VM. The RLP job can not update because its process can not stop which leads to the failed deployment.

Environment

Product Version: 2.8

Resolution

Resolution

To manually bring the Traffic Controller VM back into a state where it can be updated, you must manually stop the RLP process.

1. SSH onto the affected loggregator_trafficcontroller vm and become sudo.

2. Run the command: monit stop reverse_log_proxy

3. Run the command: ps aux | grep rlp



Locate the process ID for the RLP and kill it.

4. Run the command: kill -9 <PID of rlp>

5. Run the command: monit start reverse_log_proxy

 

Note: If you have many Traffic Controller VMs, you may wish to verify the all of them before running Apply Changes to avoid lengthy failures. This can be done by proactively checking each Traffic Controller VM. SSH onto each one and perform a monit stop reverse_log_proxy. Give it 15-20 seconds to stop.

Perform $ ps aux | grep rlp to see if the process is still running. If the rlp process is still running even after monit says it is not, proceed to run, kill -9 <PID of rlp> and then $ monit start reverse_log_proxy.

Once the RLP process has been manually restarted (via the above methods of force killing the process) on the affected machine, an Apply Changes shall be able to update it successfully.

The TAS versions with the patch for this are: 2.7.16+, 2.8.10+, 2.9.4+, 2.10+