Usage-service job fails and needs to be re-queued

Products

• VMware Tanzu Application Service for VMs
• VMware Tanzu Application Service
• VMware Tanzu Application Service

Issue/Introduction

This situation happens when an Usage service job fails to trigger for some reason and needs to be manually re-queued. The customer will notice that data in the reports appears inaccurate or empty. The solution described in this article is a simpler approach to resolution than the previously used purge and reseed process which consists of deleting data and re-importing to the get the jobs to be executed again.

Cause

There can be multiple different reasons why a usage-service job can fail to be queued by the usage-service-workers. This article covers a general solution that can help re-queue a failed job independently of the cause of the queuing failure.

Resolution

In order to solve this problem the jobs needs to be re-queued. To do this it is necessary to access the ruby on rails console CLI. Below you will find the steps that go over the process of accessing the console and re-queuing the failed jobs.

 

Step 1:
Log into the rails console. Use the CF CLI to enter the ruby on rails console to find the failed delayed jobs and restart them.

cf api https://path/to/foundation
cf login
cf ssh app-usage-worker -t -c "/tmp/lifecycle/launcher /home/vcap/app 'rails c' ''"

This will launch the rails console.

Step 2:
Identify the failed delayed jobs:

Delayed::Job.where("last_error IS NOT NULL").each do |job|
  puts "Job ID: #{job.id}"
end

This will identify the failed delayed jobs, provide the error and the ID. the ID is what we need to requeue the delayed job

 

Step 3:
Re-queue the delayed jobs:

job = Delayed::Job.find(ID_FROM_BEFORE)
job.update(run_at: Time.now, failed_at: nil)

The job will be re-queued to be executed.