Kubernetes job fails with "com.ca.cloud.oc.JobFailedException: Job is active"

Products

• Autosys Workload Automation
• Workload Automation Agent

Issue/Introduction

When running a Kubernetes job (e.g., via the Kubernetes Plugin), the job fails immediately. The following messages are observed in the job logs:

text

 

Checking for Kubernetes config file
Kubernetes config file not found
...
Job Active: Job is active
job failed: Job is active
com.ca.cloud.oc.JobFailedException: Job is active
@at com.ca.cloud.oc.OpenShift.monitorJob(OpenShift.java:####)
@at com.ca.cloud.oc.OpenShiftESJ.monitorJob(OpenShiftESJ.java:####)

 

Environment

Workload Automation Agent for Kubernetes

Cause

This behavior is driven by two factors:

1. Informational Message: The log entry Kubernetes config file not found is not an error. It is a status message confirming that the agent is using an alternative authentication method (such as OAuth or Service Account tokens) rather than a local kubeconfig file.
2. Naming Conflict: The Job is active exception occurs because a job with the same name is already running or exists in the target Kubernetes namespace. Kubernetes batch/v1 API restrictions dictate that only one job with a specific name can exist in a namespace at one time. If the previous run has not been deleted or is still in an "Active" state, the new submission will fail.

Resolution

To resolve this issue, ensure the namespace is clear of previous job instances before starting a new one:

1. Identify Active Jobs: Use the following command to check for existing jobs in the namespace: kubectl get jobs -n ####-namespace
2. Clear Existing Jobs: If a job with the same name exists, delete it manually or ensure your automation logic includes a cleanup step: kubectl delete job ####-jobname -n ####-namespace
3. Relaunch: Once the conflict is removed, restart the job from your workload automation scheduler (e.g., AutoSys).

Note: 

• Always ensure that job names are unique or that the environment is cleaned up between runs to prevent "Job is active" conflicts.
• The real problem is trying to run a Kubernetes job which is already running.  This a Kubernetes restriction with kind job. Only one job with a particular job name is allowed to run. The agent will check if an existing job with the same name is still running and will fail if it is. If the job is not running, then it will delete the job which is a Kubernetes requirement.