Composed task execution run.id already exists in DB error,

Products

• VMware Tanzu Spring Essentials

Issue/Introduction

While executing a composed task, a user may encounter an error due to a uniq run.id

Caused by: org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException: A job instance already exists and is complete for parameters={run.id=###}.  If you want to run this job again, change the parameters.

 

Cause

Background

Spring Cloud Data Flow's Composed Task Runner (CTR) is a Spring Batch application. Spring Batch requires that every execution of the same job is uniquely identified by its job parameters — this is how it tracks execution history, prevents duplicate runs, and enables restartability.

By default, the CTR uses a parameter called run.id (a sequential Long) to satisfy this uniqueness requirement. Before each launch, the CTR reads the last run.id value from the database and increments it by 1. This value is passed to the SCDF server as part of the task launch request, and the SCDF server records it in the Batch metadata tables.

The Problem with Sequential run.id

The sequential incrementer is stateful: it depends entirely on reading the correct last value from the database. Any disruption that affects the consistency of that state — such as a database restart, point-in-time restore, failover to a replica, or partial rollback — can cause the incrementer to return a value that has already been used for a successfully completed run.

When that happens, Spring Batch throws:

JobInstanceAlreadyCompleteException: A job instance already exists and is completefor parameters={run.id=<N>}.

This error repeats on every subsequent launch because the underlying database state does not change. The CTR cannot proceed regardless of how many times it is retried.

Resolution

Recommended Configuration

Add the following to the composed task launch properties:

app.composed-task-runner.uuid-instance-enabled=true

This can be set as a default deployment property on the composed task definition so that it applies to all future launches without requiring per-request configuration.

 

 

The uuid-instance-enabled Parameter

Property: app.composed-task-runner.uuid-instance-enabled=true

When set to true, the CTR replaces the sequential run.id counter with a randomly generated UUID (e.g. ######-#####-#####-######). This UUID is generated fresh at startup, without reading anything from the database. It is passed to the SCDF server as the job-identifying parameter ctr.id instead of run.id.

This makes uniqueness stateless: there is no dependency on the current contents of any database table, and no value that a DB restart or restore can invalidate.

Relationship to the SCDF Server

The SCDF server is responsible for launching the CTR as a task. The job parameters — including the uniqueness parameter — are passed from the server to the CTR at launch time and recorded in the Batch metadata tables shared between the server and the CTR. The server itself does not generate run.id or ctr.id; that is entirely the CTR's responsibility via its configured incrementer.

When uuid-instance-enabled=true, the server receives and records ctr.id instead of run.id. This is fully supported and transparent to the server — no server-side configuration change is required.

Is This Safe for Production?

Yes. uuid-instance-enabled=true is the recommended configuration for production use. Specifically:

• The sequential run.id approach (increment-instance-enabled) is deprecated and retained only for backwards compatibility.
• UUID-based identification is required when running multiple instances of the same composed task definition concurrently — sequential counters are not safe under concurrent launches, UUIDs are.
• UUID generation does not rely on any shared state, making it resilient to database disruptions of any kind.
• All SCDF server and CTR functionality (monitoring, execution history, restartability of individual steps) continues to work normally with ctr.id.

Reading the Logs After Enabling uuid-instance-enabled

With uuid-instance-enabled=true in effect, log output and error messages will reference both run.id and ctr.id. This is expected and not a cause for concern.

For example, a successful launch will show something like:

Job: [FlowJob: [name=my-composed-task]] launched with the following parameters:[{run.id=###, ctr.id=####-####-####-####-##########}]

• ctr.id is the UUID generated by the CTR. This is the value that uniquely identifies the job instance. It is what prevents collisions across launches.
• run.id is still present because it is carried forward from the previous execution's parameters by the incrementer machinery — but it is no longer the identifying parameter. It plays no role in uniqueness and can be ignored.

In practical terms: when correlating log output or database records across launches, use ctr.id as the stable identifier for a given composed task execution. run.id will continue to increment but is effectively inert.