Tanzu Application instances replaced unexpectedly during Diego database recreation

Products

• VMware Tanzu Application Service

Issue/Introduction

During a Diego Database (BBS) recreation, operator notices unintended app instance placements. This issue is reproduced by changing Diego Database VM type or AZ location. Diego Rep loses contact with BBS and we see AI's getting replaced during the BBS update.

Rep logs on Diego Cell show lock lost during BBS recreation:

$ tail -f /var/vcap/sys/log/rep/rep.stdout.log | grep lock

{"timestamp":"2026-07-22T19:54:25.292773470Z","level":"error","source":"rep","message":"rep.locket-lock.lost-lock"
{"timestamp":"2026-07-22T19:54:35.293163684Z","level":"error","source":"rep","message":"rep.locket-lock.failed-to-acquire-lock" Error while dialing: dial tcp ##.###.##.##:8891: connect: connection refused
{"timestamp":"2026-07-22T19:54:45.293988787Z","level":"error","source":"rep","

Environment

Product: Tanzu Elastic Application Runtime

Cause

This issue is caused by a defect in Diego BBS where the lock_ttl had been hardcoded to 15s and effectively the lock expires before keepalive time and timeout is reached (32s) by default. This can be happen in large environments when Diego database gets recreated.

Reference: https://github.com/cloudfoundry/diego-release/pull/1180

Resolution

This issue will be fixed in an upcoming release of Tanzu Elastic Application Runtime. The proposed fix will allow tuning of the lock_ttl to a bigger number like 45s, so that Diego rep can keep its presence and convergence will not kick off.

Please subscribe to this article for updated fix information.