High Backend Latency with Applications during Autoscaling Cold-Starts

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

During load tests or sudden traffic surges, the following symptoms are observed:

• Autoscaling: Autoscaling service adds new application instances. 
• High Response Times: Application response time spikes to a few seconds for the first few requests with the new instance. 
• Transient Latency: Latency stabilizes to sub-second levels within a few seconds of an instance starting.
• Minimal Gorouter Overhead: gorouter_time remains consistently low, while response_time is temporarily high.

Cause

The root cause is autoscaling combined with backend cold-start behavior.

1. Capacity Surge: High request volume triggers the Autoscaler to spin up new instances.
2. Initialization Latency: Initial requests hitting the new instance experience delays while internal resources, such as Spring Lazy-initialized Beans, are instantiated on first use.
3. Stability: Once the first few requests trigger full bean initialization, the instance handles subsequent traffic normally.

This is an example to simulate Spring bean lazy initialization, SlowInitService bean is being used in a Rest API controller for /api/process, 

package com.example.lazydemo;

import org.springframework.context.annotation.Lazy;
import org.springframework.stereotype.Service;

// @Lazy defers construction (and the 5s delay below) until first use instead of app startup.
@Lazy
@Service
public class SlowInitService {

    public SlowInitService() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

    public String process() {
        return "Slow service was lazily initialized and is now ready";
    }
}

In application log, the first request response time spikes to 5 seconds and the 2nd one drops under 1 second. 

2026-06-01T15:23:22.90+0200 [RTR/0] OUT spring-lazy-sample.<DOMAIN> - [2026-06-01T13:23:17.852431108Z] "GET /api/process HTTP/1.1" 200 0 72 "-" "curl/8.7.1" "###.###.###.###:34600" "###.###.###.###:61040" x_forwarded_for:"###.###.###.###, ###.###.###.###" x_forwarded_proto:"https" vcap_request_id:"6d493d8e-08dd-4c45-5b4f-c70547beb604" response_time:5.052037 gorouter_time:0.001051 app_id:"85f69166-d8c2-45fd-bd17-1c4228ff9b1b" app_index:"0" ...

2026-06-01T15:23:23.54+0200 [RTR/0] OUT spring-lazy-sample.<DOMAIN> - [2026-06-01T13:23:23.532148945Z] "GET /api/process HTTP/1.1" 200 0 69 "-" "curl/8.7.1" "###.###.###.###:34926" "###.###.###.###:61040" x_forwarded_for:"###.###.###.###, ###.###.###.###" x_forwarded_proto:"https" vcap_request_id:"a8e83229-2229-4f50-5089-e08c3f23b02f" response_time:0.010789 gorouter_time:0.000472 app_id:"85f69166-d8c2-45fd-bd17-1c4228ff9b1b" app_index:"0" ...

 

Resolution

To mitigate cold-start latency in highly dynamic environments:

• Increase Minimum Instances: Set a higher baseline instance count (e.g., minimum of 3) to absorb traffic spikes without immediately triggering the autoscaler.
• Adjust Autoscaler Thresholds: Raise the maximum instance limit to ensure sufficient overhead during sustained high load.
• Warm-up Optimization: Review application code for lazy-initialized beans that depend on external resources (databases, identity providers). Consider eager initialization for critical paths.
• Readiness Probes: Ensure health check endpoints correctly reflect "Readiness" only after the application context is fully functional to prevent the Gorouter from routing traffic too early.