Horizontal Pod Autoscaler Not Scaling Node.js Pods Due to Internal V8 Heap Exhaustion

Products

VMware vSphere Kubernetes Service

Issue/Introduction

Horizontal Pod Autoscaler (HPA) fails to trigger a scale-out event for a Node.js microservice despite the pod experiencing frequent restarts.

Users may observe the pod stuck in a restart loop, while the HPA reports metrics well below the scaling threshold.

Pod restarts intermittently with no "OOMKilled" events reported by Kubernetes (cgroup).
Logs indicate internal Node.js/V8 failures, such as: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory-----> This is seen on application level
Readiness/Liveness probes fail (often 503 errors or timeouts) due to garbage collection (GC) thrashing pinning the event loop.
HPA currentMetrics shows memory utilization significantly below the defined target (e.g., 43% vs. 70% threshold).

Sample Deployment of Autoscaler to understand the metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sample-app
  namespace: default
spec:
  behavior:
    scaleDown:
      policies:
      - periodSeconds: 300      # Look-back window for this specific policy's rate limit
        type: Pods
        value: 1                 # Policy A: remove at most 1 pod per 300-second window
      - periodSeconds: 300
        type: Percent
        value: 30                # Policy B: remove at most 30% of current replicas per 300s window
      selectPolicy: Min          # When multiple scale-down policies apply, use whichever
                                  # results in the SMALLER reduction (more conservative) —
                                  # protects against scaling down too aggressively
      stabilizationWindowSeconds: 900   # Before scaling down, HPA looks back 900s (15 min)
                                          # and uses the HIGHEST replica count recommended in
                                          # that window — prevents flapping down right after a
                                          # temporary dip in load
    scaleUp:
      policies:
      - periodSeconds: 5
        type: Pods
        value: 5                 # Policy A: add at most 5 pods per 5-second window
      - periodSeconds: 5
        type: Percent
        value: 100                # Policy B: add at most 100% (double current count) per 5s window
      selectPolicy: Max          # For scale-UP, use whichever policy allows the LARGER increase
                                  # (aggressive — get capacity online fast)
      stabilizationWindowSeconds: 0     # No delay before acting on scale-up recommendations —
                                          # react immediately, don't wait to confirm the spike
                                          # is sustained
  maxReplicas: 50               # Hard ceiling — HPA will never create more than 50 pods total
  minReplicas: 3                # Hard floor — HPA will never go below 3 pods, even at zero load
  metrics:
  - resource:
      name: memory
      target:
        averageUtilization: 70   # Trigger threshold: scale up when average memory usage
        type: Utilization          # across all pods exceeds 70% of each pod's memory REQUEST
    type: Resource
  - resource:
      name: cpu
      target:
        averageUtilization: 60   # Same idea for CPU: trigger at 60% of CPU request
        type: Utilization
    type: Resource
                                  # NOTE: with 2 metrics defined, HPA calculates desired
                                  # replicas separately for each, then takes the MAX of the two —
                                  # whichever metric wants more pods wins
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app              # This HPA controls the sample-app Deployment specifically

status:
  currentMetrics:
  - resource:
      current:
        averageUtilization: 43     # Right now: memory usage averages 43% of request across
        averageValue: 963Mi         # all 3 running replicas
      name: memory
    type: Resource
  - resource:
      current:
        averageUtilization: 2      # CPU is barely used — 2% of the 500m request
        averageValue: 51m
      name: cpu
    type: Resource
  currentReplicas: 3               # HPA currently has 3 pods running
  desiredReplicas: 3               # ...and has calculated that 3 is still the correct number —
                                     # proof HPA is working correctly given its inputs; a
                                     # brief memory spike never registered high or long enough
                                     # for HPA to act on it
  conditions:
  - type: AbleToScale
    status: "True"
    reason: ScaleDownStabilized
    message: recent recommendations were higher than current one, applying the highest recent recommendation
  - type: ScalingActive
    status: "True"
    reason: ValidMetricFound
    message: the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)
  - type: ScalingLimited
    status: "False"
    reason: DesiredWithinRange
    message: the desired count is within the acceptable range

Environment

VMware vSphere Kubernetes Service

Cause

The Node.js V8 engine reaches its internal heap memory limit (--max-old-space-size) before the container-level memory limit (cgroup limit) is reached. Because the container-level usage stays below the threshold, the Kubernetes metrics server does not report sufficient load to the HPA to trigger a scale-out.

The rapid, transient nature of these V8-internal crashes often happens too quickly for the HPA's polling interval to capture the spike.

Resolution

To resolve this issue, align the Node.js internal heap ceiling more closely with the container's allocated memory limit to ensure Kubernetes metrics accurately reflect the application's actual memory pressure.

Check with the application team for the increase of heap memory in node.js level.

Below are the possible ways:

Note: If the app is deployed via helm, make the changes on the application level and redeploy via helm. Don't change the deployment yaml as it will get reverted.

Evaluate Memory Headroom: Determine the container's memory limit (from the Deployment yaml).
- Example command: kubectl get deployment <deployment-name> -n <namespace> -o yaml
- Ensure the Node.js heap limit (--max-old-space-size) is set lower than the container memory limit (cgroup limit) to leave headroom for non-heap V8 overhead (code, JIT, etc.).

Implementation Options: Set the NODE_OPTIONS environment variable in the location corresponding to your deployment workflow:

Kubernetes Deployment Manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-deployment
  labels:
    app: sample-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app-container
        image: nginx:latest
        ports:
        - containerPort: 80
        env:
        - name: NODE_OPTIONS
          value: "--max-old-space-size=1600"

Helm Chart (values.yaml):

container:
  env:
    NODE_OPTIONS: "--max-old-space-size=1600"

Dockerfile:

Redeploy: Apply the changes via your pipeline. Monitor the service for at least one full metric polling cycle to confirm stability and that the HPA correctly observes memory utilization.