Optimizing Airflow DAG Pod Creation Time by Disabling Virtual Environment Initialization and Using Pre-built Worker Images

Products

• VMware Tanzu Application Catalog

Issue/Introduction

DAG pod creation times in Apache Airflow Helm deployments can experience significant delays during task execution. This issue typically occurs when worker pods are configured to initialize or compile python virtual environments at runtime, causing extended startup latencies before executing task workloads.

Environment

 

• Product: Apache Airflow (Helm Chart / Bitnami Airflow)

• Components: Worker Pods, CeleryExecutor

• Platform: Kubernetes

• Configurations: Virtual environment pre-compilation enabled (prepareVenvInitContainer)

 

Cause

The preparevenv init container consumes excessive time copying the virtual environment to a temporary directory to gain write permissions for package compilation during runtime pod creation.

Resolution

To reduce pod creation latency, disable runtime virtual environment preparation and utilize a custom pre-built container image containing all required virtual environments and dependencies:

1. Update the Helm chart configuration values (values.yaml) to disable the preparevenv init container:

   YAML

    

   prepareVenvInitContainer:
     enabled: false
   

2. Configure the podSecurityContext and containerSecurityContext to run as the designated user and group configured for the Airflow cluster, while setting the root filesystem to read-only:

   YAML

    

   podSecurityContext:
     enabled: true
     runAsUser: <USER_ID>
     runAsGroup: <GROUP_ID>
     fsGroup: <GROUP_ID>
   
   containerSecurityContext:
     enabled: true
     seLinuxOptions: {}
     runAsUser: <USER_ID>
     runAsGroup: <GROUP_ID>
     runAsNonRoot: true
     privileged: false
     allowPrivilegeEscalation: false
     readOnlyRootFilesystem: true
   

3. Build and deploy a custom worker image that pre-packages all required virtual environments directly into the image payload.

4. Apply the updated Helm values to the deployment release.

 

Additional Information

• Pre-building custom container images with pre-compiled Python dependencies eliminates runtime overhead and ensures consistent pod startup times across worker nodes.

• Ensure all required Python packages are baked into the target container image prior to deploying with prepareVenvInitContainer.enabled: false.