Script to mirror resources for VPAIF-N AI workloads in air-gapped environment.
search cancel

Script to mirror resources for VPAIF-N AI workloads in air-gapped environment.

book

Article ID: 388470

calendar_today

Updated On:

Products

VCF Private AI Services

Issue/Introduction

Deploying VPAIF-N AI workloads in an air-gapped Deep learning VM(DLVM) or vSphere Kubernetes Service(VKS) environment will need a local Harbor registry with all required container images, helm charts and model files.  
  
A sample script is provided with this article that will help mirror resources for VPAIF-N AI workloads to a local harbor registry.  
  
Download the script package corresponding to your target VCF version (e.g., 9.0.x or 9.1.x) from the attached zip files. Unzip the contents to locate vpaifn-airgap-res-mirror.sh, config.env, and the included BOM (.json) files.
  
Default Configuration: By default, the bom.json file contains the resource list for the latest available release within that specific series.
For example, in the 9.0.x attachment, bom.json currently defaults to the 9.0.2 release (identical to the included bom-9.0.2.json).
Similarly, in the 9.1.x attachment, the default bom.json currently defaults to the 9.1.1 release (identical to the included bom-9.1.1.json).

Manual Version Switching: If you require resources for an older release within the same series (e.g., switching from 9.0.2 to 9.0.0), you must manually overwrite the default bom.json with the desired file included in your unzipped package. 
 
Important Note on Legacy Versions: If you replace bom.json with an older version, please be aware that the availability of these legacy resources is not guaranteed. Some resources listed in older BOM files may have been deprecated or removed from the source registry, which may result in pull failures during the process.

Environment

VMware Private AI Foundation 9.0.x, 9.1.x

Resolution

The script vpaifn-airgap-res-mirror.sh is designed to mirror all required resources, including container images, helm charts, model files, for VPAIF-N AI workloads deployment in an air-gapped DLVM or VKS environment.
This script is accompanied by a configuration file config.env and a resource specification file BOM.json.
Users are required to edit file config.env to provide necessary information before executing this script. The BOM.json file contains all resources that will be mirrored.

Features

  • pull: Retrieve resources, including container images, helm charts and model profiles from internet and store them to local disk.
  • push: Push resources, including container images, helm charts and model profiles, to specified local Harbor registry.
  • all: Perform pull and push operations sequentially.
  • help: Display help information.

Prerequisites

  • Hardware requirements:
    • CPU: Minimum 4 cores
    • Memory: Minimum 8G RAM
    • Disk:  If default bom.json is used, at least 180G of free disk space on the host.
      If the environment variable 'PACK_CONTAINER' is set to 'true', an additional 100 GB disk space will be required.
      For further details, please refer to the description of 'PACK_CONTAINER' below.
    • Network: A host system with following network configuration:
      • Has direct internet access or with Proxy configured.
      • Has access to local Harbor registry.
  • Software Requirements:
    • Operating System: Ubuntu 22.04 or Ubuntu 24.04
    • Software: docker, jq, helm, vcf cli pais plugin.
  • If a mirroring host is being configured on a freshly deployed Ubuntu 22.04 or Ubuntu 24.04 OS, the following software and actions are required. Ensure the following software is installed under the system's PATH:

    SoftwareOfficial Link
    docker
    (v27.5 or earlier)    		https://docs.docker.com/engine/install/ubuntu/ 
    jqhttps://jqlang.org/download/
    helmhttps://helm.sh/docs/intro/install/
    vcf cli pais plugin

    Please refer the below instructions:

    VCF CLI(VCF 9.0.x only)

    Follow the below steps to install VCF CLI and pais plugin:

    1. Install VCF CLI
      sudo apt update
sudo apt install -y ca-certificates curl gpg
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://packages.broadcom.com/artifactory/vcfcli-debian/tools/keys/BROADCOM-PACKAGING-GPG-RSA-KEY.pub; curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/PackagesKey/public | sudo gpg --dearmor -o /etc/apt/keyrings/vcf-archive-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/vcf-archive-keyring.gpg] https://packages.broadcom.com/artifactory/vcfcli-debian noble main" | sudo tee /etc/apt/sources.list.d/vcf.list
sudo apt update
sudo apt install -y vcf-cli
    2. Install pais plugin
      vcf plugin install pais
    3. Run the following command to verify plugin installation.
      vcf plugin list

        NAME       DESCRIPTION                                   INSTALLED  STATUS
        pais       Welcome to the Private AI Services platform.  v2.1.0     installed

    Docker:

    If run docker as non-root user, please refer to following document: https://docs.docker.com/engine/install/linux-postinstall/

     

  • If the DLVM is configured as a mirroring host, the following software needs to be installed:
    • Only DLVM versions prior to 9.1.1 are supported (DLVM 9.1.1 is not supported).
    • Ensure the following software is installed under the system's PATH:
      SoftwareOfficial Link
      helmhttps://helm.sh/docs/intro/install/

       

  • Please execute the following steps on DLVM (for VCF 9.0.x only):
    $ sudo rm /etc/docker/daemon.json
$ sudo systemctl restart docker

     

  • Ensure a local Harbor registry is available for hosting all the resources that the script will download from the internet.
    • Create a dedicated project on the Harbor registry with least 180GB of free space.
    • Ensure the host can access the Harbor registry to push resources.

PROXY settings

If you are running this script through proxy, please execute the following steps:

  1. System proxy setup
    $ export HTTP_PROXY=""$HTTP_PROXY""
$ export HTTPS_PROXY=""$HTTPS_PROXY""
$ export NO_PROXY=""$NO_PROXY""
  2. Docker Daemon proxy configuration

    1. Create a systemd drop-in directory for the docker service:
      $ sudo mkdir -p /etc/systemd/system/docker.service.d
    2. Create a file named /etc/systemd/system/docker.service.d/proxy.conf that adds the HTTP_PROXY, HTTPS_PROXY, NO_PROXY environment variables:

      Example:
          $ cat <<EOF > /etc/systemd/system/docker.service.d/proxy.conf
    [Service]
    Environment="HTTP_PROXY=http://proxy.example.com:3128"
    Environment="HTTPS_PROXY=https://proxy.example.com:3129"
    Environment="NO_PROXY=localhost,127.0.0.1,docker-registry.example.com,.corp"
    EOF

       

    3. Restart docker

       $ systemctl daemon-reload
 $ systemctl restart docker


    4. Please visit docker official page for more details: https://docs.docker.com/engine/daemon/proxy/

Configuration

config.env

  • LOCAL_HARBOR_URI: Local Harbor registry URI, for example, example.harbor.com/paif-project
  • LOCAL_HARBOR_USERNAME: Local Harbor registry username, Harbor default username is 'admin'.
  • LOCAL_HARBOR_PASSWORD: Local Harbor registry password.
  • NGC_API_KEY: NVIDIA NGC api key.
    Make sure the NGC account has the NVAIE entitlement required to download resources from NGC.
    Refer to NVIDIA documents for further instructions: https://docs.nvidia.com/ngc/gpu-cloud/ngc-user-guide/index.html#ngc-api-keys
  • LOCAL_RESOURCES_DIR: Local disk storage directory to store downloaded resources.
  • PACK_CONTAINER: controls whether container images should be saved into a compressed tarball and store to local storage. Default value is false.
    This configuration is generally set to 'true' when container images download and push operations are performed across two different machines.
    By enabling this environment variable, system will bundle each container image into a tarball. You can transfer the tar file to a remote machine, and subsequently push the image to a local Harbor registry. An additional 100GB disk space will be required if set PACK_CONTAINER to 'true'.
    For example:
    If set to 'true', the script will save each pulled container images to a tarball file.
    If set to 'false', container images will not save to tarball files.

bom.json

This file specifies all resources to be mirrored by this script. Edit bom.json under the guidance of Broadcom support only.

Usage

  • Run the script using following syntax:

    $ ./vpaifn-airgap-res-mirror.sh --all

Options
-l, --pull : Pull all resources to local disk storage.
-s, --push : Push all resources to local Harbor registry.
-a, --all : Perform both pull and push operations sequentially.
-h, --help : Display help information.

Examples

  • Perform both pull and push operations
    ./vpaifn-airgap-res-mirror.sh --all
  • Pull resources to local disk only
    ./vpaifn-airgap-res-mirror.sh --pull
  • Push resources to Local Harbor registry

    ./vpaifn-airgap-res-mirror.sh --push

Additional Information

NVIDIA GPU operator deployment in an air-gapped environment typically requires a local Ubuntu package repository unless precompiled driver containers are used.

After the local Ubuntu package repository is ready, make sure the generated custom-repo.list file is also uploaded to the Harbor registry following PAIF documentation

Security Updates

Updating the TKG GPU Console Container Image

Starting in PAIF 9.1.1, a more secure option is available for the tkg-gpu-console container image, which we encourage customers to adopt.

To replace the tkg-gpu-console container image with the new deep-learning-container image, perform the following steps:

  1. Before running this script, login to the VCF registry with this command:
    echo <registry_token> | docker login dlvm.packages.broadcom.com --username <user_name> --password-stdin

<user_name> is the account the customer uses to log in to support.broadcom.com.
<registry_token> can be generated on support.broadcom.com after logging in.

    Without this step, the deep-learning-container:9.1.1.0 image cannot be pulled

  2. Modify the relevant VCFA blueprint.

    On VCFA UI, edit the AI Kubernetes Cluster blueprint. Find this line 
    image: ${variable.registryUri}/images/tkg-gpu-console:9.0.0.0

    Replace string "tkg-gpu-console:9.0.0.0" with "dlvm/ga/9.1.1.0/deep-learning-container:9.1.1.0"

  3. Then save and publish this blueprint

Attachments

vpaifn-airgap-res-mirror-9.1.x.zip get_app
vpaifn-airgap-res-mirror-9.0.x.zip get_app
Powered by Wolken Software