Script to mirror resources for VPAIF-N AI workloads in air-gapped environment.

Products

• VCF Private AI Services

Issue/Introduction

Deploying VPAIF-N AI workloads in an air-gapped Deep learning VM(DLVM) or vSphere Kubernetes Service(VKS) environment will need a local Harbor registry with all required container images, helm charts and model files.  
  
A sample script is provided with this article that will help mirror resources for VPAIF-N AI workloads to a local harbor registry.  
  
Download the script package from the attached zip file. Unzip the contents to locate vpaifn-airgap-res-mirror.sh, config.env, and bom.json.  
  
By default, the bom.json file contains the resource list for the latest VCF version (e.g., identical to bom-9.0.2.json). If you require resources for an older version, you must replace the default bom.json with the specific versioned file included in the package (e.g., bom-9.0.0.json).  
  
Important Note on Legacy Versions: If you replace bom.json with an older version, please be aware that the availability of these legacy resources is not guaranteed. Some resources listed in older BOM files may have been deprecated or removed from the source registry, which may result in pull failures during the process.

Environment

VMware Private AI Foundation 9.0.x

Resolution

The script vpaifn-airgap-res-mirror.sh is designed to mirror all required resources, including container images, helm charts, model files, for VPAIF-N AI workloads deployment in an air-gapped DLVM or VKS environment.
This script is accompanied by a configuration file config.env and a resource specification file BOM.json.
Users are required to edit file config.env to provide necessary information before executing this script. The BOM.json file contains all resources that will be mirrored.

Features

• pull: Retrieve resources, including container images, helm charts and model profiles from internet and store them to local disk.
• push: Push resources, including container images, helm charts and model profiles, to specified local Harbor registry.
• all: Perform pull and push operations sequentially.
• help: Display help information.

Prerequisites

• Hardware requirements:
  • CPU: Minimum 4 cores
  • Memory: Minimum 8G RAM
  • Disk:  If default bom.json is used, at least 180G  of free available disk space.
    If the environment variable 'PACK_CONTAINER' is set to 'true', an additional 100 GB disk space will be required.
    For further details, please refer to the description of 'PACK_CONTAINER' below.
  • Network: A host system with following network configuration:
    • Has direct internet access or with Proxy configured.
    • Has access to local Harbor registry.
• Software Requirements:
  • Operating System: Ubuntu 22.04
  • Software: docker, jq, helm, vcf cli pais plugin.
• If a mirroring host is being configured on a freshly deployed Ubuntu 22.04 OS, ensure the following software should be3 installed under the system PATH:
  
  

  Software	Official Link
  docker	https://docs.docker.com/engine/install/ubuntu/
  jq	https://jqlang.org/download/
  helm	https://helm.sh/docs/intro/install/
  vcf cli pais plugin	Please refer the below instructions: VCF CLI Follow the below steps to install VCF CLI and pais plugin: Install VCF CLI sudo apt update sudo apt install -y ca-certificates curl gpg sudo mkdir -p /etc/apt/keyrings curl -fsSL https://packages.broadcom.com/artifactory/vcfcli-debian/tools/keys/BROADCOM-PACKAGING-GPG-RSA-KEY.pub; curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/PackagesKey/public | sudo gpg --dearmor -o /etc/apt/keyrings/vcf-archive-keyring.gpg echo "deb [signed-by=/etc/apt/keyrings/vcf-archive-keyring.gpg] https://packages.broadcom.com/artifactory/vcfcli-debian noble main" | sudo tee /etc/apt/sources.list.d/vcf.list sudo apt update sudo apt install -y vcf-cli Install pais plugin vcf plugin install pais Run the following command to verify plugin installation. vcf plugin list NAME DESCRIPTION INSTALLED STATUS pais Welcome to the Private AI Services platform. v2.0.0 installed Docker: If run docker as non-root user, please refer to following document: https://docs.docker.com/engine/install/linux-postinstall/

   

• If the DLVM is configured as a mirroring host, the following software needs to be installed:
  

  Software	Official Link
  helm	https://helm.sh/docs/intro/install/

• The following steps are specific to the DLVM:
  
  • Run the below commands,
    
    

    $ sudo rm /etc/docker/daemon.json
    $ sudo systemctl restart docker

• Ensure a local Harbor registry is available for hosting all the resources that the script will download from the internet.
  Create a dedicated project on the Harbor registry with least 180GB of free space. Ensure the host can access the Harbor registry to push resources.

PROXY settings

If you are running this script through proxy, please execute the following steps:

1. System proxy setup
   

   $ export HTTP_PROXY=""$HTTP_PROXY""
   $ export HTTPS_PROXY=""$HTTPS_PROXY""
   $ export NO_PROXY=""$NO_PROXY""

2. Docker Daemon proxy configuration
   
   
   1. Create a systemd drop-in directory for the docker service:

      $ sudo mkdir -p /etc/systemd/system/docker.service.d

   2. Create a file named /etc/systemd/system/docker.service.d/proxy.conf that adds the HTTP_PROXY, HTTPS_PROXY, NO_PROXY environment variables:
      
      Example:
      

          $ cat <<EOF > /etc/systemd/system/docker.service.d/proxy.conf
          [Service]
          Environment="HTTP_PROXY=http://proxy.example.com:3128"
          Environment="HTTPS_PROXY=https://proxy.example.com:3129"
          Environment="NO_PROXY=localhost,127.0.0.1,docker-registry.example.com,.corp"
          EOF 

   3. Restart docker
      
      

       $ systemctl daemon-reload
       $ systemctl restart docker 

      
      
      
   4. Please visit docker official page for more details: https://docs.docker.com/engine/daemon/proxy/

Configuration

config.env

• LOCAL_HARBOR_URI: Local Harbor registry URI, for example, example.harbor.com/paif-project
• LOCAL_HARBOR_USERNAME: Local Harbor registry username, Harbor default username is 'admin'.
• LOCAL_HARBOR_PASSWORD: Local Harbor registry password.
• NGC_API_KEY: NVIDIA NGC api key.
  Make sure the NGC account has the NVAIE entitlement required to download resources from NGC.
  Refer to NVIDIA documents for further instructions: https://docs.nvidia.com/ngc/gpu-cloud/ngc-user-guide/index.html#ngc-api-keys
• LOCAL_RESOURCES_DIR: Local disk storage directory to store downloaded resources.
• PACK_CONTAINER: controls whether container images should be saved into a compressed tarball and store to local storage. Default value is false.
  This configuration is generally set to 'true' when container images download and push operations are performed across two different machines.
  By enabling this environment variable, system will bundle each container image into a tarball. You can transfer the tar file to a remote machine, and subsequently push the image to a local Harbor registry. An additional 100GB disk space will be required if set PACK_CONTAINER to 'true'.
  For example:
  If set to 'true', the script will save each pulled container images to a tarball file.
  If set to 'false', container images will not save to tarball files.

bom.json

This file specifies all resources to be mirrored by this script. Edit bom.json under the guidance of Broadcom support only.

Usage

• Run the script using following syntax:
  
  

  $ ./vpaifn-airgap-res-mirror.sh --all
  
  Options
  -l, --pull : Pull all resources to local disk storage.
  -s, --push : Push all resources to local Harbor registry.
  -a, --all : Perform both pull and push operations sequentially.
  -h, --help : Display help information.
  

Examples

• Perform both pull and push operations
  

  ./vpaifn-airgap-res-mirror.sh --all

• Pull resources to local disk only
  

  ./vpaifn-airgap-res-mirror.sh --pull

• Push resources to Local Harbor registry
  
  

  ./vpaifn-airgap-res-mirror.sh --push