Script to mirror resources for VPAIF-N AI workloads in air-gapped environment.
search cancel

Script to mirror resources for VPAIF-N AI workloads in air-gapped environment.

book

Article ID: 388470

calendar_today

Updated On:

Products

VMware Private AI Foundation

Issue/Introduction

Deploying VPAIF-N AI workloads in an air-gapped Deep learning VM(DLVM) or vSphere Kubernetes Service(VKS) environment will need a local Harbor registry with all required container images, helm charts and model files.

A sample script is provided with this article that will help mirror resources for VPAIF-N AI workloads to a local harbor registry.

You can download the sample script and two configuration files from the zip file attached to this article.  Unzip the attached zip file and ensure that the files vpaifn-airgap-res-mirror.sh, config.env and bom.json, are located in the same directory

Environment

VMware Private AI Foundation 9.0

Resolution

The script vpaifn-airgap-res-mirror.sh is designed to mirror all required resources, including container images, helm charts, model files, for VPAIF-N AI workloads deployment in an air-gapped DLVM or VKS environment.
This script is accompanied by a configuration file config.env and a resource specification file BOM.json.
Users are required to edit file config.env to provide necessary information before executing this script. The BOM.json file contains all resources that will be mirrored.

Features

  • pull: Retrieve resources, including container images, helm charts and model profiles from internet and store them to local disk.
  • push: Push resources, including container images, helm charts and model profiles, to specified local Harbor registry.
  • all: Perform pull and push operations sequentially.
  • help: Display help information.

Prerequisites

  • Hardware requirements:
    • CPU: Minimum 4 cores
    • Memory: Minimum 8G RAM
    • Disk:  If default bom.json is used, at least 180G  of free available disk space.
      If the environment variable 'PACK_CONTAINER' is set to 'true', an additional 100 GB disk space will be required.
      For further details, please refer to the description of 'PACK_CONTAINER' below.
    • Network: A host system with following network configuration:
      • Has direct internet access or with Proxy configured.
      • Has access to local Harbor registry.
  • Software Requirements:
    • Operating System: Ubuntu 22.04
    • Software: docker, jq, helm, vcf cli pais plugin.
  • If a mirroring host is being configured on a freshly deployed Ubuntu 22.04 OS, ensure the following software should be3 installed under the system PATH:

    Software Official Link
    docker https://docs.docker.com/engine/install/ubuntu/ 
    jq https://jqlang.org/download/
    helm https://helm.sh/docs/intro/install/
    vcf cli pais plugin

    Please refer the below instructions:

    VCF CLI

    Follow the below steps to install VCF CLI and pais plugin:

    1. Install VCF CLI
      sudo apt update
sudo apt install -y ca-certificates curl gpg
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://packages.broadcom.com/artifactory/vcfcli-debian/tools/keys/BROADCOM-PACKAGING-GPG-RSA-KEY.pub; curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/PackagesKey/public | sudo gpg --dearmor -o /etc/apt/keyrings/vcf-archive-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/vcf-archive-keyring.gpg] https://packages.broadcom.com/artifactory/vcfcli-debian noble main" | sudo tee /etc/apt/sources.list.d/vcf.list
sudo apt update
sudo apt install -y vcf-cli
    2. Install pais plugin
      vcf plugin install pais
    3. Run the following command to verify plugin installation.
      vcf plugin list

        NAME       DESCRIPTION                                   INSTALLED  STATUS
        pais       Welcome to the Private AI Services platform.  v2.0.0     installed

    Docker:

    If run docker as non-root user, please refer to following document: https://docs.docker.com/engine/install/linux-postinstall/

     

  • If the DLVM is configured as a mirroring host, the following software needs to be installed:
    Software Official Link
    helm https://helm.sh/docs/intro/install/
  • The following steps are specific to the DLVM:
    • Run the below commands,

      $ sudo rm /etc/docker/daemon.json
$ sudo systemctl restart docker
  • Ensure a local Harbor registry is available for hosting all the resources that the script will download from the internet.
    Create a dedicated project on the Harbor registry with least 180GB of free space. Ensure the host can access the Harbor registry to push resources.

PROXY settings

If you are running this script through proxy, please execute the following steps:

  1. System proxy setup
    $ export HTTP_PROXY=""$HTTP_PROXY""
$ export HTTPS_PROXY=""$HTTPS_PROXY""
$ export NO_PROXY=""$NO_PROXY""
  2. Docker Daemon proxy configuration

    1. Create a systemd drop-in directory for the docker service: 
      $ sudo mkdir -p /etc/systemd/system/docker.service.d
    2. Create a file named /etc/systemd/system/docker.service.d/proxy.conf that adds the HTTP_PROXY, HTTPS_PROXY, NO_PROXY environment variables:

      Example:
          $ cat <<EOF > /etc/systemd/system/docker.service.d/proxy.conf
    [Service]
    Environment="HTTP_PROXY=http://proxy.example.com:3128"
    Environment="HTTPS_PROXY=https://proxy.example.com:3129"
    Environment="NO_PROXY=localhost,127.0.0.1,docker-registry.example.com,.corp"
    EOF
    3. Restart docker

       $ systemctl daemon-reload
 $ systemctl restart docker


    4. Please visit docker official page for more details: https://docs.docker.com/engine/daemon/proxy/

Configuration

config.env

  • LOCAL_HARBOR_URI: Local Harbor registry URI, for example, example.harbor.com/paif-project
  • LOCAL_HARBOR_USERNAME: Local Harbor registry username, Harbor default username is 'admin'.
  • LOCAL_HARBOR_PASSWORD: Local Harbor registry password.
  • NGC_API_KEY: NVIDIA NGC api key.
    Make sure the NGC account has the NVAIE entitlement required to download resources from NGC.
    Refer to NVIDIA documents for further instructions: https://docs.nvidia.com/ngc/gpu-cloud/ngc-user-guide/index.html#ngc-api-keys
  • LOCAL_RESOURCES_DIR: Local disk storage directory to store downloaded resources.
  • PACK_CONTAINER: controls whether container images should be saved into a compressed tarball and store to local storage. Default value is false.
    This configuration is generally set to 'true' when container images download and push operations are performed across two different machines.
    By enabling this environment variable, system will bundle each container image into a tarball. You can transfer the tar file to a remote machine, and subsequently push the image to a local Harbor registry. An additional 100GB disk space will be required if set PACK_CONTAINER to 'true'.
    For example:
    If set to 'true', the script will save each pulled container images to a tarball file.
    If set to 'false', container images will not save to tarball files.

bom.json

This file specifies all resources to be mirrored by this script. Edit bom.json under the guidance of Broadcom support only.

Usage

  • Run the script using following syntax:

    $ ./vpaifn-airgap-res-mirror.sh --all

Options
-l, --pull : Pull all resources to local disk storage.
-s, --push : Push all resources to local Harbor registry.
-a, --all : Perform both pull and push operations sequentially.
-h, --help : Display help information.

Examples

  • Perform both pull and push operations
    ./vpaifn-airgap-res-mirror.sh --all
  • Pull resources to local disk only
    ./vpaifn-airgap-res-mirror.sh --pull
  • Push resources to Local Harbor registry

    ./vpaifn-airgap-res-mirror.sh --push

Attachments

vpaifn-airgap-res-mirror.zip get_app
Powered by Wolken Software