Private AI Services (PAIS) deployment fails due to incorrect tagging of the GPU operator chart and image in the local Harbor repository

Products

• VCF Private AI Services

Issue/Introduction

1.PAIS deployment tasks hang on gpu-operator pod creation failures:

kubectl -n gpu-operator describe pod gpu-operator-xxxxxx

Error:failed to create contained task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during cntainer init: exec: "gpu-operator": executable file not found in $PATH

2.PAIF installed in disconnected environment and all needed resource already upload to local harbor refer to below doc:

https://techdocs.broadcom.com/us/en/vmware-cis/private-ai/foundation-with-nvidia/9-1/deploying-private-ai-foundation-with-nvidia/installing-and-configuring-private-ai-services/upload-the-private-ai-services-components-to-a-disconnected-environment.html

Environment

VCF Private AI Service 2.1

Resolution

1. Check the gpu-operator path on the local Harbor registry. You will see two files: one image type and one chart type.

 

2. Manually remove the tag from the chart file and apply it to the image file instead.

 

3. Delete the problematic pod using the following command:

 

kubectl delete pod -n gpu-operator -l app=gpu-operator

 

After a short wait, a new gpu-operator pod should successfully deploy in the gpu-operator namespace.

 

4. Manually remove the tag from the image file and assign it back to the chart file. If the gpu-operator-upgrade-crd job fails while waiting for the PAIS configuration to complete, run the following command:

 

kubectl delete job gpu-operator-upgrade-crd -n gpu-operator